Date | 21 |
Authors | |
Status | In Progress |
Summary | ETL failed after an initial attempt at moving it into our new AWS ECS environment |
Impact | Downstream NDW and Hicom processes failed - data in NDW was 1 day out of date (this has now been fixed) |
Non-technical Description
TIS team moved our ETL from Jenkins to AWS environment. Although it ran, it started later than needed and took longer to complete. As a result the downstream NDW processes that begin at 03.45 had no fresh data to work with, and the Hicom pull into Leave Manager could not pull live data.
...
Trigger
Moving the ETL from Jenkins to ECS, not starting it at the right time and not optimising the speed with which it completed to ensure had it started at the right time, it would have completed in time for the downstream NDW and Hicom processes to be able to use current data.
...
Detection
No alerts in the #monitoring-ndw channel (alerts had been moved to the #monitoring-prod channel).
...
Data lead and NDW team member enquiries in shared #tis-ndw-etl channel Slack channel.
...
Resolution
Revert to a manual reuse of the Jenkins ETL in order that downstream processes can be completed later today.
...
Put a process in place to ensure the successful running of TIS NDW ETLs, after 0245 and before 03:45 each morning, over the weekend/Monday morning.
...
Timeline
: 03:15 - No alert in the #monitoring-ndw channel at the expected time (c.03.15 - 03:35)
...
: 12:39 - Pepe confirms the successful re-running of the TIS NDW ETL, and informs NDW they can re-run their downstream job
Root Cause(s)
Focused on getting the ETLs to work in ECS. Lack of focus on the scheduling of the ETLs and the speed with which they needed to be processed in order to not impact dependencies.
Lessons Learned
Be carefully when modifying ETLs. Modify the ETL on Stage first. Consider carefully whether it would be best to run this overnight before then promoting the modification to Prod the next day on confirmation of successful execution of modification on Stage. Refer to the image to the right when modifying in future (and keep that image up to date). Keep the overall ETL Timings (re-draft - for discussion) page up to date |
|
.