Date |
|
Authors | |
Status | Documenting |
Summary | ETL failed after modifying the configuration |
Impact | Data in NDW was 1 day out of date for a short period out-of-hours |
Non-technical Description
As part of moving the HEE ETLs, the configuration identified additional availability zone (AZ) that this ETL could run in. The ETL attempted to run in this additional AZ but it was not fully configured and failed.
Trigger
Additional AZ added to configuration, meant the job could have and did run in a location that didn’t have the required access.
Detection
Notifications in the #monitoring-ndw channel.
TIS Data Manager confirmed no downstream processes were affected.
Resolution
Re-ran in fully configured part of network.
Long-term:
Altered configuration to run on the same subnet as the database.
Removed unused/partially functional subnet
Timeline
: 02:30 - Failure message in the #monitoring-ndw channel
: 03:00 to 03:20 - Configuration modified and job rerun
: - Configuration (IaC) definition modified to use the subnet that the source database runs on.
w/c :- Additional unused infrastructure decommisioned
Root Cause(s)
Additional infrastructure created outside of normal processes.
Change applied with assumptions about correctness of IaC definitions.
Lessons Learned
Just because it looks right, it doesn’t mean it is.
Action Items
Action Items | Owner | Status |
|---|---|---|
Update the ETL timings page. Use wider list of ETLs, i.e. image with swim-lanes above. | Page updated. Diagram with swimlanes/showing full dependency graph still needs to be updated. |
Add Comment