Date |
|
Authors | |
Status | Documenting |
Summary | The ETL that feeds reporting failed and meant that partial information was picked up and made available in the NDW |
Impact | Reports would have been missing Post Funding and based on a mix of data as of 25th & 26th Jan |
Non-technical Description
Trigger
Automated deployment to prod
Detection
Slack
Resolution
Re-ran ETL with previous version
Set scheduled run to use previous version
Timeline
All times in GMT unless indicated
- Change to the preprod environment that required a change in the ETL
-
- Updated the scheduled event to use latest task definitions
Root Cause(s)
We got a slack message that the ETL was running for NHS E & NIMDTA and there was no completion message
The ETL was retrying the step that creates Post Funding
The step was failing because the SQL included a field which didn’t exist
The ETL relied on some database changes which hadn’t been released to production sites yet
The ETL workflow automatically deploys unless it is cancelled in a 5 minute window
The workflow can’t have an approval step while the source is private and part of the current subscription
Action Items
Action Items | Owner | |
|---|---|---|
Add max retries for all steps to reduce the scope of the failure | ||
Spike: What are the options for adding workflow approvals? explore options for enabling GHA workflow approvals:
|
Add Comment