2020-08-03 TIS NDW ETLs didn't run
Date | Aug 3, 2020 |
Authors | @Andy Nash (Unlicensed) |
Status | Complete |
Summary | TIS NDW ETLs didn't run on Stage or Prod |
Impact | Downstream NDW processes based on ‘stale’ data |
Root Cause(s)
Trigger
Resolution
Detection
Action Items
Root Cause(s)
The scheduled task didn’t execute
The build server (with the schedule) didn’t get triggered in the expected timeframe
The build server was unavailable
Automated change requests caused a massive spike in load
Trigger
Unable to run the build on schedule
Resolution
Restarted Jenkins.
Manually triggered the
tis-ndw-etl-cron-prod-to-prod
,tis-ndw-etl-cron-stage-to-uat
andtis-esr-etl-cron-notification-daily-load
jobs.Notified NDW team via. slack channel.
Detection
Noticed while checking weekly ETL run
Action Items
Action Item | Owner | Issue |
---|---|---|
Trigger scheduled tasks differently. |
|
|
Slack: https://hee-nhs-tis.slack.com/
Jira issues: https://hee-tis.atlassian.net/issues/?filter=14213