2019-05-24 GMC Sync and Interpid-dr-etl failure

Owned by John Simmons (Deactivated)
24 May 2019


Date
 
Authors
John Simmons (Deactivated)
StatusGMC Sync and Interpid-dr-etl failure
Summary

Docker wasn't able to run the services as the containers didn't exist in the correct location

ImpactNo current information in Reval 



Impact

No new data in the Reval App

Root Causes

  • GMC-SYNC and Interpid-dr-etl docker containers were not synced to new ACR location

Trigger

  • Slack notification

Resolution

  • Copied the missing docker images/manifests from old repo to new repo and reran the jenkins jobs

Detection / Timeline

  • 2019-05-24 01:05 & 02:00 slack alerts for failed jobs
  • 2019-05-24 06:50 Noticed that jobs hadnt run
  • 2019-05-24 07:00 notified slack I was working on the issue
  • 2019-05-24 07:37 all images copied over to new registry
  • 2019-05-24 07:39 etl restarted
  • 2019-05-24 07:55 all etls completed


Lessons Learned

  • Should have taken ETL's into consideration of docker images that needed moving to new repo

What went well

  • Fast fix, and re-run of etl's meaning that the correct data was in prod before the sart of the working day.

What went wrong

  • Shouldnt have missed the etl's in the first place


Slack: https://hee-nhs-tis.slack.com/
Jira issues: https://hee-tis.atlassian.net/issues/?filter=14213