2021-08-30 GMC sync failed

Owned by Andy Nash (Unlicensed)
Last updated: 02 Sept 2021 by John Simmons (Deactivated)
1 min read

Date

Aug 30, 2021

Authors

@John Simmons (Deactivated)

Status

Documenting

Summary

Overnight weekend GMC sync failed

Impact

Data was left unsynchronised on Reval for half a day

Non-technical Description

GMC sync failed - Due to GMC working on the Service on the Bank Holiday weekend. Data at our end didn’t initially sync. We spotted the problem, re-ran the jobs and all data on Reval is now back up to date.

Trigger

  • Awaiting clarification from GMC.

Detection

  • Monitoring alert on our Slack channel.

Resolution

  • Re-ran the jobs.

Timeline

  • Aug 30, 2021 - 01.05: Slack alert that the GMC sync failed.

  • Aug 30, 2021 - 08.40: Slack alert that the GMC sync was still failing.

  • Aug 30, 2021 - 14.51: GMC sync jobs were re-run successfully.

 

Root Cause(s)

  • Maintenance on the GMC servers made the unreachable when the ETL ran, after the maintenance was completed we reran the jobs that were then completed successfully.

Action Items

Action Items

Owner

Action Items

Owner

Add section to GMC-Sync to check for connection before proceeding.

@John Simmons (Deactivated)

Add section to GMC-Sync to check for free disk space before proceeding

@John Simmons (Deactivated)

 

 

 

 

Lessons Learned

  • TBC.

 

Slack: https://hee-nhs-tis.slack.com/
Jira issues: https://hee-tis.atlassian.net/issues/?filter=14213