Date	Aug 30, 2021
Authors	@John Simmons (Deactivated)
Status	Documenting
Summary	Overnight weekend GMC sync failed
Impact	Data was left unsynchronised on Reval for half a day

Non-technical Description

GMC sync failed - Due to GMC working on the Service on the Bank Holiday weekend. Data at our end didn’t initially sync. We spotted the problem, re-ran the jobs and all data on Reval is now back up to date.

Trigger

Awaiting clarification from GMC.

Detection

Monitoring alert on our Slack channel.

Resolution

Re-ran the jobs.

Timeline

Aug 30, 2021 - 01.05: Slack alert that the GMC sync failed.
Aug 30, 2021 - 08.40: Slack alert that the GMC sync was still failing.
Aug 30, 2021 - 14.51: GMC sync jobs were re-run successfully.

Root Cause(s)

Maintenance on the GMC servers made the unreachable when the ETL ran, after the maintenance was completed we reran the jobs that were then completed successfully.

Action Items

Action Items	Owner

Action Items	Owner
Add section to GMC-Sync to check for connection before proceeding.	@John Simmons (Deactivated)
Add section to GMC-Sync to check for free disk space before proceeding	@John Simmons (Deactivated)

Lessons Learned

TBC.

TIS21 Confluence Space

2021-08-30 GMC sync failed

Analytics