Non-technical Description

GMC sync for old reval didn't work last night as were being blocked by the GMC

Trigger

Unknown change on GMC firewall

Detection

Slack notification received

on inspection of the logs we found this:

Resolution

emailed GMC who stated that they had made a security change overnight and that had made the service unavailable to us, they made a change on their end and asked us to rerun our job that was completed successfully.

Timeline

• 03 Dec 2021 - 00:05 alert send by nightly job to Slack

• 03 Dec 2021 - 08: 57 reran job to see if it was a temporary network error

• 03 Dec 2021 - 09:10 Adewale Adekoya sent an email to GMC asking if the problem could be looked into

• 03 Dec 2021 - 09:39 Peter McNair from the GMC replied to the email asking us to try again

• 03 Dec 2021 - 09:43 ran the job again and it was completed successfully.

Root Cause(s)

Security change at GMC end caused the nightly ETL not to be able to connect to get the data needed.

Action Items

Lessons Learned