Non-technical Description

GMC sync for old reval didn't work last night as were being blocked by the GMC

Trigger

Unknown change on GMC firewall

Detection

Slack notification received

Open

on inspection of the logs we found this:

Open

Resolution

emailed GMC who stated that they had made a security change overnight and that had made the service unavailable to us, they made a change on their end and asked us to rerun our job that was completed successfully.

Timeline

• Dec 3, 2021 - 00:05 alert send by nightly job to Slack

• Dec 3, 2021 - 08: 57 reran job to see if it was a temporary network error

• Dec 3, 2021 - 09:10 @Adewale Adekoya sent an email to GMC asking if the problem could be looked into

• Dec 3, 2021 - 09:39 Peter McNair from the GMC replied to the email asking us to try again

• Dec 3, 2021 - 09:43 ran the job again and it was completed successfully.

Root Cause(s)

Security change at GMC end caused the nightly ETL not to be able to connect to get the data needed.

Action Items

Lessons Learned