Non-technical Description

GMC sync failed

Trigger

Unknown change on GMC firewall

Detection

Slack notification received

on inspection of the logs we found this:

Resolution

TODO: .

Timeline

• 06 Dec 2021 - 00:05 alert send by nightly job to Slack

• 06 Dec 2021 - 09: 50 reran job to see if it was a temporary network error. Adewale Adekoya sent an email to GMC asking if the problem could be looked into

• 06 Dec 2021 - Began investigating with GMC.

• 07 Dec 2021 - 00:05 GMC-Sync succeeded

• 07 Dec 2021 - 09:20 Re-attempted changing connetions but this failed

• 07 Dec 2021 - Issue narrowed down to an IP address but this hasn’t resolved the issue

• 07 Dec 2021 - Further debugging info has been exchanged

• 08 Dec 2021 - A further fix did not resolve the issue, and

• Dec 9, 2021 - Sync job ran successfully however no updates from GMC on any changes made on their side

• Dec 9, 2021 - Adewale Adekoya further enquires for any updates from the GMC but no response received

Root Cause(s)

• Alerting from Slack (GMC Sync) & on MS Teams (Connections failure)

• Logs show failed Web Service requests

Action Items

Lessons Learned