Non-technical Description

GMC sync failed and subsequently failed to retrieve any doctors

Trigger

Unknown change by the GMC. There was initially a change to the firewall. The continuing problem, with a different way of failing suggests that there was another GMC which also caused an issue.

Detection

Slack notification received

on inspection of the logs we found this:

Resolution

Contacted GMC as the issue was on their service.

GMC have not elaborated on the cause or resolution

Timeline

• 06 Dec 2021 - 00:05 alert send by nightly job to Slack

• 06 Dec 2021 - 09: 50 reran job to see if it was a temporary network error. Adewale Adekoya sent an email to GMC asking if the problem could be looked into

• 06 Dec 2021 - Began investigating with GMC.

• 07 Dec 2021 - 00:05 GMC-Sync succeeded

• 07 Dec 2021 - 09:20 Re-attempted changing connetions but this failed

• 07 Dec 2021 - Issue narrowed down to an IP address but this hasn’t resolved the issue

• 07 Dec 2021 - Further debugging info has been exchanged

• 08 Dec 2021 - A further fix did not resolve the issue, and

• 09 Dec 2021 - Sync job ran successfully however no updates from GMC on any changes made on their side

• 09 Dec 2021 - Adewale Adekoya further enquires for any updates from the GMC but no response received

Root Cause(s)

• Alerting from Slack (GMC Sync) & on MS Teams (Connections failure)

• Logs show failed Web Service requests

Action Items

Lessons Learned