Non-technical Description

GMC sync failed and subsequently failed to retrieve any doctors

Trigger

Unknown change by the GMC. There was initially a change to the firewall. The continuing problem, with a different way of failing suggests that there was another GMC which also caused an issue.

Detection

Slack notification received

Open

on inspection of the logs we found this:

Open

Resolution

Contacted GMC as the issue was on their service.

GMC have not elaborated on the cause or resolution

GMC got back to HEE on 19 January 22 by an email. They applied a security patch in their BE environment. The webserver serving the API was’t included in the first patch, so it was not tested.

They have now included it in the testing plan to ensure that this is not repeated again.

Timeline

• Dec 6, 2021 - 00:05 alert send by nightly job to Slack

• Dec 6, 2021 - 09: 50 reran job to see if it was a temporary network error. @Adewale Adekoya sent an email to GMC asking if the problem could be looked into

• Dec 6, 2021 - Began investigating with GMC.

• Dec 7, 2021 - 00:05 GMC-Sync succeeded

• Dec 7, 2021 - 09:20 Re-attempted changing connetions but this failed

• Dec 7, 2021 - Issue narrowed down to an IP address but this hasn’t resolved the issue

• Dec 7, 2021 - Further debugging info has been exchanged

• Dec 8, 2021 - A further fix did not resolve the issue, and

• Dec 9, 2021 - Sync job ran successfully however no updates from GMC on any changes made on their side

• Dec 9, 2021 - @Adewale Adekoya further enquires for any updates from the GMC but no response received

• Jan 19,2022 - @Adewale Adekoya has received an email reply from GMC

Root Cause(s)

• Alerting from Slack (GMC Sync) & on MS Teams (Connections failure)

• Logs show failed Web Service requests

Action Items

Lessons Learned

GMC response to our query took a longer time than expected. We need have mulitple contacts in GMC that can handle technical queries.