2022-04-01 Revalidation doctor overnight sync failed due to GMC connection block

Owned by Cai Willis
Last updated: 01 Apr 2022
1 min read

Date

Apr 1, 2022

Authors

@Cai Willis

Status

Documenting

Summary

Overnight sync job for recommendations failed

Impact

No doctors were available on recommendations page and thus recommendations was unusable for most of the day

Non-technical Description

GMC had blocked a lot of traffic due to suffering a “cyber attack”, and it looks like our applications were caught up in this (old and new, pre-production and production)

Trigger

  • GMC blocked all traffic from our applications

Detection

  • Visual inspection of the site in the morning

  • We had noticed 403 errors the day before but incorrectly assumed they were part of the previous incident

Resolution

  • Contacted GMC who restored our applications' access to their API

Timeline

  • Apr 1, 2022 08:00 It was noticed that no doctors were showing in reval recommendations

  • Apr 1, 2022 08:25 Peter McNair contacted by email

  • Apr 1, 2022 08:55 Attempt to contact GMC service desk over phone

  • Apr 1, 2022 10:07 Received response from GMC informing us of “cyber attack”

  • Apr 1, 2022 14:20 Re-running sync jobs is successful, issue appears resolved (no confirmation from GMC at this stage)

Root Cause(s)

  • GMC blocked access to our application

Action Items

Action Items

Owner

Action Items

Owner

 

Lessons Learned

  •  

Slack: https://hee-nhs-tis.slack.com/
Jira issues: https://hee-tis.atlassian.net/issues/?filter=14213