Non-technical Description

GMC was inadvertently blocking our connection to their production environment.

What was the issue?

The API issue was coming from their Cloudflare infrastructure with the error indicating HEE was no longer one of the organisations permitted to access their API.

We contacted GMC, to highlight the issue. They confirmed their oversight, confirming that they had implemented some additional security on 20/07/2021 due to some nefarious activity that was hitting their service. That security update was a little too zealous and blocked our connection to the API. On 21/07/2021 at 11am, GMC added HEE to a ‘whitelist’ which resolved the problem.

Trigger

Additional security had been added to Cloudflare's firewall due to them being attacked the previous day. This blocked HEE access to their API.

Detection

Alert in monitoring channel:

Resolution

Comms with GMC who stated:

We then re-ran the GMC sync and the associated ETL’s and all responded well.

Timeline

21 Jul 2021: 09:40 - Joseph Kelly noticed issue in monitoring channel

21 Jul 2021: 10:26 - Katy raised issue on Teams

21 Jul 2021: 10:26 - John raised issue on Slack

21 Jul 2021: 11:33 - Ade raised issue with GMC

21 Jul 2021: 11:55 - GMC requested for more details

21 Jul 2021: 12:35 - Ade supplied more details

21 Jul 2021: 14:16 - GMC emailed issue resolved

Root Cause(s)

Firewall update at GMC that was a little restrictive and therefore blocked a lot of connections, including ours.

Lessons Learned

No Lessons learned as the problem was completely at the GMC end, and there isn't anything we could have put in place to mitigate this.

As we’d noticed the issue via our monitoring, we could have alerted Reval Admins in Teams before they highlighted the problem.