...

GMC was inadvertently blocking our connection to their production environment.

What was the issue?

The API issue was coming from their Cloudflare infrastructure with the error indicating HEE was no longer one of the organisations permitted to access their API.

We contacted GMC, to highlight the issueWhen trying to connect to the GMC database for our nightly synchronisation ETL, our application was getting blocked by the firewall at the GMC, we contacted the GMC to highlight the issue after double-checking our infrastructure to make sure it wasn't anything we were responsible for. They confirmed their oversight, confirming that they had implemented some additional security on 20/07/2021 due to some nefarious activity that was hitting their service. That security update was a little too zealous and blocked our connection to the API. On 21/07/2021 at 11am, GMC added the HEE IP addresses to a ‘whitelist’ which resolved the problem of us connecting to them, but still keeps the unwanted traffic away.

...

Trigger

Additional security had been added to Cloudflare's firewall due to them being attacked the previous day. This blocked HEE access to their API.

...

As we’d noticed the issue via our monitoring, we couldarguably shouldhave alerted Reval Admins in Teams before they highlighted the problem. And then regularly updated progress/resolution (for this incident, the turnaround was very quick anyway, as it goes).