Non-technical Description

GMC was blocking our connection to their prod. environment somewhere.

 What was the issue?

The API issue was coming from their cloudflare infrastructure, a quick google of the error states: “Cloudflare Error 1020: Access Denied indicates that you’ve violated a firewall rule and your connection request has been blocked” This code (1020) was not on the list of GMC codes.

After contacting GMC, we got to know that GMC implemented some additional security on 20/07/2021 due to some nefarious activity that was hitting their service.  That mitigation blocked our connection to the API. On 21/07/2021 at 11am, GMC added a whitelist to that mitigation and that resolved the problem.

Trigger

Additional security had been added to Cloudflare's firewall due to them being attacked the previous day.

Detection

Alert in monitoring channel:

Resolution

Comms with GMC who stated:

We then re-ran the GMC sync and the associated ETL’s and all responded well.

Timeline

21/07/2021 at 09:40 Joseph Kelly noticed issue in monitoring channel

21/07/2021 at 10:26 Katy raised issue

21/07/2021 at 10:26 John raised issue on Slack

21/07/2021 at 11:33 Ade raised issue with GMC

21/07/2021 at 11:55 GMC requested for more details

21/07.2021 at 12: 35 Ade supplied more details

21/07/2021 at 14:16 GMC emailed issue resolved

Root Cause(s)

Firewall update at GMC that was a little restrictive and therefore blocked a lot of connections, including ours.

Lessons Learned

No Lessons learned as the problem was completely at the GMC end, and there isn't anything we could have put in place to mitigate this.