Non-technical Description

Not able to connect to the GMC servers to get the revalidation data we require

Trigger

GMC Replaced an SSL/TLS certificate on their server last night and it hadn't been applied correctly or hadn't been tested correctly. therefore blocking anybody from making secure connections to the service.

Detection

Slack notification from Ade @ 08:39 on 23/07/2021, pointing out that the GMC-sync-prod job had frailed again

Resolution

Sent an email to GMC stating that we had performed tests from our servers and from personal devices and that the results were the same for all connections.

Sent:

Reply:

Timeline

23 Jul 2021: 01:05 - An alert in the #monitoring channel of ETL success but in the wrong time slot.

23 Jul 2021: 08:39 - Ade alerted dev team

23 Jul 2021: 08:51 - John started investigation

23 Jul 2021: 10: 53 - Ade emailed GMC about problem

23 Jul 2021: 11:15 - GMC responded that issue raised with their service desk

23 Jul 2021: 12:25 - GMC confirmed certificate change was made 22/07/2021 which may be the reason for the issue

Root Cause(s)

A replacement SSL/TLS Certificate had been applied incorrectly to the Server at the GMC where we connect to get the GMC Data into TIS.

Lessons Learned