Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Non-technical Description

Not able to connect to the GMC servers to get the revalidation data we require

...

Trigger

GMC Replaced an SSL/TLS certificate on their server last night and it hadn't been applied correctly or hadn't been tested correctly. therefore blocking anybody from making secure connections to the service.

...

Detection

Slack notification from Ade @ 08:39 on 23/07/2021, pointing out that the GMC-sync-prod job had frailed again

...

...

Resolution

Sent an email to GMC stating that we had performed tests from our servers and from personal devices and that the results were the same for all connections.

Sent:

...

Reply:

...

Timeline

: 01:05 - An alert in the #monitoring channel of ETL success but in the wrong time slot.

...

: 12:25 - GMC confirmed certificate change was made 22/07/2021 which may be the reason for the issue

Root Cause(s)

A replacement SSL/TLS Certificate had been applied incorrectly to the Server at the GMC where we connect to get the GMC Data into TIS.

Lessons Learned