Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Date

Authors

John Simmons (Deactivated) Adewale Adekoya Jayanta Saha

Status

Implementing

Summary

Legacy GMC sync failed and Connections showing an error

Impact

Data in Reval out of date and Admins unable to use Connections

Non-technical Description

GMC sync failed

...

Trigger

Unknown change on GMC firewall

...

Detection

Slack notification received

...

on inspection of the logs we found this:

...

...

Resolution

TODO: .

...

Timeline

  • - 00:05 alert send by nightly job to Slack

  • - 09: 50 reran job to see if it was a temporary network error. Adewale Adekoya sent an email to GMC asking if the problem could be looked into

  • - Began investigating with GMC.

  • - 00:05 GMC-Sync succeeded

  • - 09:20 Re-attempted changing connetions but this failed

  • - Issue narrowed down to an IP address but this hasn’t resolved the issue

  • - Further debugging info has been exchanged

...

Root Cause(s)

  • Alerting from Slack (GMC Sync) & on MS Teams (Connections failure)

  • Logs show failed Web Service requests

...

Action Items

Action Items

Owner

...

Lessons Learned