Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

All times in BST unless indicated

  • : 08:53 : User reported (on Teams) revalidation module is showing one person under notice

  • : 08:53 : User reported (on Teams) revalidation module is showing one person under notice

  • : 08:53 : User reported (on Teams) revalidation module is showing one person under notice

  • 09:30 : Stopped the 2-hourly checks of submitted recommendation, shortly after stopped the service temporarily to stop unhelpful logging

  • : 09:41 : Moved sync start messages to new queues for debugging

  • : 09:43 : Found logging to suggest incident started at 00:05 - around the time of the gmc sync job starting

  • : ~ 9:45 : Stopped gmc-client task on prod

  • : ~10:00 : Restarted gmc-client task on prod, observed the same issue, task stopped again.

  • : 10:15 : Changed Log level for gmc-client (set to debug) and pushed to preprod

  • 11:30 added JAVA_TOOL_OPTIONS in task definition, then updated memory from 512M to 2G.

Root Cause(s)

...

Action Items

...