Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  • :

  • :

  • 08:53 : User reported (on Teams) revalidation module is showing one person under notice

  • 09:30 : Stopped the 2-hourly checks of submitted recommendation, shortly after stopped the service temporarily to stop unhelpful logging

  • 09:41 : Moved sync start messages to new queues for debugging

  • 09:43 : Found logging to suggest incident started at 00:05 - around the time of the gmc sync job starting

  • ~ 9:45 : Stopped gmc-client task on prod

  • ~10:00 : Restarted gmc-client task on prod, observed the same issue, task stopped again.

  • 10:15 : Changed Log level for gmc-client (set to debug) and pushed to preprod

  • 11:30 added Added JAVA_TOOL_OPTIONS in task definition, then updated memory from 512M to 2G.

  • 12:10 Triggered GMC sync again on preprod.

Root Cause(s)

...

Action Items

...