Non-technical Description

The overnight sync procedure for TIS was unable to run. This meant only some trainees were being shown on the person search page.

Trigger

Detection

• Detected when slack notifications in the #monitoring-prod channel failed to appear

Resolution

• Created a new cluster to use based on the terraform description

• The sync jobs run fine when manually triggered however this is a temporary solution

Timeline

• 12 Feb 2021 - Question raised by user on Teams

• 12 Feb 2021 - 06:29 - Quickest fix (re-running the job) didn’t work on STAGE

• 12 Feb 2021 - We rebuilt the infrastructure similarly to NIMDTA.

• 12 Feb 2021 - 07:51-07:58 - Ran the Sync job again; trainees becoming visible during this time.

• 12 Feb 2021 - 08:00 - confirmed with users fixed

Root Cause(s)

Action Items

Lessons Learned

• Te