• Root Cause(s)

• Trigger

• Resolution

• Detection

• Action Items

• Timeline

Root Cause(s)

• Job ran in parallel, one on each of the servers.

• The ‘locking’ to prevent this only takes account of scheduled runs

• Container restarted in the ~10 minute window where there would be a problem

• (any idea why it restarted? I’m assuming all the jobs were overlapping during that time, but any data changes were overwriting the same records in the same way, only the elasticsearch job creates new records in the elastic search index…?

Trigger

• Teams notification

• 

  Slack notification

  

Resolution

• TIS Team manually re-ran the Person Sync job from the Sync administration panel: https://apps.tis.nhs.uk/sync

Detection

• A user reported in Teams Support Channel. The issue was also raised in the TIS tis-dev-team Slack channel.

• The overlapping jobs could be viewed in the server logs
  

  

  
  and also in the monitoring-prod Slack channel (started 1:29 AM and 1:33 AM):

  

Timeline

• 04 Feb 2021 08:21 - Notification on Teams

• 04 Feb 2021 09:25 - Job run again

Action Items

Lessons Learned

• The ‘locking’ to prevent the job running in parallel only takes account of scheduled runs. Any container restarts or manually running the job can cause duplication if it overlaps with the job running on the other server instance.