Non-technical Description

Trainee profile data is copied into TIS Self-Service from TIS. This happens whenever data is added or updated in TIS. Our monitoring tool alerted us that some TIS programme membership data was repeatedly failing to be loaded successfully into TIS Self-Service. The issue arose on 28 Dec 2023, and was resolved on 12 Jan 2023.

Trigger

• The ID’s of some CurriculumMembership records had changed. We use the ID’s as a primary key. (The assumed consequence of this might be duplication in TSS, not a failure to sync the data successfully, so this needs to be clarified)

Detection

• CloudWatch Alarm in the #monitoring Slack channel for messages in the tis-trainee-sync-prod-dlq (Since 2022-12-28 10:23:34)

Resolution

• 12 Jan 2023 11:25 - Alerted users on TIS Self-Service Teams channel that we would be doing maintenance, and proceeded with the following:
  Clear-down the curriculumMembership table in the sync database
  Clear-down the tis-trainee-sync-prod-dlq
  Reload the TCS CurriculumMembership data in the DMS

• 12 Jan 2023
  

Timeline

All times in GMT unless indicated

• 29 Dec 2022: 09:00 - First CloudWatch Alarm in the #monitoring Slack channel for messages in the tis-trainee-sync-prod-dlq

• 10 Jan 2023: 12:44 - Investigation into the messages on the DLQ

Root Cause(s)

Action Items

Lessons Learned

• We need to investigate Production DLQ messages more promptly, since they indicate data issues that will affect PGDiTs