...
Trainee profile data is copied into TIS Self-Service from TIS. This happens whenever data is added or updated in TIS. Our monitoring tool alerted us that some TIS programme membership data was repeatedly failing to be loaded successfully into TIS Self-Service. The issue arose on 28 Dec 2023, and was resolved on _____12 Jan 2023.
...
Trigger
(To confirm) The ID’s of some CurriculumMembership records had changed. We use the ID’s as a primary key. (The assumed consequence of this might be duplication in TSS, not a failure to sync the data successfully, so this needs to be clarified)
...
CloudWatch Alarm in the #monitoring Slack channel for messages in the tis-trainee-sync-prod-dlq (Since 2022-12-28 10:23:34)
...
Resolution
11:25 - Alerted users on TIS Self-Service Teams channel that we would be doing maintenance, and proceeded with the following:
Clear-down the curriculumMembership table in the sync database
Clear-down the tis-trainee-sync-prod-dlq
Reload the TCS CurriculumMembership data in the DMS
...
Timeline
All times in GMT unless indicated
...
Action Items
Action Items | Owner | |
|---|---|---|
Clear and resync curriculumMembership data | ||
...
Lessons Learned
We need to investigate Production DLQ messages more promptly, since they indicate data issues that will affect PGDiTs