Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 3 Next »

Date

Authors

Reuben Roberts

Status

Implementing

Summary

Some TISSS trainee profiles have incomplete Programme Membership data

Impact

TISSS trainee profile data missing for some trainees

Non-technical Description

Trainee profile data is copied into TIS Self-Service from TIS. This happens whenever data is added or updated in TIS. Our monitoring tool alerted us that some TIS programme membership data was repeatedly failing to be loaded successfully into TIS Self-Service. The issue arose on 28 Dec 2023, and was resolved on 12 Jan 2023.


Trigger

  • The ID’s of some CurriculumMembership records had changed. We use the ID’s as a primary key. (The assumed consequence of this might be duplication in TSS, not a failure to sync the data successfully, so this needs to be clarified)


Detection

  • CloudWatch Alarm in the #monitoring Slack channel for messages in the tis-trainee-sync-prod-dlq (Since 2022-12-28 10:23:34)


Resolution

  • 11:25 - Alerted users on TIS Self-Service Teams channel that we would be doing maintenance, and proceeded with the following:
    Clear-down the curriculumMembership table in the sync database
    Clear-down the tis-trainee-sync-prod-dlq
    Reload the TCS CurriculumMembership data in the DMS



Timeline

All times in GMT unless indicated

  • : 09:00 - First CloudWatch Alarm in the #monitoring Slack channel for messages in the tis-trainee-sync-prod-dlq

  • : 12:44 - Investigation into the messages on the DLQ

Root Cause(s)


Action Items

Action Items

Owner

Clear and resync curriculumMembership data

Reuben Roberts

DONE

TIS21-4064 - Getting issue details... STATUS

John Simmons (Deactivated)


Lessons Learned

  • We need to investigate Production DLQ messages more promptly, since they indicate data issues that will affect PGDiTs

  • No labels