Non-technical Description

When a placement is deleted in TIS, a message is sent to TSS to make a corresponding delete of its own record of that data. When deleted, the trainees profile would be updated to exclude that placement, and any placement onboarding notifications scheduled in the future would be removed.

In some instances, TSS is failing to correctly delete the placement, or is deleting and immediately reinserting it, resulting in trainee profiles featuring, and notifications being sent for, these non-existent placements.

Trigger

Almost concurrent deletion events for placements and placement specialties are not always being handled in the correct sequence, resulting in the reinsertion of just-deleted placement records.

Detection

Issue reported by LOs 13 Nov 2024.

• Initial report: https://hee-nhs-tis.slack.com/archives/C03G3UK3U82/p1731513213318419

5 Whys (or other analysis of Root Cause)

• The example given is for placement 2568353 for trainee 319330. We sent a ‘zombie’ 12-week placement notification for this:
  2024-11-13T01:24:45.607Z INFO 1 --- [eduler_Worker-1] u.n.t.t.n.service.NotificationService : Sent DEFAULT.PLACEMENT_UPDATED_WEEK_12-2568353 notification for 2568353 (In Post, starting 2025-02-05) to mubarakdanyaro@gmail.com using template v1.0.0

• We still have that placement in our sync MongoDb, though it no longer exists in TIS.

• We received a delete instruction in tis-trainee-sync for the placement specialty and then the placement at roughly2024-10-25T09:31:11.3 . After the placement specialty delete, the service tried to resync the placement, apparently retrieving it at 2024-10-25T09:31:11.369Z before it was deleted from the db/cache, and then reinserting it.

  The key tis-trainee-sync log lines show this (full logs attached):

  2024-10-25T09:31:11.368Z DEBUG 1 --- [ontainer#2-1234] .t.t.s.e.PlacementSpecialtyEventListener : After placement specialty delete, search for placement 2568353 to re-sync.
  
  2024-10-25T09:31:11.369Z DEBUG 1 --- [ontainer#2-1234] .t.t.s.e.PlacementSpecialtyEventListener : Placement Record(tisId=2568353, data={placementAddedDate=2024-05-24T08:50:40Z, lifecycleState=APPROVED, gradeId=306, siteCode=J81052, postId=233420, dateFrom=2025-02-05, placementWholeTimeEquivalent=1, traineeId=319330, placementType=In Post, dateTo=2025-08-05, siteId=16489, id=2568353}, metadata={}, type=null, operation=null, schema=tcs, table=Placement) found, queuing for re-sync.

  Code lines from: https://github.com/Health-Education-England/tis-trainee-sync/blob/958dd07fb543a2a9c304b4c93a039a399e582f4c/src/main/java/uk/nhs/hee/tis/trainee/sync/event/PlacementSpecialtyEventListener.java#L123

  Even though the subsequent request for the placement specialty from TIS will not succeed, this does not affect the ‘enrichment’ status of the re-inserted placement record, which will then be synced to the details service. We are unable to verify at this point that the request ended up in the corresponding DLQ (tis-trainee-sync-prod-request-interim-dlq.fifo) as the retention period is only 4 days.

• The deleted and resurrected placement seems to have been deleted and then reinserted into the trainee profile as expected. Logs as follows:

  2024-10-25T09:31:11.444Z  INFO 1 --- [io-8203-exec-79] u.n.h.t.details.api.PlacementResource    : Delete placement with TIS ID 2568353 of trainee with TIS ID 319330
  
  2024-10-25T09:31:11.466Z  INFO 1 --- [io-8203-exec-95] u.n.h.t.details.api.PlacementResource    : Update placement of trainee with TIS ID 319330 with PlacementDto PlacementDto(tisId=2568353, startDate=2025-02-05, endDate=2025-08-05, site=SiteDto(name=LIFEBOAT QUAY MEDICAL CENTRE, knownAs=LIFEBOAT QUAY MEDICAL CENTRE (J81052), location=UNIT A, LIFEBOAT QUAY, POOLE ), otherSites=[], grade=ST1, specialty=null, subSpecialty=null, postAllowsSubspecialty=false, otherSpecialties=[], placementType=In Post, employingBody=University Hospitals Dorset NHS Foundation Trust, trainingBody=NHS DORSET ICB - 11J, wholeTimeEquivalent=1, status=null, signature=null)

• The first delete instruction was also actioned by the notification service:

  2024-10-25T09:31:11.529Z  INFO 1 --- [ntContainer#6-6] u.n.t.t.n.event.PlacementListener        : Handling placement delete event PlacementEvent[tisId=2568353, recrd=RecordDto(data={placementAddedDate=2024-05-24T08:50:40Z, lifecycleState=APPROVED, gradeId=306, siteCode=J81052, postId=233420, dateFrom=2025-02-05, placementWholeTimeEquivalent=1, traineeId=319330, placementType=In Post, dateTo=2025-08-05, siteId=16489, id=2568353}, metadata={})].
  
  2024-10-25T09:31:11.577Z  INFO 1 --- [ntContainer#6-6] u.n.t.t.n.service.NotificationService    : Removed any stale notification scheduled for PLACEMENT_UPDATED_WEEK_12-2568353
  
  2024-10-25T09:31:11.586Z  INFO 1 --- [ntContainer#6-6] u.n.t.t.n.service.NotificationService    : Removed any stale notification scheduled for PLACEMENT_ROLLOUT_2024_CORRECTION-2568353

  and then the reinserted record was in turn rescheduled a minute later, leading to the incorrect notification mentioned above:

  2024-10-25T09:32:11.509Z  INFO 1 --- [Container#5-216] u.n.t.t.n.event.PlacementListener        : Handling placement update event PlacementEvent[tisId=2568353, recrd=RecordDto(data={placementAddedDate=2024-05-24T08:50:40Z, lifecycleState=APPROVED, gradeId=306, siteCode=J81052, postId=233420, dateFrom=2025-02-05, placementWholeTimeEquivalent=1, traineeId=319330, placementType=In Post, dateTo=2025-08-05, siteId=16489, id=2568353, employingBodyName=University Hospitals Dorset NHS Foundation Trust, trainingBodyName=NHS DORSET ICB - 11J, owner=Wessex, postAllowsSubspecialty=false, site=LIFEBOAT QUAY MEDICAL CENTRE, siteLocation=UNIT A, LIFEBOAT QUAY, POOLE , siteKnownAs=LIFEBOAT QUAY MEDICAL CENTRE (J81052), otherSites=[], gradeAbbreviation=ST1, otherSpecialties=[]}, metadata={})].
  
  2024-10-25T09:32:11.569Z  INFO 1 --- [Container#5-216] u.n.t.t.n.service.NotificationService    : Notification for PLACEMENT_UPDATED_WEEK_12-2568353 scheduled for Wed Nov 13 00:00:00 UTC 2024
  

Stakeholder Demo

Resolution

Timeline

All times GMT unless otherwise indicated.

• 13 Nov 2024 15:05 Incorrect notifications reported by LOs. In addition, there has been a steady stream of requests for manual profile re-syncs which may well reflect these incorrect placements retained in TSS.

• 13 Nov 2024 15:53 Issue reported on trainee-team-general Slack channel.

• 14 Nov 2024 11:30 Initial investigation into specific example identifies potential issue.
  

Action Items

See also:

Lessons Learned