Versions Compared

Old Version 1

changes.mady.by.user Reuben Roberts

Saved on

compared with

New Version 2

changes.mady.by.user Reuben Roberts

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Issue reported by LOs 13 Nov 2024.

...

5 Whys (or other analysis of Root Cause)

  • The example given is for placement 2568353 for trainee 319330. We sent a ‘zombie’ 12-week placement notification for this:
    2024-11-13T01:24:45.607Z INFO 1 --- [eduler_Worker-1] u.n.t.t.n.service.NotificationService : Sent DEFAULT.PLACEMENT_UPDATED_WEEK_12-2568353 notification for 2568353 (In Post, starting 2025-02-05) to mubarakdanyaro@gmail.com using template v1.0.0

  • We still have that placement in our sync MongoDb, though it no longer exists in TIS.

  • We received a delete instruction in tis-trainee-sync for the placement specialty and then the placement at roughly2024-10-25T09:31:11.3 . After the placement specialty delete, the service tried to resync the placement, apparently retrieving it at 2024-10-25T09:31:11.369Z before it was deleted from the db/cache, and then reinserting it.

    The key tis-trainee-sync log lines show this (full logs attached):

    Code Block
    2024-10-25T09:31:11.368Z DEBUG 1 --- [ontainer#2-1234] .t.t.s.e.PlacementSpecialtyEventListener : After placement specialty delete, search for placement 2568353 to re-sync.

2024-10-25T09:31:11.369Z DEBUG 1 --- [ontainer#2-1234] .t.t.s.e.PlacementSpecialtyEventListener : Placement Record(tisId=2568353, data={placementAddedDate=2024-05-24T08:50:40Z, lifecycleState=APPROVED, gradeId=306, siteCode=J81052, postId=233420, dateFrom=2025-02-05, placementWholeTimeEquivalent=1, traineeId=319330, placementType=In Post, dateTo=2025-08-05, siteId=16489, id=2568353}, metadata={}, type=null, operation=null, schema=tcs, table=Placement) found, queuing for re-sync.

    Code lines from: https://github.com/Health-Education-England/tis-trainee-sync/blob/958dd07fb543a2a9c304b4c93a039a399e582f4c/src/main/java/uk/nhs/hee/tis/trainee/sync/event/PlacementSpecialtyEventListener.java#L123

    Even though the subsequent request for the placement specialty from TIS will not succeed, this does not affect the ‘enrichment’ status of the re-inserted placement record, which will then be synced to the details service. We are unable to verify at this point that the request ended up in the corresponding DLQ (tis-trainee-sync-prod-request-interim-dlq.fifo) as the retention period is only 4 days.

  • The deleted and resurrected placement seems to have been deleted and then reinserted into the trainee profile as expected. Logs as follows:

    Code Block
    2024-10-25T09:31:11.444Z  INFO 1 --- [io-8203-exec-79] u.n.h.t.details.api.PlacementResource    : Delete placement with TIS ID 2568353 of trainee with TIS ID 319330

2024-10-25T09:31:11.466Z  INFO 1 --- [io-8203-exec-95] u.n.h.t.details.api.PlacementResource    : Update placement of trainee with TIS ID 319330 with PlacementDto PlacementDto(tisId=2568353, startDate=2025-02-05, endDate=2025-08-05, site=SiteDto(name=LIFEBOAT QUAY MEDICAL CENTRE, knownAs=LIFEBOAT QUAY MEDICAL CENTRE (J81052), location=UNIT A, LIFEBOAT QUAY, POOLE ), otherSites=[], grade=ST1, specialty=null, subSpecialty=null, postAllowsSubspecialty=false, otherSpecialties=[], placementType=In Post, employingBody=University Hospitals Dorset NHS Foundation Trust, trainingBody=NHS DORSET ICB - 11J, wholeTimeEquivalent=1, status=null, signature=null)

  • The first delete instruction was also actioned by the notification service:

    Code Block
    2024-10-25T09:31:11.529Z  INFO 1 --- [ntContainer#6-6] u.n.t.t.n.event.PlacementListener        : Handling placement delete event PlacementEvent[tisId=2568353, recrd=RecordDto(data={placementAddedDate=2024-05-24T08:50:40Z, lifecycleState=APPROVED, gradeId=306, siteCode=J81052, postId=233420, dateFrom=2025-02-05, placementWholeTimeEquivalent=1, traineeId=319330, placementType=In Post, dateTo=2025-08-05, siteId=16489, id=2568353}, metadata={})].

2024-10-25T09:31:11.577Z  INFO 1 --- [ntContainer#6-6] u.n.t.t.n.service.NotificationService    : Removed any stale notification scheduled for PLACEMENT_UPDATED_WEEK_12-2568353

2024-10-25T09:31:11.586Z  INFO 1 --- [ntContainer#6-6] u.n.t.t.n.service.NotificationService    : Removed any stale notification scheduled for PLACEMENT_ROLLOUT_2024_CORRECTION-2568353

    and then the reinserted record was in turn rescheduled a minute later, leading to the incorrect notification mentioned above:

    Code Block
    2024-10-25T09:32:11.509Z  INFO 1 --- [Container#5-216] u.n.t.t.n.event.PlacementListener        : Handling placement update event PlacementEvent[tisId=2568353, recrd=RecordDto(data={placementAddedDate=2024-05-24T08:50:40Z, lifecycleState=APPROVED, gradeId=306, siteCode=J81052, postId=233420, dateFrom=2025-02-05, placementWholeTimeEquivalent=1, traineeId=319330, placementType=In Post, dateTo=2025-08-05, siteId=16489, id=2568353, employingBodyName=University Hospitals Dorset NHS Foundation Trust, trainingBodyName=NHS DORSET ICB - 11J, owner=Wessex, postAllowsSubspecialty=false, site=LIFEBOAT QUAY MEDICAL CENTRE, siteLocation=UNIT A, LIFEBOAT QUAY, POOLE , siteKnownAs=LIFEBOAT QUAY MEDICAL CENTRE (J81052), otherSites=[], gradeAbbreviation=ST1, otherSpecialties=[]}, metadata={})].

2024-10-25T09:32:11.569Z  INFO 1 --- [Container#5-216] u.n.t.t.n.service.NotificationService    : Notification for PLACEMENT_UPDATED_WEEK_12-2568353 scheduled for Wed Nov 13 00:00:00 UTC 2024

Stakeholder Demo

...

Resolution

...

Timeline

All times GMT unless otherwise indicated.

...