Non-technical Description

ESR sent through a number of FULL FILES on 1st of March that did not load or fully load/reconcile . There was a false assumption that they didn’t fully load or “reconcile” to then send applicants against subsequently for St Helens & Knowsley Trust (StH&K). The data had all been processed but the VPD (Virtual Private Directory) identifier for StH&K was sent as 96 rather than the expected “096”. This meant a delay in sending across Applicant Files to ESR.that the Medical Rotation (MEDROT) spreadsheet was not correct and there is the significant risk that trainee pay has not been updated to reflect those starting/finishing placements.

This Live defect was originally raised on 1st March as a response to the overwhelming number of files ESR sent to us. ESR told us previously that the maximum number of full files they would could send us was 3 per week. On Monday w/c 1st of March, they sent through 9 files , for 8 distinct with 1 duplicate“Deaneries”.

...

Trigger

Unexpected number of FULL FILES (RMF) causing an overload on the TIS-ESR interface.

DE_EMD_RMF_20210301_00002598.DAT
DE_KSS_RMF_20210301_00002788.DAT
DE_EOE_RMF_20210301_00002798.DAT
DE_MER_RMF_20210301_00002766.DAT
DE_OXF_RMF_20210301_00002759.DAT
DE_NWN_RMF_20210301_00002766.DAT
DE_LDN_RMF_20210301_00003186.DAT
DE_WMD_RMF_20210301_00002906.DAT
DE_WMD_RMF_20210302_00002907.DAT

...

Detection

• Alerting in our monitoring channel

• An issue raised on Teams regarding missing applicants from Liam Lofthouse (NWM Data Lead)

• 

...

Resolution

• Filter the notification files for those that St. Helens needed (VPD: 96) and modify the VPD to be 3 digits

• Modified the part of the system that writes to CSV to ensure the VPD is at least 3 digits.

...

Timeline

• 01 Mar 2021 - 17:28-17:38 - 7 RMF files received (EMD, KSS, EOE, MER, OXF, NWN, LDN)

• 01 Mar 2021 - 17:45 - MongoDB down

• 01 Mar 2021 - 18:09 - MongoDB manually restarted

• 01 Mar 2021 - 18:10 - MongoDB up

• 01 Mar 2021 - 19:10 - MongoDB down

• 01 Mar 2021 - 19:20 - MongoDB up

• 01 Mar 2021 - 21:36 - MongoDB down

• 01 Mar 2021 - 22:31 - MongoDB up

• 02 Mar 2021 - 15:19 - Live defect https://hee-tis.atlassian.net/browse/

  Jira Legacy
  server System JIRA serverId 4c843cd5-e5a9-329d-ae88-66091fcfe3c7 key TIS21-1265

  created

• 02 Mar 2021 - 16:57 - WMD RMF received

• 02 Mar 2021 - 18:30 - WMD RMF received

• 03 Mar 2021 - 15:21 - Query raised on teams about unreceived data

• 04 Mar 2021 - 13:28-16:52 - One-off processes to generate and send files with alterred “VPD”.

• 05 Mar 2021 - 11:00 - Daily call identifying that the MEDROT has not been updated.

• 08 Mar 2021 - 11:00 - Daily call identifying that the MEDROT has now been updated.

• 09 Mar 2021 - 11:00 - Daily call identifying that West Midlands (WMD) positions are not showing.

• 10 Mar 2021 - 11:00 - Daily call where we raised that (WMD) positions hadn’t been received by TIS.

...

Root Cause(s)

• We sized Mongo on the basis we wouldn’t get many full files.

• There isn’t anything to signal that Mongo is struggling before a fatal VM failure.

• The VM doesn’t stop and start in a reasonable time.

• The replica set isn’t as ‘highly available’ as it should be (not on separate VMs)

• For the MEDROT we were sending values for their “Virtual Private Directory” “96” rather than the “096” they wanted

• ESR treats the number 96 as being different to 096

• The TIS integration followed ESRs specification that this was a “variable length number”, up to 3 digits long.

...

Action Items

...

Lessons Learned

• Time to crack on with Tech Improvement work (Switch to a DBaaS)