Non-technical Description

ESR sent through a number of FULL FILES on 1st of March that did not load or fully load/reconcile to then send applicants against subsequently for St Helens & Knowsley Trust. This meant a delay in sending across Applicant Files to ESR.

ESR told us previously that the maximum number of full files they would send us was 3 per week. On Monday 1st of March, they sent through 9 files, 8 distinct with 1 duplicate.

Trigger

Unexpected number of FULL FILES (RMF) causing an overload on the TIS-ESR interface.

DE_EMD_RMF_20210301_00002598.DAT
DE_KSS_RMF_20210301_00002788.DAT
DE_EOE_RMF_20210301_00002798.DAT
DE_MER_RMF_20210301_00002766.DAT
DE_OXF_RMF_20210301_00002759.DAT
DE_NWN_RMF_20210301_00002766.DAT
DE_LDN_RMF_20210301_00003186.DAT
DE_WMD_RMF_20210301_00002906.DAT
DE_WMD_RMF_20210302_00002907.DAT

Detection

• An issue raised on Teams regarding missing applicants from Liam Lofthouse (NWM Data Lead)

• 

Resolution

Timeline

• 01 Mar 2021 - 17:28-17:38 - 7 RMF files received (EMD, KSS, EOE, MER, OXF, NWN, LDN)

• 01 Mar 2021 - 17:45 - MongoDB down

• 01 Mar 2021 - 18:09 - MongoDB manually restarted

• 01 Mar 2021 - 18:10 - MongoDB up

• 01 Mar 2021 - 19:10 - MongoDB down

• 01 Mar 2021 - 19:20 - MongoDB up

• 01 Mar 2021 - 21:36 - MongoDB down

• 01 Mar 2021 - 22:31 - MongoDB up

• 02 Mar 2021 - 15:19 - Live defect https://hee-tis.atlassian.net/browse/TIS21-1265 created

• 02 Mar 2021 - 16:57 - WMD RMF received

• 02 Mar 2021 - 18:30 - WMD RMF received

• 03 Mar 2021 - 15:21 - Query raised on teams about unreceived data

Root Cause(s)

Action Items

Lessons Learned