Non-technical Description

ESR sent through a number of FULL FILES on 1st of March that did not load or fully load/reconcile to then send applicants against subsequently for St Helens & Knowsley Trust. This meant a delay in sending across Applicant Files to ESR.

ESR told us previously that the maximum number of full files they would send us was 3 per week. On Monday 1st of March, they sent through 9 files, 8 distinct with 1 duplicate.

Trigger

Unexpected number of FULL FILES (RMF) causing an overload on the TIS-ESR interface.

Detection

• An issue raised on Teams regarding missing applicants from Liam Lofthouse (NWM Data Lead)

• 

Resolution

Timeline

• 01 Mar 2021 - 17:28-17:38 - 7 RMF files received (EMD, KSS, EOE, MER, OXF, NWN, LDN)

• 01 Mar 2021 - 17:45 - MongoDB down

• 01 Mar 2021 - 18:09 - MongoDB manually restarted

• 01 Mar 2021 - 18:10 - MongoDB up

• 01 Mar 2021 - 19:10 - MongoDB down

• 01 Mar 2021 - 19:20 - MongoDB up

• 01 Mar 2021 - 21:36 - MongoDB down

• 01 Mar 2021 - 22:31 - MongoDB up

• 02 Mar 2021 - 15:19 - Live defect https://hee-tis.atlassian.net/browse/TIS21-1265 created

• 02 Mar 2021 - 16:57 - WMD RMF received

• 02 Mar 2021 - 18:30 - WMD RMF received

• 03 Mar 2021 - 15:21 - Query raised on teams about unreceived data

Root Cause(s)

Action Items

Lessons Learned