...

• Root Cause(s)

• Trigger

• Resolution

• Detection

• Action Items

• Timeline

...

• The ESR service restarted in the middle of an ETL applicant load run

• The exact cause of the restart is not known but looking at the logs, 500+ records was were sent at one time to the ESR service to save the logs straight after suggests that something restarted the service. This could have been Docker if it thought the service was unresponsive (due to the load) or it might have been a memory issue (but there isn’t a heap dump that would suggest that)

Trigger

• A scheduled job

...

• Due to backslash in file. Line below

  • POS,7607373,,"C","34611829","Medical and Dental\","Specialty Registrar (Cornerways)","800","General Practice","WES/J82150/800/GPST/003","WES","RBD","NHS","MS02","Specialty Registrar ST2/ SpR2",""

Trigger

• Processing bad RMT file from ESR

Resolution

• Resolved error in file through manula deletion of \ and reran the applicant load job.

• For records which hadnt exported (status READY_TO_EXPORT) we Modified effective date to today and changed the tisstaus to errored. We then reran the export job with no date override etc… (Previous attempts to rerun using a previous date and those fields had failed). It is not clear why this worked and of the 4 records which had an errored status, one was changed to exported and the 4 each had another applicant exported record created.

• Rerun the job to pull down the files and load them into the system, they should then be picked up in tomorrow export run (Did not work as expected)

Detection

• Alert in Slack

• Query of ESR DB table

Action Items

Timeline

• Recieved file on wednesday 5th August from ESR

• Applicant load failed afternoon of the 5th

• Expected export to pick up non exported files on 6th, which didnt happen

• on the 7th reran multiple times with resolution, described above.