...
Date |
|
Authors | |
Status | |
Summary | TIS ESR ETL Applicant process Failed |
Impact | N/A |
Root Cause(s)
Trigger
Resolution
Detection
Action Items
Timeline
Root Cause(s)
The ESR service restarted in the middle of applicant load run
The exact cause of the restart is not known but looking at the logs, 500+ records were sent at one time to the ESR service to save the logs straight after suggests that something restarted the service.
Due to backslash in file. Line below
POS,7607373,,"C","34611829","Medical and Dental\","Specialty Registrar (Cornerways)","800","General Practice","WES/J82150/800/GPST/003","WES","RBD","NHS","MS02","Specialty Registrar ST2/ SpR2",""
Trigger
Processing bad RMT file from ESR
Resolution
Resolved error in file through manula deletion of \ and reran the applicant load job.
For records which hadnt exported (status READY_TO_EXPORT) we Modified effective date to today and changed the tisstaus to errored. We then reran the export job with no date override etc… (Previous attempts to rerun using a previous date and those fields had failed). It is not clear why this worked and of the 4 records which had an errored status, one was changed to exported and the 4 each had another applicant exported record created.
Rerun the job to pull down the files and load them into the system, they should then be picked up in tomorrow export run (Did not work as expected)
Detection
Alert in Slack
Query of ESR DB table
Action Items
Action Item | Owner | Issue |
|---|---|---|
Run Applicant load again in the evening | Paul | Done |
Validate behaviour of new code for backslash. | Sachin | |
Investigate why malformed csv error wasnt seen in sentry |
Timeline
Recieved file on wednesday 5th August from ESR
Applicant load failed afternoon of the 5th
Expected export to pick up non exported files on 6th, which didnt happen
on the 7th reran multiple times with resolution, described above.