2020-07-01 Applicant Confirmations from ESR not received

Jira Live Defect

TISNEW-4910 - Getting issue details... STATUS

Root Causes

• Job failure meant not all applicant confirmations were processed and others were duplicated a number of times
• Failure due to data in the file (2 files, each containing multiple instances of 1 badly formatted applicant record)
• We send ESR data they consider invalid (Invalid/badly formatted post codes)
• Date specificity of jobs means the system is fragile to errors
• The jobs lack 'bulkheads' to prevent the 'whole process failure'

Trigger

• Postcodes "Doha, Qatar" and "YO12 5EX,"

Resolution

• The preference would have been to remove the corrupted files and let ESR know so they could send corrected files.

Instead because of the urgent issue sending applicants to ESR;

• Manually changed the corrupted lines.
• Moved files to 2nd July folder and re-ran job.
• Removed files from 2nd July folder to prevent being re-processed by the scheduled job.

Detection / Timeline

• 1st 20:01 to 21:07 - Prompted by Slack notification, several attempted re-runs of the failed job
• 2nd 11:09 - ESR Service was restarted
• 2nd 12;00 to 13:47 - Discussed whether to exclude problem files or modify files from ESR to fix the format.

Action Items

• Improved the ability to pickup/process files for a different day
• Awaiting confirmation from TIS users (& Trusts)?
• Admins to correct data where they can
• Talk to ESR hub about improving the format of error messages in files.

What went well

• Found the problem lines in the files relatively quickly with little manual work, thanks to new code developed
• Limited to confirmations file rather than data needed for Trusts/Doctors

What went wrong

• The use of this information isn't widely understood; there could be a wider impact.
• We attempted to re-run it several times before identifying the cause; the way data is stored means there is some duplicate data.
• It is unclear whether there is a process to pick up those applicants which ESR rejected.