Date | |
Authors | |
Status | Investigating |
Summary | Placements added by ESR late on 30/07/2021 for 04/08/2021 starters had a |
Impact | Some records might not be exported in time back to ESR, implying that some manual hiring will have to be undertaken instead. |
Non-technical Description
AppRecords meant to be exported on 31/07/2021 haven’t been exported back to ESR.
Trigger
Placements added on 30/07/2021 not going through the ESR interface as expected on 31/07/2021.
Detection
MS Teams: message on the ESR Support channel.
Resolution
GeneratedAppRecords with a start date after 03/08/2021 have been found and an appfilegenerationcommand file has been manually placed on the queue leading to the EsrDataExporter for each deanery that appeared to have non-exported GeneratedAppRecords.
Timeline
: some Placements are being added in the late afternoon for August 4th starters.
: GeneratedAppRecords related to those placements are not getting exported as expected.
: 12:10 PM BST: MS Teams message notifying these GeneratedAppRecords are still marked as
TO_EXPORT: 15:40 PM BST: Some messages containing the “command” message to export GeneratedAppRecords for MER are being re-sent on the appropriate queue in the ESR interface to retrigger their export.
: 16:53 PM BST: Messages containing the “command” message to export GeneratedAppRecords for EOE, LDN, OXF and WES are being re-sent on the appropriate queue in the ESR interface to retrigger their export.
: 17:06 PM BST: Users are notified via MS Teams that the export of these files has been completed.
– meanwhile other GeneratedAppRecords qualifying for file generation were not being picked up –
: 16:45 PM BST: Manual trigger of the generation of files for the SEV deanery was successful.
: 09:33 AM BST: Manual trigger of the generation of files for the rest of the deaneries was successful.
Root Cause(s)
Still investigating why those placements have not been picked up on 31/07/2021.
The issue has probably affected all deaneries, and has been ongoing for days, since there were several GeneratedAppRecords stuck in TO_EXPORT status that were not being picked up for the generation of exportable files, even though they matched the criteria for export.
When the cron job sends (every day at 13:00 UTC) the “command” (RabbitMQ message on esr.queue.appfilegenerationcommand.create.dataexporter) to generate the exportable app files, it fails because of a mongo based WriteConflict (com.mongodb.MongoCommandException: Command failed with error 112 (WriteConflict)).
There’s no conflict when a command is sent manually, so it’s possible to trigger the generation of those files manually if necessary to make up for the ESR interface’s shortcomings in the meantime.
Add Comment