Date | |
Authors | |
Status | Investigating |
Summary | Placements added by ESR late on 30/07/2021 for 04/08/2021 starters had a |
Impact | Some records might not be exported in time back to ESR, implying that some manual hiring will have to be undertaken instead. |
...
: some Placements are being added in the late afternoon for August 4th starters.
: GeneratedAppRecords related to those placements are not getting exported as expected.
: 12:10 PM BST: MS Teams message notifying these GeneratedAppRecords are still marked as
TO_EXPORT: 15:40 PM BST: Some messages containing the “command” message to export GeneratedAppRecords for MER are being re-sent on the appropriate queue in the ESR interface to retrigger their export.
: 16:53 PM BST: Messages containing the “command” message to export GeneratedAppRecords for EOE, LDN, OXF and WES are being re-sent on the appropriate queue in the ESR interface to retrigger their export.
: 17:06 PM BST: Users are notified via MS Teams that the export of these files has been completed.
– meanwhile other GeneratedAppRecords qualifying for file generation were not being picked up –
: 16:45 PM BST: Manual trigger of the generation of files for the SEV deanery was successful.
: 09:33 AM BST: Manual trigger of the generation of files for the rest of the deaneries was successful.
: Retry policy for messages re-enabled
: Generation of export files (both app and notification) dated 01/July/2021 has been manually triggered, in an attempt to export these data to ESR.
Root Cause(s)
Still investigating why those placements have not been picked up on 31/07/2021.
...
There’s no conflict when a command is sent manually, so it’s possible to trigger the generation of those files manually if necessary to make up for the ESR interface’s shortcomings in the meantime.
...
that qualified for export has been triggered manually. The reason is likely to be rooted in the fact that manual “commands” on the queue involves a developer sending the appropriate “command” messages one by one, whereas the cron job sends them all at the same time, and end up conflicting. In particular, the conflicts occurred when the processing of the message involved the update of the Counter collection on the EsrDataExportService database. The conflicts where normally resolved by allowing the messages to be replayed, however a change in the retry policy no longer allowed for it, which resulted in the conflicts remaining unresolved.
Loss of the retry policy that was in place probably occurred while migrating to AWS. Now that’s been enabled again, the ESR interface operates as it used to.
Lessons Learned
Check policies are still retained when migrating services to different platforms.