...
Slack notification from Sentry @ 10:11 on 30/07/2021.
...
Resolution
The DataExportService not being on the new Rabbit broker for longer than a day made us presume that evidence of missing data could be found by investigating the esr.queue.apprecord.created.dataexporter queue, where messages would be likely to stall unconsumed by any service (until the deletion of the old broker). The Audit database though would still have these messages, so we’d be able to see if anything went through that queue during that period of time, and if that flow of data was successful or not.
We looked at a message found to have gone through that queue during the period the DataExportService was not the new Rabbit broker (a placement update, at 2:31 AM on 30/07/2021) and retrieved its trail through its correlationId: metabase link.
We checked whether that record had been exported by the DataExporter or not and it seems like it did, presumably proving data has been exported regardless of this issue (probably stored/pent up somewhere in the ESR interface and then finally flowing through once the DataExportService got put on the new Rabbit broker).
Timeline
: 10:05 - Old RabbitMQ broker deleted from AWS by Liban
...