Versions Compared

Version Old Version 6 New Version Current
Changes made by

Joseph (Pepe) Kelly

Andy Nash (Unlicensed)

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Resolved: Investigating

Date

Authors

Marcello Fabbri (Unlicensed) Liban Hirey (Unlicensed)

Status

Done

Summary

When a new RabbitMQ broker was created in AWS, we forgot to migrate omitted migrating the ESR DataExport service to it (all other services were migrated fine). When the old broker was deleted it triggered errors on the DataExport service.

Impact

No Users reported any problems during the incident.

...

  • We looked at a message found to have gone through that queue during the period the DataExportService was not the new Rabbit broker (a placement update, at 2:31 AM on 30/07/2021) and retrieved its trail through its correlationId: metabase link.

  • We checked whether that record had been exported by the DataExporter or not and it seems like it did, presumably proving data has been exported regardless of this issue (probably stored/pent up somewhere in the ESR interface and then finally flowing through once the DataExportService got put on the new Rabbit broker).

This list of GeneratedAppRecords appears to have been successfully exported by the EsrDataExportService, as seen on the PendingExport table. However some Placements have not updated their esr status on TIS and will require to be manually corrected.

Timeline

: ??:?? 15:00 - Other services migrated to new RabbitMQ broker (STAGE)

: 11:25 - Other services migrated to new RabbitMQ broker (PROD)

: 10:05 - Old RabbitMQ broker deleted from AWS by Liban

...