Date | |
Authors | |
Status | |
Summary | When a new RabbitMQ broker was created in AWS, we forgot to migrate the ESR DataExport service to it (all other services were migrated fine). When the old broker was deleted it triggered errors on the DataExport service. |
Impact | No Users reported any problems during the incident. |
Non-technical Description
The ESR DataExport service was not migrated over in time therefore there was a possibility of having some application and notification data missing following the deletion of the old broker.
Trigger
Deletion of the old RabbitMQ broker in AWS.
Detection
Slack notification from Sentry @ 10:11 on 30/07/2021.
Resolution
Timeline
: 10:05 - Old RabbitMQ broker deleted from AWS by Liban
: 10:11 - An alert in the #sentry-esr channel
: 10:32 - Pepe alerted the dev team
: 10:56 - Liban switches DataExport service over to the new RabbitMQ broker after realising that it doesn’t exist in it
: 11:11 - Liban and Marcello start investigating to verify whether any data is missing
Root Cause(s)
Deletion of the old RabbitMQ broker in AWS.
0 Comments