Date	30 Jul 2021
Authors	Marcello Fabbri (Unlicensed) Liban Hirey (Unlicensed)
Status
Summary	When a new RabbitMQ broker was created in AWS, we forgot to migrate the ESR DataExport service to it (all other services were migrated fine). When the old broker was deleted it triggered errors on the DataExport service.
Impact	No Users reported any problems during the incident.

Non-technical Description

The ESR DataExport service was not migrated over in time therefore there was a possibility of having some application and notification data missing following the deletion of the old broker.

Trigger

Deletion of the old RabbitMQ broker in AWS.

Detection

Slack notification from Sentry @ 10:11 on 30/07/2021.

Resolution

Timeline

30 Jul 2021: 10:05 - Old RabbitMQ broker deleted from AWS by Liban

30 Jul 2021: 10:11 - An alert in the #sentry-esr channel

30 Jul 2021: 10:32 - Pepe alerted the dev team

30 Jul 2021: 10:56 - Liban switches DataExport service over to the new RabbitMQ broker after realising that it doesn’t exist in it

30 Jul 2021: 11:11 - Liban and Marcello start investigating to verify whether any data is missing

Root Cause(s)