Date |
| ||||||||
Authors | |||||||||
Status | Documenting | ||||||||
Summary | The absence of notifications was clocked and on investigation it was found that none had been sent from 10 August | ||||||||
Impact |
|
| Table of Contents |
|---|
Non-technical Description
...
The service did not generate the files for 11th - ?? .19th Aug
ESR and TIS data would have been exchanged, so this was not impacted.
No warnings received as RabbitMQ not doing
...
Trigger
...
Detection
We noticed a lack of “Confirmation” messages from ESR
...
Resolution
Upsized the resources to enable to exporting jobs to run.
...
Timeline
Shovel setup but was persistent rather than temporary.
notifications not exported since 14:10 Last successful notification file generation.
14:00 Repeated failures prevent .
issue noticed We noticed the delay to notification confirmations beyond ‘normal’ delays
11:30 resized cluster, and it took 12mins
11:50 deleted the shovel which was moving every error to
esr.dlq.all.shovel.31072024.sending errors to a temporary queue12:10 and around indication on Metabase that the files for notifications have been created.
15:27 - Received confirmation files
5 Whys (or other analysis of Root Cause)
...
)
We didn’t receive DCC conformation files because we didn’t send any files for ESR to confirm receipt of.
The attempts to build notifcation files failed (errors were logged as warnings but not reported via Sentry)
Database transactions timed out.
The database didn’t have the resources to complete the transaction within the time limit.
...
Action Items
Action Items | Owner | Comments |
|---|---|---|
See also:
...
Lessons Learnt
RabbitMQ awareness.