Date |
| ||||||||
Authors | |||||||||
Status | Documenting | ||||||||
Summary | The absence of notifications was clocked and on investigation it was found that none had been sent from 10 August | ||||||||
Impact |
|
| Table of Contents |
|---|
Non-technical Description
Notifications are usually sent to ESR every day, describing changes and updates to the “now & next” people in positions. The service did not generate the files for 11th - ?? .
Trigger
19th Aug because it was unable to complete the necessary queries and updates to the database within the allocated transaction time. Throughout this time, Applicant records were sent and confirmations received as normal.
Any notifications which should have gone out during this time and were still valid on the 20th were sent. For example the “now &next”(TM Pepe) notifications for job changeovers between 11th and 19th were sent on the 20th.
...
Trigger
We are confident without reproducing this that the quantity of notifications amongst other operational use triggered the combination of factors leading to this unplanned delay.
...
Detection
We noticed a lack of “Confirmation” messages files from ESR.
...
Resolution
Upsized the resources to enable to exporting jobs to run.
...
Timeline
Shovel setup but was persistent rather than temporary.
notifications not exported since 14:10 Last successful notification file generation.
14:00 Repeated failures prevent .
issue noticed We noticed the delay to notification confirmations beyond ‘normal’ delays
11:30 resized cluster, and it took 12mins
11:50 deleted the shovel which was moving every error to
esr.dlq.all.shovel.31072024.sending errors to a temporary queue12:10 and around indication on Metabase that the files for notifications have been created.
15:27 - Received confirmation files
5 Whys (or other analysis of Root Cause)
We didn’t receive DCC conformation files because we didn’t send any files for ESR to confirm receipt of.
The attempts to build notifcation files failed (errors were logged as warnings but not reported via Sentry)
...
…
Database transactions timed out.
The database didn’t have the resources to complete the transaction within the time limit.
...
Action Items
Action Items | Owner | Comments | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
Maintain increased (maximum) tier. Offset additional cost by scheduling cluster availability and size |
| |||||||||
| ||||||||||
See also:
...
Lessons
...