...
Notifications are usually sent to ESR every day, describing changes and updates to the “now & next” people in positions. The service did not generate the files for 11th - 19th Aug
ESR and TIS data would have been exchanged, so this was not impacted.
No warnings received as RabbitMQ not doing
...
because it was unable to complete the necessary queries and updates to the database within the allocated transaction time. Throughout this time, Applicant records were sent and confirmations received as normal.
Any notifications which should have gone out during this time and were still valid on the 20th were sent. For example the “now &next”(TM Pepe) notifications for job changeovers between 11th and 19th were sent on the 20th.
...
Trigger
We are confident without reproducing this that the quantity of notifications amongst other operational use triggered the combination of factors leading to this unplanned delay.
...
Detection
We noticed a lack of “Confirmation” messages files from ESR.
...
Resolution
Upsized the resources to enable to exporting jobs to run.
...
Action Items | Owner | Comments | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
Maintain increased (maximum) tier. Offset additional cost by scheduling cluster availability and size |
| |||||||||
| ||||||||||
See also:
...
Lessons
...