Date | |
Authors | |
Status | Documenting |
Summary | Database used for exchanging information with ESR failed |
Impact | Delayed updates to TIS & TSS with trainee and position information from ESR. If Friday’s file for Severn area contains information conflicting with Saturday/Sunday’s files, the older information would have been used. |
Non-technical Description
ESR had another period of failing to send files on the day they were generated, this meant a greater number of files, generated between Friday 29th July and Monday 1st August were all sent in a short space of time. This is usually handled by application but this time, the database stopped responding.
The services that store information failed and a number of files were not processed. The built in alerting notified the team and after verifying the status of a number of failed individual transactions, we resolved the immediate problem and resent the instructions to process the files listed below.
A miscommunication meant that the file for Severn which was expected on Friday was processed after the files expected on Saturday & Sunday. If files from Saturday/Sunday contain information for trainees and positions which were also in the file from Friday, the earlier updates will have been processed incorrectly as more recent updates.
Trigger
Exceptions reported via Slack
Detection
Sentry alerting
Resolution
Force stopped the database server and restarted it, then requested processing of a number of files
Timeline
BST unless otherwise stated
2022-08-01 16:11 ESR processing failed messages start appearing on Slack #monitoring-esr channel
2022-08-01 16:17 Notifications picked up the team
2022-08-01 16:30ish ESR processes on Prod blue and green stopped
2022-08-01 16:32ish Prod MongoDB server stopped
2022-08-01 18:24 Prod MongoDB server started
2022-08-01 20:43 All ESR processes restarted in defined order
2022-08-01 20:36-21:21 Failed and missed RMC files processed in order defined below
Root Cause(s)
Database instability
2
3
4
5
Selected log entries (note, this is an edited listing, it does not include all log entries):
TODO
Grafana performance metrics for the Mongo cluster:
2022-08-01 failure
2022-08-04 failure
Action Items
Action Items | Owner |
|---|---|
Look at list of CSV files that were received (as per esr.queue.csv.invalid and any others subsequent to stopping the ESR services that would have not been processed at all because the services were stopped) | |
Review RabbitMQ esr.dlq.all messages [to identify any issues (such as?)] | |
Generate some Neo4J queries to check if a given {message id} was processed despite being in the DLQ | |
Once turned-on MongoDB - check which files exported to ESR today, that they were processed before 16:11 failure | DONE Reuben Roberts |
Manually resend rabbit messages for reprocessing (excluding any found in neo4j that were processed despite being in the DLQ) | Joseph (Pepe) Kelly Whilst verifying successful replay of first found that it was already queued. Message IDs: 1db2f10c-2400-4a16-83bb-440d9962091e REPLAYED but UNNEEDED |
Restart ESR services (instructions for sequencing for this are here: https://hee-nhs-tis.slack.com/archives/C01C7V5GT43/p1612957581030400 ) | |
Consider: for the position, placement and post queues, it is possible that create+delete messages will be processed in the incorrect order. Is there a way to check this? | |
Work out how to retrigger file processing (as per list of CSV files to be reprocessed that will be found in S3 bucket and others that were not processed at all and were received after approx. 16:11) DE_SEV_RMC_20220730_00001157.DAT (16:10:36) DE_SEV_RMC_20220731_00001158.DAT (16:11:00) DE_WES_RMC_20220729_00003589.DAT DE_WES_RMC_20220730_00003590.DAT DE_WES_RMC_20220731_00003591.DAT (16:12:40) DE_WMD_RMC_20220729_00003421.DAT DE_WMD_RMC_20220730_00003422.DAT DE_WMD_RMC_20220731_00003423.DAT DE_YHD_RMC_20220729_00003512.DAT DE_YHD_RMC_20220730_00003513.DAT DE_YHD_RMC_20220731_00003514.DAT DE_SEV_RMC_20220729_00001156.DAT DE_EMD_RMC_20220801_00003116.DAT …checked through deneries to find the additional RMC files… DE_EOE_RMC_20220801_00003316.DAT | DONE Joseph (Pepe) Kelly |
Fix restart config for stage mongo & mongo_exporter services | |
Modify S3 File trigger to put file notification in a queue (rather than TIS21-146 - Getting issue details... STATUS ) |
0 Comments