Date |
| ||||||||
Authors | |||||||||
Status | Impacted | ||||||||
Summary |
| ||||||||
Impact | The recommendations search page was not being updated for a number of hours through the day |
...
BST unless otherwise stated
- Some things happened
02:26 to 08:07 - Queue to recommendation for ‘doctor view’ update built steadily to ~83K -
- Some things happened
08:21 - First report in user channel
12:07 - Picked up for investigation
12:07 to 14:00ish - Checked database & ElasticSearch index
- Some things happened 13:00 - Checked the return list of GMC for north west
- Some things happened 14:00ish - Found messages in reval.queue.masterdoctorview.updated.recommendation didn’t get consumed
- Some things happened 16:00ish - Force a new start of recommendation service
- Some things happened
.
...
Root Cause(s)
Doctors reported as not showing in the search list
ElasticSearch Index for Recommendation Service isn’t being updated
Large backlog of messages stuck on a queue for updating the index
Message Consumers disappeared but after the final
aws ecs update-service --force-new-deploymentdropped to one before going back up to 3?
...
Action Items
Action Items | Comments | Owner |
|---|---|---|
Monitoring for queue depth, consumption or some other combined metric to say whether messages are being processed ‘acceptably’. | ||
...