Non-technical Description

.

Trigger

Detection

• Message from 2 users

Resolution

• A “force redeployment” update of the recommendation service

• Gave users links to details pages for doctors which could not be found

Timeline

BST unless otherwise stated

• 23 Nov 2022 -

• 23 Nov 2022 02:26 to 08:07 - Queue to recommendation for ‘doctor view’ update built steadily to ~83K

• 23 Nov 2022 -

• 23 Nov 2022 08:21 - First report in user channel

• 23 Nov 2022 12:07 - Picked up for investigation

• 23 Nov 2022 12:07 to 14:00ish - Checked database & ElasticSearch index

• 23 Nov 2022 13:00 - Checked the return list of GMC for north west

• 23 Nov 2022 14:00ish - Found messages in reval.queue.masterdoctorview.updated.recommendation didn’t get consumed

• 23 Nov 2022 16:00ish - Force a new start of recommendation service

• 23 Nov 2022 -

• .

Root Cause(s)

• Doctors reported as not showing in the search list

• ElasticSearch Index for Recommendation Service isn’t being updated

• Large backlog of messages stuck on a queue for updating the index

• Message Consumers disappeared but after the final aws ecs update-service --force-new-deployment dropped to one before going back up to 3

• ?

Action Items

Lessons Learned