Non-technical Description

Users on Revalidation were experiencing/receiving error messages, e.g. when searching for ‘Under Notice’ doctors. No records were showing. This is because a request to replace the tasks for the recommendation feature, called a “deployment”, was mistakenly bringing in tasks that were being identified as healthy before they were ready to take user’s requests.

A little over an hour after the deployment began, the service did eventually reach a point were the tasks that were taking requests did fully start up before receiving requests. The TIS team have been giving their focus to improvements in the application defined as higher priority so are yet to address the way that tasks are identified as healthy or unhealthy. As part of another incident, more resource was made available to the tasks and that meant that they are ready to take requests much quicker.

Trigger

Re-deployment

Detection

User Notification on Slack

Resolution

Service reached stability.

Timeline

All times in GMT unless indicated

• Nov 28, 2023 09:33am A deployment was triggered. 2/3 tasks reported “unhealthy” in load balancer monitoring. This was up and down until 10:30 when it remained at 2 unhealthy tasks.

• Nov 28, 2023 10:02am Several Logs of issues retrieving user’s profile information

• Nov 28, 2023 10:30am User reported with TIS Reval with getting constant error messages (oops something went wrong)

• Nov 28, 2023 10:32am another user reported that system is very slow and my under notice list just produced same error message as above

• Nov 28, 2023 11:44am first responder reported issue was being investigated and likely now resolved ? Recommendation service has reached a steady state.

• Nov 28, 2023 15:26pm User reported all working fine now.

Root Cause

• The Ooops message was being displayed

• The recommendation service was returning errors

• Unhealthy tasks were being used

• Tasks were repeatedly started and failing as part of a deployment

Action Items

There are a number of actions outstanding based on a similar occurrence which are yet to reach the top of the backlog

Lessons Learned