Non-technical Description

Users on Revalidation were experiencing/receiving error messages, e.g. when searching for ‘Under Notice’ doctors. No records were showing

Trigger

Re-deployment

Detection

User Notification on Slack

Resolution

Service reached stability.

Timeline

All times in GMT unless indicated

• 28 Nov 2023 09:33am A deployment was triggered. 2/3 tasks reported “unhealthy” in load balancer monitoring. This was up and down until 10:30 when it remained at 2 unhealthy tasks.

• 28 Nov 2023 10:02am Several Logs of issues retrieving user’s profile information

• 28 Nov 2023 10:30am User reported with TIS Reval with getting constant error messages (oops something went wrong)

• 28 Nov 2023 10:32am another user reported that system is very slow and my under notice list just produced same error message as above

• 28 Nov 2023 11:44am first responder reported issue was being investigated and likely now resolved ? Recommendation service has reached a steady state.

• 28 Nov 2023 15:26pm User reported all working fine now.

Root Cause

Action Items

Lessons Learned