Date |
|
Authors | |
Status | Documenting TIS21-5454 - Getting issue details... STATUS |
Summary | Display error message on Reval ‘under notice’ ( |
Impact | For approximately 90 minutes, some requests to do anything with recommendations (including doctor’s details) didn’t work |
Non-technical Description
Users on Revalidation were experiencing/receiving error messages, e.g. when searching for ‘Under Notice’ doctors. No records were showing
Trigger
Re-deployment
Detection
User Notification on Slack
Resolution
Service reached stability.
Timeline
All times in GMT unless indicated
09:33am A deployment was triggered. 2/3 tasks reported “unhealthy” in load balancer monitoring. This was up and down until 10:30 when it remained at 2 unhealthy tasks.
10:02am Several Logs of issues retrieving user’s profile information
10:30am User reported with TIS Reval with getting constant error messages (
oops something went wrong)10:32am another user reported that system is very slow and my under notice list just produced same error message as above
11:44am first responder reported issue was being investigated and likely now resolved ? Recommendation service has reached a steady state.
15:26pm User reported all working fine now.
Root Cause
The Ooops message was being displayed
The recommendation service was returning errors
Unhealthy tasks were being used
Tasks were repeatedly started and failing as part of a deployment
Action Items
There are a number of actions outstanding based on a similar occurrence which are yet to reach the top of the backlog
Action Items | Owner | |
|---|---|---|
Increased service resources (CPU): Tasks now start more quickly | DONE | |
Additionally, dependant on service provider support responses, we will recreate the service | ||
Cards are still outstanding for improving the observability of the services |
0 Comments