...
09:16 : User reports on Teams
I have tried to make a revalidation recommendation in TIS this morning and got 'An error occurred! - Please try again'. i have checked GMC Connect and it looks like it has gone though, can you advise if there are any issues.10:15 : User contacted requesting GMC number
11:19 : Call with user to demo the issue
11:49 : Huddle to debug and understand if isolated or system wide
13:00 : Identify that requests to profile service from integration service are not appearing in logs so redeployed integration service
13:10 : Call with user to test. Still broken.
13:30 : Other admins confirm they are also experiencing the same issue.
15:30 : Checked API-Gateway logs and found no logs for submitting recommendations on Monday.
16:00 : Found there were lots of timeouts in
prod-revalidation-api-gateway-authoriser, then updated it from 3s to 10s.19:30 : Updated the version of
nodeto run Lambdaprod-revalidation-api-gateway-authoriserfrom 12 to 16.08:45 : Checked
awslogs-prod-tis-revalidation-recommendationslogs for "submitting request to gmc for recommendation:" which were showing this morning09:00 : Verified with users that recommendations were being submitted.
5 Whys (or other analysis of Root Cause)
...