Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 6 Next »

Date

Authors

Status

Documenting TIS21-4706 - Getting issue details... STATUS

Summary

Impact

Non-technical Description


Trigger

  • When a Reval admin submits a Recommendation, a success message is shown in the UI followed by an error, but not submission is made to GMC


Detection

  • User reported issue on teams


Resolution


Timeline

All times in BST unless indicated

  • 09:16 : User reports on Teams
    I have tried to make a revalidation recommendation in TIS this morning and got 'An error occurred! - Please try again'. i have checked GMC Connect and it looks like it has gone though, can you advise if there are any issues.

  • 10:15 : User contacted requesting GMC number

  • 11:19 : Call with user to demo the issue

  • 11:49 : Huddle to debug and understand if isolated or system wide

  • 13:00 : Identify that requests to profile service from integration service are not appearing in logs so redeployed integration service

  • 13:10 : Call with user to test. Still broken.

  • 13:30 : Other admins confirm they are also experiencing the same issue.

  • 15:30 : Checked API-Gateway logs and found no logs for submitting recommendations on Monday.

  • 16:00 : Found there were lots of timeouts in prod-revalidation-api-gateway-authoriser , then updated it from 3s to 10s.

  • 19:30 : Updated the version of node to run Lambda prod-revalidation-api-gateway-authoriser from 12 to 16.

  • 08:45 : Checked awslogs-prod-tis-revalidation-recommendations logs for "submitting request to gmc for recommendation:" which were showing this morning

  • 09:00 : Verified with users that recommendations were being submitted.

5 Whys (or other analysis of Root Cause)



Action Items

Action Items

Owner

Comments


Lessons Learned

  • No labels

0 Comments

You are not logged in. Any changes you make will be marked as anonymous. You may want to Log In if you already have an account.