Date | |
Authors | Cai WillisDoris.Wong Jayanta Saha Joseph (Pepe) Kelly Adewale Adekoya |
Status | |
Summary | TIS21-2653 - Getting issue details... STATUS |
Impact |
Non-technical Description
Errors being generated by some bad data (for one gmc number only- GMC No: 7064627.) being returned by the hourly recommendation status check from gmc. these errors are also creating snapshots, so mongoDB might be filling up
Trigger
Detection
awslogs-prod-tis-revalidation-recommendation logs are up to 1.2GB, and looks like we were getting lots of
Execution of Rabbit message listener failed . . . Caused by: java.lang.NullPointerExceptionerrors this morning (not sure if this is what accounts for that 1.2GB though) on the recommendation status check. looking into it. Prod seems fine though (edited)
Resolution
Timeline
9:03 - Cai Willis reported the errors
9:35 - Investigation started
9:40 - issue reported to users and Recommendation paused for 5 minutes
9:50 - Temporary fix made
9: 50- Comms sent to users that recommendation was back
Root Cause(s)
Action Items
Action Items | Owner |
|---|---|
Add Comment