Non-technical Description

Errors being generated by some bad data (for one gmc number only- GMC No: 7064627.) being returned by the hourly recommendation status check from gmc. these errors are also creating snapshots, so mongoDB might be filling up

Trigger

Detection

• awslogs-prod-tis-revalidation-recommendation logs are up to 1.2GB, and looks like we were getting lots of Execution of Rabbit message listener failed . . . Caused by: java.lang.NullPointerException errors this morning (not sure if this is what accounts for that 1.2GB though) on the recommendation status check. looking into it. Prod seems fine though (edited)

Resolution

Timeline

• 09 Feb 2022 9:03 - Cai Willis reported the errors

• 09 Feb 2022 9:35 - Investigation started

• 09 Feb 2022 9:40 - issue reported to users and Recommendation paused for 5 minutes

• 09 Feb 2022 9:50 - Temporary fix made

• 09 Feb 2022 9: 50- Comms sent to users that recommendation was back

Root Cause(s)

Action Items

Lessons Learned