...

• User detected the issue

...

Resolution

...

• Reduced frequency of recommendation status check

• Improved logging in gmc-client

• Prevented sending of further invalid requests

...

Timeline

• 2022-03-04. 09:50 TV user reported the issue in teams

• 2022-03-04. 09:57 it was picked up by the team for investigation

• 2022-03-04. 10:17 It was discovered during investigation, it was probably affecting TV only

• 2022-03-04. 10:27 TV list of doctors was back

• 2022-03-04. 11:40 Update cron parameter for triggering GMC Recommendation status check 3 times a day within office hour instead of once per hour

• 2022-03-04. 12:20 Additional logging revealed old data (from initial migration back when) had records that were “Under Review” but also had a null gmcRecommendationId

• 2022-03-04. 15:07 Quick fix pushed to production

• 2022-03-04. 16:02 Confirmation of quick fix success

• 2022-03-07. 08:30

  Jira Legacy
  server System JIRA serverId 4c843cd5-e5a9-329d-ae88-66091fcfe3c7 key TIS21-2459

  updated for future cleanup of data

...

Root Cause(s)

• Old data (from initial migration back when)

...

• .had records that were “Under Review” but also had a null gmcRecommendationId

• Insufficient logging

...

Action Items

...

Lessons Learned

• Don’t ignore errors for too long.

• Take opportunities to fix potential issues where possible - this data issue was spotted a while ago (to discuss - pragmatism)

• Well thought out logging is invaluable