2022-03-03 TV list of doctors did not sync for Under notice and All doctors
Date | Mar 3, 2022 |
Authors | @John Simmons (Deactivated) @Cai Willis @Doris.Wong |
Status | In progress |
Summary | TV list of doctors did not sync for Under notice and All doctors |
Impact | TV Reval admins couldn’t process Under notice doctors, nor see a list of any of their doctors |
Non-technical Description
The recommendations list was showing up as empty for Thames Valley doctors as they had failed to sync overnight.
Trigger
.
Detection
User detected the issue
Resolution
Reduced frequency of recommendation status check
Improved logging in gmc-client
Prevented sending of further invalid requests
Timeline
2022-03-04. 09:50 TV user reported the issue in teams
2022-03-04. 09:57 it was picked up by the team for investigation
2022-03-04. 10:17 It was discovered during investigation, it was probably affecting TV only
2022-03-04. 10:27 TV list of doctors was back
2022-03-04. 11:40 Update cron parameter for triggering GMC Recommendation status check 3 times a day within office hour instead of once per hour
2022-03-04. 12:20 Additional logging revealed old data (from initial migration back when) had records that were “Under Review” but also had a null gmcRecommendationId
2022-03-04. 15:07 Quick fix pushed to production
2022-03-04. 16:02 Confirmation of quick fix success
2022-03-07. 08:30 https://hee-tis.atlassian.net/browse/TIS21-2459 updated for future cleanup of data
Root Cause(s)
Old data (from initial migration back when) had records that were “Under Review” but also had a null gmcRecommendationId
Insufficient logging
Action Items
Action Items | Owner |
---|---|
| @Doris.Wong |
| @Doris.Wong @Cai Willis |
| @Cai Willis |
Lessons Learned
Don’t ignore errors for too long.
Take opportunities to fix potential issues where possible - this data issue was spotted a while ago (to discuss - pragmatism)
Well thought out logging is invaluable
Slack: https://hee-nhs-tis.slack.com/
Jira issues: https://hee-tis.atlassian.net/issues/?filter=14213