Date | |
Authors | |
Status | In progress |
Summary | TV list of doctors did not sync for Under notice and All doctors |
Impact | TV Reval admins couldn’t process Under notice doctors, nor see a list of any of their doctors |
Non-technical Description
The recommendations list was showing up as empty for Thames Valley doctors as they had failed to sync overnight.
...
Trigger
.
Detection
User detected the issue
...
Resolution
...
Reduced frequency of recommendation status check
Improved logging in gmc-client
Prevented sending of further invalid requests
...
Timeline
2022-03-04. 09:50 TV user reported the issue in teams
2022-03-04. 09:57 it was picked up by the team for investigation
2022-03-04. 10:17 It was discovered during investigation, it was probably affecting TV only
2022-03-04. 10:27 TV list of doctors was back
2022-03-04. …………………………..
Root Cause(s)
.11:40 Update cron parameter for triggering GMC Recommendation status check 3 times a day within office hour instead of once per hour
2022-03-04. 12:20 Additional logging revealed old data (from initial migration back when) had records that were “Under Review” but also had a null gmcRecommendationId
2022-03-04. 15:07 Quick fix pushed to production
2022-03-04. 16:02 Confirmation of quick fix success
2022-03-07. 08:30
updated for future cleanup of dataJira Legacy server System JIRA serverId 4c843cd5-e5a9-329d-ae88-66091fcfe3c7 key TIS21-2459
...
Root Cause(s)
Old data (from initial migration back when) had records that were “Under Review” but also had a null gmcRecommendationId
Insufficient logging
...
Action Items
Action Items | Owner |
|---|---|
| |
| |
|
...
Lessons Learned
...
Don’t ignore errors for too long.
Take opportunities to fix potential issues where possible - this data issue was spotted a while ago (to discuss - pragmatism)
Well thought out logging is invaluable