Date | |
Authors | |
Status | In progress |
Summary | TV list of doctors did not sync for Under notice and All doctors |
Impact | TV Reval admins couldn’t process Under notice doctors, nor see a list of any of their doctors |
Non-technical Description
We’ve been experiencing problems in Stage which we believe to be sync issues from GMC. These appear to now be affecting ProdThe recommendations list was showing up as empty for Thames Valley doctors as they had failed to sync overnight.
...
Trigger
.
Detection
User detected the issue
...
Resolution
Reduced frequency of recommendation status check
Improved logging in gmc-client
Prevented sending of further invalid requests
...
Timeline
2022-03-04. 09:50 TV user reported the issue in teams
2022-03-04. 09:57 it was picked up by the team for investigation
2022-03-04.
Resolution
.
...
10:17 It was discovered during investigation, it was probably affecting TV only
2022-03-04. 10:27 TV list of doctors was back
2022-03-04. 11:40 Update cron parameter for triggering GMC Recommendation status check 3 times a day within office hour instead of once per hour
2022-03-04. 12:20 Additional logging revealed old data (from initial migration back when) had records that were “Under Review” but also had a null gmcRecommendationId
2022-03-04. 15:07 Quick fix pushed to production
2022-03-04. 16:02 Confirmation of quick fix success
2022-03-07. 08:30
updated for future cleanup of dataJira Legacy server System JIRA serverId 4c843cd5-e5a9-329d-ae88-66091fcfe3c7 key TIS21-2459
...
Root Cause(s)
...
Old data (from initial migration back when) had records that were “Under Review” but also had a null gmcRecommendationId
Insufficient logging
...
Action Items
Action Items | Owner |
|---|
Lessons Learned
...
| |
| |
|
...
Lessons Learned
Don’t ignore errors for too long.
Take opportunities to fix potential issues where possible - this data issue was spotted a while ago (to discuss - pragmatism)
Well thought out logging is invaluable