Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

Version 1 Next »

Date

Authors

Cai Willis Yafang Deng Jayanta Saha Joseph (Pepe) Kelly

Status

Done

Summary

Impact

The GMC sync job doesn’t update the data of index trainees in ES, so users might see some stale data.

Non-technical Description

.

Trigger

Detection

  • Message from 2 users (assumed a similar / same root cause?)

Resolution

  • Get the index up to date

  • Give users links to pages

Timeline

BST unless otherwise stated

  • - 01:05 Slack alert on failure job

  • Checked database

  • ElasticSearch

  • Aug 30, 2022 - 09:07 Sync was started manually and it still failed

  • Aug 31, 2022 - 01:05 Slack alert on failure job

  • Aug 31, 2022 - 10:21 Found Sentry alerting of Profile service

  • Sep 01, 2022 - 01:05 Slack alert on failure job

  • Sep 01, 2022 - 11:39 PR was merged to remove the doctors with null GMC number from the list sent to Profile service, but didn’t work

  • Sep 01, 2022 - 12:01 Found the image of TIS-GMC-SYNC running on Prod was built 3 years ago

  • Sep 01, 2022 - 12:15 PR was merged to bump the build version of TIS-GMC-SYNC service on TIS-DEVOPS, but didn’t work

  • Sep 01, 2022 - 13:32 Image version issue was fixed on Prod, but Slack still alert on failure job

  • Sep 01, 2022 - 14:29 PR was merged to add debug logging for duplicate GMC numbers

  • Sep 01, 2022 - 15:01 TIS-GMC-SYNC job failed again, but duplicate GMC number was logged

  • Sep 01, 2022 - 15:30ish Duplicate GMC number was identified from Curl output

  • Sep 01, 2022 - 15:37 PR of adding debug logging was reverted

  • Sep 05, 2022 - 11:30ish Got reply from GMC and looked into the logs in the midnight. We still got duplicates. Rob sent a further email to GMC.

  • Sep 13, 2022 - 10:15ish We checked the logs and no duplicates found. https://hee-nhs-tis.slack.com/archives/C03GBMYGZD4/p1663060704518199?thread_ts=1663060383.191659&cid=C03GBMYGZD4

  • Sep 20, 2022 a sentry monitoring was added for duplicates from GMC.

Root Cause(s)

  • We have received duplicate doctors sent by GMC with same gmc ids for HEE South West (1-AIIDMQ). The GMC id is 7134553 . The doctors is being sent to us twice, which previously never happened and is not expected. 

  • Based on the assumption that GMC won’t send us duplicates, Profile service use GMC number as key of Map without filtering out duplicate GMC numbers, which causes the error.

Action Items

Action Items

Comments

Owner

Send an email or message to check with GMC

Rob Pink

Do we want to improve the CI CD process for TIS-GMC-SYNC?

The effort may not worthy to fix this ATM

Investigate why the exceptions were not recorded in Profile Cloudwatch logs

We need a ticket to look into this. We came to the issue from Sentry

TIS21-3426 - Getting issue details... STATUS

Yafang Deng

Check if this issue affects new revalidation

Do we need to double check what does the new Reval Sync do with the duplicates received from GMC?

Jayanta Saha Cai Willis

Isolate duplicates in TIS-GMC-SYNC service

Done

Fix isolate duplicates from gmc by hihilary · Pull Request #30 · Health-Education-England/TIS-GMC-SYNC (github.com)

Yafang Deng Jayanta Saha

Create Sentry alert for TIS-GMC-SYNC to capture logging when GMC sends duplicates

Done

TIS21-3513 - Getting issue details... STATUS

Yafang Deng Jayanta Saha

Lessons Learned

  • Add more debug logging if current logging are not enough to identify the cause.

  • Check Sentry if logs are not found as expected on Cloudwatch.

  • No labels

Slack: https://hee-nhs-tis.slack.com/
Jira issues: https://hee-tis.atlassian.net/issues/?filter=14213