2020-12-13 Reval Legacy/Old GMC Sync - GMC return code 98, GMC internal error

Date	Dec 14, 2020
Authors	@Philip Wilsdon (Unlicensed)
Status	Completed
Summary	`gmc-sync-prod` didnt run - logs reporting `GMC return code 98, GMC internal error` Contacted GMC - the did some server patching on the Dec 13, 2020 After rebooting their API servers, we were able to connect and run the sync jobs on both legacy and new reval successfully
Impact	User could not manage connections

1 Non-technical summary
2 Timeline
3 Root Causes
4 Trigger
5 Resolution
6 Detection
7 Actions
8 Lessons Learned (Good and Bad)

Non-technical summary

Reval was showing information that had not been updated from GMC, on further investigation it turned out that the jobs that run overnight to get the data from GMC and then add it into TIS had not run successfully. Once they had been fixed and rerun, Reval showed the correct information.

Timeline

Dec 14, 2020 09:23 am
09:45	we re-ran gmc sync - returning 0 doctors per designated body
10:00 Ish	investigations found that `gmc-sync-prod` didnt run and that the logs are showing the error `GMC return code 98, GMC internal error` 2020-12-13 00:00:41.557 INFO 1 --- [ scheduling-1] u.n.h.t.g.c.service.GmcConnectService : Get doctors for designated body 1-AIIDWA 2020-12-13 00:00:41.557 INFO 1 --- [ scheduling-1] u.n.h.t.g.c.service.GmcConnectService : GMC Connect Url 2020-12-13 00:00:44.263 INFO 1 --- [ scheduling-1] u.n.h.t.g.c.service.GmcDoctorsService : 2982 Doctors has been found for 1-AIIDWA body 2020-12-13 00:00:44.756 INFO 1 --- [ scheduling-1] u.n.h.t.g.c.service.GmcConnectService : Get doctors for designated body 1-AIIDVS 2020-12-13 00:00:44.756 INFO 1 --- [ scheduling-1] u.n.h.t.g.c.service.GmcConnectService : GMC Connect Url 2020-12-13 00:00:48.562 INFO 1 --- [ scheduling-1] u.n.h.t.g.c.service.GmcDoctorsService : 4340 Doctors has been found for 1-AIIDVS body 2020-12-13 00:00:49.153 INFO 1 --- [ scheduling-1] u.n.h.t.g.c.service.GmcConnectService : Get doctors for designated body 1-AIIDWI 2020-12-13 00:00:49.154 INFO 1 --- [ scheduling-1] u.n.h.t.g.c.service.GmcConnectService : GMC Connect Url 2020-12-13 00:00:52.752 INFO 1 --- [ scheduling-1] u.n.h.t.g.c.service.GmcDoctorsService : 4022 Doctors has been found for 1-AIIDWI body 2020-12-13 00:00:52.847 INFO 1 --- [ scheduling-1] u.n.h.t.g.c.service.GmcConnectService : Get doctors for designated body 1-AIIDSI 2020-12-13 00:00:52.847 INFO 1 --- [ scheduling-1] u.n.h.t.g.c.service.GmcConnectService : GMC Connect Url 2020-12-13 00:00:55.954 INFO 1 --- [ scheduling-1] u.n.h.t.g.c.service.GmcDoctorsService : 3107 Doctors has been found for 1-AIIDSI body 2020-12-14 00:00:00.000 INFO 1 --- [ scheduling-1] u.n.h.t.g.c.service.GmcDoctorsService : Fetching doctors info for Designated Bodies [1-AIIDHJ, 1-AIIDMQ, 1-AIIDNQ, 1-AIIDMY, 1-AIIDQQ, 1-AIIDWT, 1-AIIDR8, 1-AIIDSA, 1-AIIDH1, 1-AIIDWA, 1-AIIDVS, 1-AIIDWI, 1-AIIDSI] 2020-12-14 00:00:00.001 INFO 1 --- [ scheduling-1] u.n.h.t.g.c.service.GmcConnectService : Get doctors for designated body 1-AIIDHJ 2020-12-14 00:00:00.001 INFO 1 --- [ scheduling-1] u.n.h.t.g.c.service.GmcConnectService : GMC Connect Url 2020-12-14 00:00:00.103 ERROR 1 --- [ scheduling-1] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task uk.nhs.hee.tis.gmc.client.exception.GMCResponseException: GMC return code 98, GMC internal error
10:15 AM	Emailed GMC
12:06	More correspondance with the GMC as they asked for calrifcation on envrionments
05:17 PM	Clarifcation of logs
7 PM	Logs from legacy/old production envrionment sent to GMC
Dec 15, 2020 Reply from GMC	They patched their survers on the Dec 13, 2020 evening
10:01	Triggered off old reval New reval - gmc sync completed Suspect that GMC applied fixes early AM
10:03	gmc-sync ran sucsessfully and intredpid-reval-etl-all-prod run sucessfully
10:18	User confirmed everything looks ok
10:19	replied to GMC to let them know that both sync jobs on legacy and new reval ran successfully
10:23

Root Causes

GMC server patching on Dec 13, 2020

Trigger

A user reported in Teams Support Channel that their connections had not been working correctly

Resolution

GMC restarted their API servers on the morning of the Dec 15, 2020
TIS team ran GMC sync jobs for legacy and new reval

Detection

A user reported in Teams Support Channel

Actions

GMC said they will add monitoring
TIS team will also add monitoring and slack notifications for new reval

Lessons Learned (Good and Bad)

Education for GMC about how we retrieve the data by designated bodies