Non-technical Description

A Revalidation Admin drew our attention to some data that did not appear to be correct. In early October, we corrected how we handle whether a doctor is “under notice” for revalidation. As we only receive partial information from the GMC, we removed stale and potentially incorrect data from a number of records and proactively remove unreliable data as we get some information. One of our services relied on always maintaining knowledge of whether or not a doctor was “under notice”, which was no longer possible so it failed to keep those doctors records updated.

This meant for a small number of doctors who were connected to a NHS E designated body outside of TIS Revalidation after 5th October did not appear until Friday 3rd November, the first sync after releasing a fix. While investigating the problem on 2nd November, we came across an intermittent issue loading pages. This was resolved in part initially and fully within 3 hours of being noticed by the team and within 2 hours of being reported by users.

Having addressed the bug, we have identified that there are other places in the code which can be improved in the same way. For the issue of partial failures leading to error messages, we will be putting in place the kind of monitoring that will enable us to detect when there is an issue for 1 user and improve our ability to respond before it becomes a more widespread problem.

Trigger

• Updating under notice

• Service Degradation?

Detection

• User query on Teams

Resolution

• Bugfix released 2nd Nov pm

• Redeploying ?recommendation? service: tasks that were taking a long time to respond
  

Timeline

All times in BST unless indicated

• 05 Oct 2023: Change released to correct data and processing for whether doctors are “under notice”.

• 06 Oct 2023 ~00:10 : Logs show that there are approx. 2 “Null Pointer Exception”s that mean we don’t 2 doctors as connected.

• 30 Oct 2023: 18:03 User queries connection status - Trainee is showing connected on GMC Connect but not TIS Revalidation.

• 31 Oct 2023: 10:47 First responder requested when they were last connected via. GMC Connect.

• 31 Oct 2023: 10:57 Reported that trainee are not even in the connections list on TIS but are on GMC connect - 18/10/2023.

• 31 Oct 2023: 11:04 reported that another doctor was connected the day before but remains on the discrepancy list as if they weren’t connected. It initially appeared to be different to the other doctor listed above.

• 31 Oct 2023: 11:40 Responder notified that the last time their records in revalidation were updated were the 18/10 & yesterday(1/11/23)

• 31 Oct 2023: Unable to manual get doctors for debugging failed: Request was blocked, possibly because of earlier bad requests.

• 01 Nov 2023: Manual verification of GMC responses showed 1 doctor was not in the list of connected doctors but the other was.

• 02 Nov 2023: Further debugging identified the cause.

• 02 Nov 2023: While investing, errors in the Reval app point to another issue: Gateway timeouts some of the time

• 02 Nov 2023: Investigation

• 02 Nov 2023 11:35 : replaced recommendation tasks in production

• 02 Nov 2023 11:53 : replaced connection tasks in production

• 02 Nov 202312:06 : replaced integration & core tasks in production

Root Cause(s)

Doctors weren’t appearing as Connected because they were marked as existsInGmc=false and no connected Designated Body but were in GMC Connect as connected.

The data remained without a connection because the nightly sync could not update information from the GMC.

Updates from the GMC failed because there was a NullPointerException.

There was a NullPointerException because our service didn’t handle null values in under notice and possibly others.

It was unhandled because we had relied on the fields always being populated. We changed this as the data available when it changes externally can be outdated.

---

We (and then users) were unable load pages consistently because it sometimes came up with errors.

Errors were caused by “timeouts”.

The timeouts were reported by the API gateway because a downstream service did not respond within a maximum allowable time. API Gateway logs showed timeouts between 09:45 and 11:54 (HTTP 504) and latency

The connections service sometimes didn’t respond in the allowable time, possibly because it couldn’t be reached. There were no indications of a HTTP 504 in our services? We note that the correlation with “Unhealthy Routing Flow Count” for 2 of our 3 availability zones. This is why some actions were successful.

Action Items

Lessons Learned