2024-04-09 Certain Trusts unable to see doctors in person search list in TIS

Owned by catherine.odukale
Last updated: 24 Apr 2024 by Joseph (Pepe) Kelly
2 min read

Date

Apr 9, 2024

Authors

@Steven Howard

Status

https://hee-tis.atlassian.net/browse/TIS21-5918

Summary

The person search screen in TIS was not displaying all doctors following the overnight sync jobs. Re-running the Person Elastic Search Sync fixes the issue until the next overnight sync job runs.

Impact

Initially reported by certain Trusts in the North East but also impacted other users, particularly Trust users: Some users were unable to search for trainees by their personal information, only able to navigate to them via the posts that they were/will be placed in.

Non-technical Description

Some of the overnight jobs that keep TIS up to date are dependant on earlier jobs. We observed that from the 4th April, one of the jobs that connects a “Training Body” to a “Person” in TIS was taking significantly longer than it previously has done.

 

Trigger

Failure in overnight sync

Detection

  • User feedback

Resolution

Direct issue resolved by running the “Search” job later.

Timeline

 

  • Apr 4, 2024 02:54: The job to link each training body finishes compared to earlier than 02:15 in the preceding few days

  • Apr 9, 2024 11:22 Sarah Hill raises issue on Teams reporting 2 trusts unable to see doctors in person search

  • Apr 9, 202412:36 Responded querying possible issue with specific user accounts

  • Apr 11, 202407:38 Person sync job re-run

  • Apr 11, 2024 08:12 Updated user with possible cause due to failures in the overnight sync - user confirmed person search displaying as expected

  • Apr 15, 2024 08:46 Same users again reporting issues with missing doctors in person search

  • Apr 15, 2024 08:50 Person sync job re-run and user confirmed issue resolved

  • Apr 16, 2024 08:48 Same users again reporting issues with missing doctors in person search

  • Apr 16, 2024 09:13 Person sync job re-run and user confirmed issue resolved

  • Apr 16, 2024 11:21 Investigation into sync jobs shows that since 4 April the PersonPlacementEmployingBodyTrustJob takes roughly 30mins longer than previously. However, this still completes prior to the person sync job starting.

  • Apr 17, 2024 Made a change to run the job later. This now overlaps with other ETL job. See https://hee-tis.atlassian.net/wiki/spaces/NTCS/pages/509968433.

5 Whys (or other analysis of Root Cause)

Users with Trust access see a limited set of people based on a link between a person and the trust. They were not able to see some people because the “cached” data in OpenSearch didn’t have the link to their trust.

We have an OpenSearch build job which runs every night to save the latest data. Before that runs, 2 jobs need to finish, which join a person and the trusts that are permitted to see them.

One of the jobs that adds/recreates the links between a person and the trust, specifically the one for the “Training” Body, hadn’t created all the joins when the OpenSearch job started.

The job that creates the joins ran for much longer that the expected time but we aren’t certain why this job is taking much longer.

 

Action Items

Action Items

Owner

Comments

Action Items

Owner

Comments

Investigate why it is taking longer for the job(s) to complete

 

Ideally, we’d look into it but resource constraints mean we are mitigating this issue for now.

Do some previously refined work

https://hee-tis.atlassian.net/browse/TIS21-5628

 

Needs to be bumped up the backlog

 

 

 

 

Lessons Learned

  •  

 

Slack: https://hee-nhs-tis.slack.com/
Jira issues: https://hee-tis.atlassian.net/issues/?filter=14213