Date |
| ||||||||
Authors | |||||||||
Status | Documenting | ||||||||
Summary | Trusts were unable to find a number of trainees in their search results. We narrowed in on there being an issue with the copy of information that gets searched and reran the job that builds it.
| ||||||||
Impact | It wasn’t immediately obvious that some records were not showing in the person search |
...
All times in GMT unless indicated
01: - Other Jobs ran for longer than usual and ran beyond the start of the Person ?ES? Job
01:?? - Job ran for xx ?? minutes, when it usually completes in ~15.
12:01 - Message on Teams about Trust users not finding their trainees in the search.
13:37 - Started debugging and confirming the cause / that there were no other data related issues.
14:15 - Confirmed that other regions are affected. A reindex was scheduled.
15:45 & 16:00 - Confirmed that records were visible as expected.
As part of building the timeline, we didn’t identify an earlier occurrence of this defect so we have not sought to extensively reproduce and remedy this issue.
Root Cause(s)
N.B. We have developed a reasonable but not definitive explanation of what has happened.
Users in more than one region/Local Office couldn’t find trainees they were expecting because the search index didn’t have all the records it should have but we believe it did have many/most of the trainees they expected.
The ElasticSearch job completed but ran for longer than expected, as did other jobs
The ElasticSearch Job is dependent on on other jobs running successfully, roughly before it starts.
ElasticSearch, & other jobs work through pages of ids so where jobs overlap, this can lead to partial information being used instead of complete information
...
Action Items
Action Items | Owner | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
Alert when jobs (or just this job) runs outside the “normal”/”expected”/”acceptable” bounds, e.g.
|
| |||||||||
Space jobs out more to allow more time for each to run | ||||||||||
We could: rebuild as a batch job but won’t right now as it would be a significant piece of work |
...