...
...
...
...
...
...
...
...
...
...
...
...
...
...
STILL WIP
| Expand | ||
|---|---|---|
| ||
Problems:
• Hypothesis: It would all be much simpler if we were copying all programme memberships separately
Principals:
Solutions:
Three ways of caching data, the first might not be viable because of connection specific info:
|
Aims
It’s worth remembering what it is we are trying to achieve here.
...
This discussion is fundamentally about looking at how we have structured the indexes (roughly equivalent to tables in a database) for Revalidation, and whether or not it is fit for purpose going forwards.
What do we have now?
For both Recommendations and Connections we have a list of doctors with fields aggregated from TIS, GMC and our Revalidation applications:
...
The ES Resync Job - This is a seldom used but very important operation in which clears out all of the Elasticsearch indexes and re-fetches all of the data from TIS and the GMC to re-populate Elasticsearch. Typical use cases for this are first-time data loading, recovery from catastrophic pipeline failure, mass population of a new field, etc. In terms of the current connections work, we need this working as all the data in connections is stale and needs re-populating.
Change Data Capture (CDC) - Great so we have a way of populating Elasticsearch, how do we keep it up to date? We listen for changes in TIS (using event listeners), GMC (using cron jobs) and our own Revalidation Applications (using Database changelogs) and send messages to the Integration service. The Integration service updates the masterdoctorindex, and these updates are propagated (copied) to the other indexes managed by the different services.
GET Requests - The data is in Elasticsearch and up to date! How do I see it? Load the web page! When you go on revalidation.tis.nhs.uk/recommendations or revalidation.tis.nhs.uk/connections the application sends a GET request to the relevant service which queries Elasticsearch to get the correct data. Elasticsearch is fast. In recommendations a fairly complex query with wildcard matching (pseudocode shown below) run against over 300,000 doctors takes between 500 and 750 milliseconds to complete!
Code Block underNotice = "Yes" AND designatedBody matches any from parameter 1 AND existsInGmc = true AND ( doctorFirstName = parameter 0 (including partial) OR doctorLastName = parameter 0 (including partial) OR gmcNumber = parameter 0 (including partial) OR programmeName = parameter 0 (including partial) OR return everything (i.e if no parameter provided) )
So what’s the problem?
The process we have for the ES Resync Job was incredibly slow - taking over 6 hours to complete. As the list page of the revalidation application is unavailable during this sync process, this means that running this on prod would result in a whole day’s worth of “downtime” (admins do most of their work based on this list)!
...
The approach of “pre-sorting” the data was also fine before as the exact same code was used for CDC and the ES Resync job. However, in order to repeat the massive time saving we achieved in
| Jira Legacy | ||||||
|---|---|---|---|---|---|---|
|
Summary
Having multiple indexes makes GET requests simpler
performance has been raised as a potential benefit, but when more complex queries on large data sets take less than a second it’s questionable how much benefit this would really give.
Multiple indexes means duplicating data
Multiple indexes makes requires multiple updates for a single data change
Because we have separate CDC and Resync processes, and because the Java approach is prohibitively slow for the Resync process, we would have to write and maintain the business logic in separate places in separate languages
Tasks to complete TIS21-3774 with this approach
Copy implementation for reindexing used by recommendations
Implement elasticsearch query to match java logic in CDC process for use as part of reindex api call
BONUS: figure out if we can run the reindex for connections and recommendation separately (might be impractical)?
Alternative Approach 1 - Single Connection index
TODO - it’s late
| Drawio | ||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
Tasks to complete TIS21-3774 with this approach
Copy implementation for reindexing used by recommendations
Replace Connected, Disconnected and Exception repositories with single Connection repositiory
Delete code for pre-sorting the doctors (ElasticsearchIndexUpdateHelper)
Have CDC listener update new index directly (see step 3)
Write ES query to GET Connected Doctors (based on currently implemented logic only)
Write ES query to GET Discrepancies (based on currently implemented logic only)
Interlude - Schema Design
Alternative Approach 2 - While we’re at it, single Reval index?
| Drawio | ||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
...