| Expand | ||
|---|---|---|
| ||
Problems:
• Hypothesis: It would all be much simpler if we were copying all programme memberships separately
Principals:
Solutions:
Three ways of caching data, the first might not be viable because of connection specific info:
|
...
For both Recommendations and Connections we have a list of doctors with fields aggregated from TIS, GMC and our Revalidation applications:
...
This data in this list is stored in Elasticsearch. For Recommendations, everything is run from a single index - recommendationsIndex (Note that there are two tabs available, “All Doctors” and “Under Notice”).
...
For connections this is slightly different.
...
The problem is the structure of the Connection service’s indexes, and the fact that each of these indexes requirs requires business legic logic to be applied at the point where we insert the data. For Recommendations the reindex is very straightforward - just a 1:1 copy of the contents of masterdoctorindex with some mapping configuration (terminology note: “mapping” in this case is more like metadata attached to specific fields that enable certain search features in Elasticsearch, not mapping as you might think - though it can also be like that . . . anyway . . .).
...
The approach of “pre-sorting” the data was also fine before as the exact same code was used for CDC and the ES Resync job. However, in order to repeat the massive time saving we achieved in
| Jira Legacy | ||||||
|---|---|---|---|---|---|---|
|
Summary
Having multiple indexes makes GET requests simplerperformance has been raised as a potential benefit, but when more complex queries on large data sets take less than a second it’s questionable how much benefit this would really give.
- It’s what we’ve got already 🤷♀️
Multiple indexes means duplicating data- Multiple indexes makes requires multiple updates for a single data change
- Because we have separate CDC and Resync processes, and because the Java approach is prohibitively slow for the Resync process, we would have to write and maintain the business logic in separate places in separate languagesseparate languages
- Every time we make a change to the business logic, we would have to do a full resync!
Tasks to complete TIS21-3774 with this approach
...
masterdoctorindex Fields | Required by Recommendations | Required by Connections |
|---|---|---|
id | ✅ | ✅ |
tcsPersonId | ✅ | ✅ |
gmcReferenceNumber | ✅ | ✅ |
doctorFirstName | ✅ | ✅ |
doctorLastName | ✅ | ✅ |
submissionDate | ✅ | ✅ |
ProgrammeName | ✅ | ✅ |
membershipType | ✅ | ✅ |
designatedBody | ✅ | ✅ |
gmcStatus | ✅ | |
tisStatus | ✅ | |
admin | ✅ | |
lastupdatedDate | ✅ | |
underNotice | ✅ | |
tcsDesignatedBody | ✅ | |
programmeOwner | ✅ | |
curriculumEndDate | ✅ | |
connectionStatus | ✅ | |
membershipStartDate | ✅ | |
membershipEndDate | ✅ | |
existsInGmc | ✅ | |
exceptionReason* *this field is currently in the code in connections but doesn’t exist in masterdoctorindex, appears to have been overlooked | ✅ |
As we can see, both services share a lot of fields, so this could be motivation for either:
Take the redesign even further, having both just use a single shared index
Using an Ingest pipeline to only copy across relevant fields to the relevant service
Other things to note:
...
There’s no “isException” flag. Whether or not something appears in discrepanceiss is calculated by the values of certain fields, e.g. membershipType = “visitor” or programmeMembershipEndDate > now
Why do we even bother with a masterdoctorindex? - As both services generally need the same data, a shared main schema allows us to use exactly the same pipelines for getting GMC and TIS data loaded and aggregated into the revalidation system
Alternative Approach 1 - Single Connection index
...
This should be a fairly straightforward conversion, for example where we currently pre-sort with Java if(<conditions for discrepancy>) we would instead GET with a Where <fieldValue> = <condition for discrepancy> in Elasticsearch.
Summary
- Generally simplifies the system architecture
- This approach means only need to implement the business logic in one place in one language
- We can make changes to the business logic without having to resync all the data!
- A single index means we’re not duplicating data unnecessarily and simplifies the update process
- Removing the “pre-sort” step greatly simplifies the CDC and Resync processes and makes it more consistent with how we do Recommendations
- GET requests become more complicated than in the current approach
Although no more complicated than what we have on Recommendations, and implementing filters becomes more consistent and straightforward
- Is there a business logic case we couldn’t replicate using a query language as opposed to Java?
Tasks to complete TIS21-3774 with this approach
...
| Drawio | ||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
Summary
(all the advantages and disadvantages of the single connection index approach, and:)
- Massively simplifies the system architecture
- A single index means we’re not duplicating data unnecessarily and simplifies the update process
- Doesn’t save any significant impact on the speed of the sync process (reindex is really quick!)
- Awkward request design, either having to implement API call methods for different services in the Integration service, or having to make extra “back and forth” requests between services - less “separation of concerns”?
- Less flexible if we have different filtering requirements for the same fields in different services (when calling reindex, we can specify field mapping metadata that enables different search behaviour e.g. wildcard)
Tasks to complete TIS21-3774 with this approach
...