Versions Compared

Version Old Version 11 New Version 12
Changes made by

Cai Willis

Cai Willis

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

STILL WIP

Expand
titleInitial Refinement informing Design work a.k.a. Connections (... and Reval) ~Syncing~ Cache Design

Problems:
Same work as recommendations assumption, we SHOULD do the same

  • CDC and the "separate indexes"

  • Level of change: we could do minimal updates to what is implemented now

  • “Logic” Implementing in different languages / at different times

    • Principal: Do it once!

    • Logic in writing? "just the mapping"? Mapping is logic... still have 2 processes

    • Logic

  • Filters would need to be duplicated if using different indexes:

    • Implementation-dependent

  • Hypothesis: Performance of querying a single index vs. separate indexes

  • Hypothesis: Performance of writing to several indexes

• Hypothesis: It would all be much simpler if we were copying all programme memberships separately

  • What level of mapping is required between the base (master index) and connections?

    • Next to none... probably not actually necessary

Principals:

  • Write it (code) once - Questions over this. How much does this run counter to Microservice Design

  • Unnecessary redundancy. Data stored in several places.

    • When should we have copies of the data?:

Solutions:

  • Assumption:

  • Fields available in the Base (Master) Index similar enough to the fields needed across all connections screens.

  • Data Design

    • Steve's query on schema design

    • When do we Specifically do we need to persist:

      • connection status field, given we have gmcDesignatedBody

      • Programme Memberships & Whether a discrepancy is hidden

Three ways of caching data, the first might not be viable because of connection specific info:

  • Single Index across REVAL? (Probably not, given the likely need of holding connection specific info?)

  • Single Index for Connections

  • Single Index for each tab

...

The approach of “pre-sorting” the data was also fine before as the exact same code was used for CDC and the ES Resync job. However, in order to repeat the massive time saving we achieved in

Jira Legacy
serverSystem JIRA
serverId4c843cd5-e5a9-329d-ae88-66091fcfe3c7
keyTIS21-3416
for the Connections service, we have to use the Elasticsearch “reindex” operation, which means we would have to duplicate the logic we have written in Java now in ES query language as part of the reindex request - and then maintain both separately.

Summary

  • Having multiple indexes makes GET requests simpler

    • performance has been raised as a potential benefit, but when more complex queries on large data sets take less than a second it’s questionable how much benefit this would really give.

  • Multiple indexes means duplicating data

  • Multiple indexes makes requires multiple updates for a single data change

  • Because we have separate CDC and Resync processes, and because the Java approach is prohibitively slow for the Resync process, we would have to write and maintain the business logic in separate places in separate languages

Tasks to complete TIS21-3774 with this approach

...

Drawio
mVer2
zoom1
simple0
inComment0
pageId3696689344
custContentId3695936234
lbox1
diagramDisplayNameUntitled Diagram-1674068170357.drawio
contentVer1
revision1
baseUrlhttps://hee-tis.atlassian.net/wiki
diagramNameUntitled Diagram-1674068170357.drawio
pCenter0
width670
links
tbstyle
height831

For this approach, in the ES Resync Job we remove the “pre-sorting” step featured in the current approach and just perform a straightforward 1:1 copy (excluding schema discussion for now) from masterdoctorindex to a single connectionIndex, in the same way we do things for recommendationindex,

We also remove the “pre-sorting” from the CDC process and once again do the same as in recommendations - a simple 1:1 update.

Instead of “pre-sorting”, we now implemented the business logic as part of an Elasticsearch query on the GET side of things.

This should be a fairly straightforward conversion, for example where we currently pre-sort with Java if(<conditions for discrepancy>) we would instead GET with a Where <fieldValue> = <condition for discrepancy> in Elasticsearch.

Tasks to complete TIS21-3774 with this approach

...