Versions Compared

Version Old Version 9 New Version 10
Changes made by

Cai Willis

Cai Willis

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

STILL WIP

Expand
titleInitial Refinement informing Design work a.k.a. Connections (... and Reval) ~Syncing~ Cache Design

Problems:
Same work as recommendations assumption, we SHOULD do the same

  • CDC and the "separate indexes"

  • Level of change: we could do minimal updates to what is implemented now

  • “Logic” Implementing in different languages / at different times

    • Principal: Do it once!

    • Logic in writing? "just the mapping"? Mapping is logic... still have 2 processes

    • Logic

  • Filters would need to be duplicated if using different indexes:

    • Implementation-dependent

  • Hypothesis: Performance of querying a single index vs. separate indexes

  • Hypothesis: Performance of writing to several indexes

• Hypothesis: It would all be much simpler if we were copying all programme memberships separately

  • What level of mapping is required between the base (master index) and connections?

    • Next to none... probably not actually necessary

Principals:

  • Write it (code) once - Questions over this. How much does this run counter to Microservice Design

  • Unnecessary redundancy. Data stored in several places.

    • When should we have copies of the data?:

Solutions:

  • Assumption:

  • Fields available in the Base (Master) Index similar enough to the fields needed across all connections screens.

  • Data Design

    • Steve's query on schema design

    • When do we Specifically do we need to persist:

      • connection status field, given we have gmcDesignatedBody

      • Programme Memberships & Whether a discrepancy is hidden

Three ways of caching data, the first might not be viable because of connection specific info:

  • Single Index across REVAL? (Probably not, given the likely need of holding connection specific info?)

  • Single Index for Connections

  • Single Index for each tab

...

The approach of “pre-sorting” the data was also fine before as the exact same code was used for CDC and the ES Resync job. However, in order to repeat the massive time saving we achieved in

Jira Legacy
serverSystem JIRA
serverId4c843cd5-e5a9-329d-ae88-66091fcfe3c7
keyTIS21-3416
for the Connections service, we have to use the Elasticsearch “reindex” operation, which means we would have to duplicate the logic we have written in Java now in ES query language as part of the reindex request - and then maintain both separately.

Summary

  • Having multiple indexes makes GET requests simpler

    • performance has been raised as a potential benefit, but when more complex queries on large data sets take less than a second it’s questionable how much benefit this would really give.

  • Multiple indexes means duplicating data

  • Multiple indexes makes requires multiple updates for a single data change

  • Because we have separate CDC and Resync processes, and because the Java approach is prohibitively slow for the Resync process, we would have to write and maintain the business logic in separate places in separate languages

Tasks to complete TIS21-3774 with this approach

  1. Copy implementation for reindexing used by recommendations

  2. Implement elasticsearch query to match java logic in CDC process for use as part of reindex api call

  3. Clearly document that any changes made to the CDC logic in Connections must be repeated in Integration (is there something fancy we can do in github?)

  4. BONUS: figure out if we can run the reindex for connections and recommendation separately (might be impractical)?

...