...

...

...

...

...

...

...

...

Known Issues:

04/07/2022 Post deployment review / release washup

Running the sync job

The reval sync job can be run manually from the API Gateway for prod and preprod at /api/sync.

Services used

• tis-revalidation-integration (ECS)

• tis-revalidation-recommendation (ECS)

• tis-revalidation-connection (ECS)

• TIS-TCS (ECS)

• ElasticSearch (Amazon OpenSearch Service)

• RabbitMQ (Amazon MQ)

• SQS (SQS)

• Revalidation DocumentDB

Sync job data flow

1. When /sync endpoint called in Integration service, the Master index is cleared and rebuilt
   

2. “syncStart“ Message sent to TCS to start extracting data

   Code Block
   Rabbit Queue: exchange: reval.exchange queue: reval.queue.connection.syncstart routingKey: reval.connection.syncstart

3. TCS send trainee data with latest programme/curriculum information of the trainee one by one via the queue. “syncEnd“ signal will be sent following the last trainee.

   Code Block
   Rabbit Queue: exchange: reval.exchange queue: reval.queue.connection.syncdata routingKey: reval.connection.syncdata

   Integration service process to insert trainee data from TCS to the Master Index
   

4. When the “syncEnd” message is received by Integration, “gmcSyncStart” message is sent to Recommendation to get GMC data from DocumentDB.

   Code Block
   Rabbit Queue: exchange: reval.exchange queue: reval.queue.recommendation.syncstart routingKey: reval.recommendation.syncstart

5. Recommendation get trainee data from “DoctorsForDB” and send it over to Integration one by one via the queue. “syncEnd“ signal will be sent following the last trainee.

   Code Block
   SQS Queue: Prod: tis-revalidation-sync-gmc-queue-prod Preprod: tis-revalidation-sync-gmc-queue-preprod

   Integration service process to insert/update (if exist) trainee data from Recommendation (DoctorsForDb data + the latest recommendation’s gmc outcome) to the Master Index.
   

6. When the “syncEnd“ message is received by Integration, it backup the current elasticsearch recommendationindex and reindex from masterdoctorindex to a new recommendationindex, which ensure all updates in masterdoctorindex are synchronised to recommendationindex.
   As for how to trigger the reindexing, please refer to Reindex from masterdoctorindex to recommendation index .

Steps to be continued

The step 6 above was:

When the “syncEnd” message is received by Integration, “getMaster” message is sent to Recommendation and Connection service at the same time to get data from Master Index

Rabbit Queue:  
  exchange: reval.exchange
  Recommendation queue: reval.queue.indexrebuildgetmastercommand.requested.recommendation
  Connection queue: reval.queue.indexrebuildgetmastercommand.requested.connection
  routingKey: reval.indexrebuildgetmastercommand.requested

As of this PR this step no longer happens. This part of the sync process is an incredibly inefficient and roundabout lookup for each individual doctor from the recommendation service to integration and back to recommendation. This a) takes a ridiculous amount of time and b) tends to crash mongoDB and/or the CDC lamdba, potenitally incurring costs

...

We have implemented the automatic process to reindex from masterdoctorindex to recommendationindex. This ticket is for considering reindexing from masterdoctorindex to all 3 Connection indices.

[Old version 1]

...

[Old version 2]

...