Elastic Search Rebuild Sync Job

Known Issues:

04/07/2022 Post deployment review / release washup

Running the sync job

The reval sync job can be run manually from the API Gateway for prod and preprod at /api/sync.

Services used

• tis-revalidation-integration (ECS)

• tis-revalidation-recommendation (ECS)

• tis-revalidation-connection (ECS)

• TIS-TCS (EC2)

• ElasticSearch (EC2)

• RabbitMQ (Amazon MQ)

• SQS (SQS)

Sync job data flow

1. 

   When /sync endpoint called in Integration service, the Master index is cleared and rebuilt
   

2. “syncStart“ Message sent to TCS to start extracting data

   Rabbit Queue:  
     exchange: reval.exchange
     queue: reval.queue.connection.syncstart
     routingKey: reval.connection.syncstart

3. TCS send trainee data with latest programme/curriculum information of the trainee one by one via the queue. “syncEnd“ signal will be sent together with the last trainee.

   Rabbit Queue:  
     exchange: reval.exchange
     queue: reval.queue.connection.syncdata
     routingKey: reval.connection.syncdata

   Integration service process to insert trainee data from TCS to the Master Index
   

4. When the “syncEnd” message is received by Integration, “gmcSyncStart” message is sent to Recommendation to get GMC data

   Rabbit Queue:  
     exchange: reval.exchange
     queue: reval.queue.recommendation.syncstart
     routingKey: reval.recommendation.syncstart

5. Recommendation get trainee data from “DoctorsForDB” and send it over to Integration one by one via the queue. “syncEnd“ signal will be sent together with the last trainee.

   SQS Queue:  
     Prod: tis-revalidation-sync-gmc-queue-prod
     Preprod: tis-revalidation-sync-gmc-queue-preprod

   Integration service process to insert/update (if exist) trainee data from Recommendaition to the Master Index
   
   

6. When the “syncEnd” message is received by Integration, “getMaster” message is sent to Recommendation and Connection service at the same time to get data from Master Index

   Rabbit Queue:  
     exchange: reval.exchange
     Recommendation queue: reval.queue.indexrebuildgetmastercommand.requested.recommendation
     Connection queue: reval.queue.indexrebuildgetmastercommand.requested.connection
     routingKey: reval.indexrebuildgetmastercommand.requested

7. Recommendation service cleans and rebuilds Recommendation Index, then maps and populates the trainee data to Recommendation Index
   Connection service cleans and rebuilds Connected, Disconnected, Exception Index, then maps and populates the trainee data to corresponding indexes according to business logic

[Old version]

Sync job RabbitMQ configuration