Reval Legacy Data Migration Job

The migration ETL is written in Spring Batch in tis-revalidation-migration service.

It consists of an ItemRreader to read data from the legacy Reval MySQL database, an ItemProcessor to map the trainee data into the format we need in the new Reval, and an ItemWriter to insert the data into new Reval MongoDB.

Technical Instructions are available on: GitHub - Health-Education-England/tis-revalidation-migration: A historical data migration ETL for Revalidation.

1. Clear/drop the revalidation.recommendation collection in the Reval MongoDB
   
   

2. The migration will start automatically once the service runs.
   To run the task manually:
   i. Find the subnet id for the AZ the database is in, e.g. 
   aws ec2 describe-subnets --output text --query "Subnets[*].SubnetId" --filters Name=tag:Name,Values=tis_preprod_private_*
   
   Then set the SUBNET_ID with those output subnet id result, i.e.
   export SUBNET_ID="subnet-04cbf46dd04e8435c,subnet-0482da6e63fa8d820,subnet-0bf03bdd2b401a9e9"
   
   ii. Find the security group to run with, e.g. 
   export SECURITY_GROUP=`aws ec2 describe-security-groups --filters Name=group-name,Values=tis_preprod* --output text --query "SecurityGroups[*].GroupId"`
   
   iii. Now run the task with the specific task-definition version, e.g.
   aws ecs run-task --task-definition tis-revalidation-migration:25 --cluster revalidation-preprod --network-configuration "awsvpcConfiguration={subnets=[$SUBNET_ID],securityGroups=[$SECURITY_GROUP]}"
   
   

3. Data will be populated to the revalidation.recommendation collection in the Reval MongoDB after migration.

MySQL user permission:

The MySQL user reval-migration_user is used to access the source MySQL database.
Both READ and WRITE permission are needed for the revalidation database, while only READ permission is needed for the auth database.