/
Reval Legacy Data Migration Job

Reval Legacy Data Migration Job

The migration ETL is written in Spring Batch in tis-revalidation-migration service.

It consists of an ItemRreader to read data from the legacy Reval MySQL database, an ItemProcessor to map the trainee data into the format we need in the new Reval, and an ItemWriter to insert the data into new Reval MongoDB.

 

Technical Instructions are available on: https://github.com/Health-Education-England/tis-revalidation-migration

 

 

  1. Clear/drop the revalidation.recommendation collection in the Reval MongoDB

  2. The migration will start automatically once the service runs.
    To run the task manually:
    i. Find the subnet id for the AZ the database is in, e.g. 
    aws ec2 describe-subnets --output text --query "Subnets[*].SubnetId" --filters Name=tag:Name,Values=tis_preprod_private_*

    Then set the SUBNET_ID with those output subnet id result, i.e.
    export SUBNET_ID="subnet-04cbf46dd04e8435c,subnet-0482da6e63fa8d820,subnet-0bf03bdd2b401a9e9"

    ii. Find the security group to run with, e.g. 
    export SECURITY_GROUP=`aws ec2 describe-security-groups --filters Name=group-name,Values=tis_preprod* --output text --query "SecurityGroups[*].GroupId"`

    iii. Now run the task with the specific task-definition version, e.g.
    aws ecs run-task --task-definition tis-revalidation-migration:25 --cluster revalidation-preprod --network-configuration "awsvpcConfiguration={subnets=[$SUBNET_ID],securityGroups=[$SECURITY_GROUP]}"

  3. Data will be populated to the revalidation.recommendation collection in the Reval MongoDB after migration.

 

MySQL user permission:

The MySQL user reval-migration_user is used to access the source MySQL database.
Both READ and WRITE permission are needed for the revalidation database, while only READ permission is needed for the auth database.

Related content

Elastic Search Index Update Event for Connections (outdated)
Elastic Search Index Update Event for Connections (outdated)
More like this
Recommendations CDC documentDB to ElasticSearch
Recommendations CDC documentDB to ElasticSearch
Read with this
Reval Data Migration Fields Mapping (based on new Reval data schema)
Reval Data Migration Fields Mapping (based on new Reval data schema)
More like this
Elastic Search Rebuild Sync Job
Elastic Search Rebuild Sync Job
More like this
Migrate Trainee Data from TIS to TIS Self Service Infra
Migrate Trainee Data from TIS to TIS Self Service Infra
More like this
Migrating Mongo Data (2022)
Migrating Mongo Data (2022)
More like this