Introduction

Build out a high-level description of each service to complement the Application Architecture diagram

...

• ESR-ETL receives and sends different files from and to ESR. The details of files and data sets are outlined here: ESR - Datasets (Historical)
• It receives RMT files in changed (RMC) or full (RMF) mode. It processes and stores POR and POS records from the files.
  • The POR and POS records are persisted in ESR database (mySQL) using ESR service.
• Each of the above files are received per deanery/local office and all the files produced are as well per deanery.
• For each of the deanery posts ESR requests corresponding APP file containing applicant records in placement for the post is generated. The following restriction applies for the selection:
  • Only the future applicant placements are selected.
  • The selected future placements should have a start date between 2 days from that day to 3 months. 
  • It uses TCS service to fetch the post, placement and trainee details.
  • The applicant records are stored in ESR database using ESR service.
• A notification file (DNF - full notifications and DNC - changed notifications) is sent per deanery on a daily basis.
  • The notification records tracks all the events happening in TIS and creates different notification types outlined in the ESR spec. This is a way of updating ESR about the changes happening in TIS.
  • The TIS notification events are tracked and stored in TCS database in the EsrNotification table using the TCS service. This records only the TIS contents.
  • All of the final notification records generated are stored in ESR database using the ESR service.
• An acknowledgement acknowledgment is generated for each RMT file processed and similarly an acknowledgement acknowledgment file (APC) is received from ESR for each APP file sent.
  • These APC records are stored in the ESR database using the ESR service.
• The ESR service and ESR-ETL use profile/keycloak for user authentication and have an ESR specific user configured for the same.

SYNC (Service)

Summary

• TIS-SYNC is a container for background and long-running 'batch' jobs that synchronise data and update cached values.
• It currently .
  • ESR ETL: A spring-boot application which runs on a daily basis multiple times with different load types. 
  • ESR service: A spring-boot application running as a service and is used to persist and retrieve ESR data.

• The overall data flow and systems integration can be seen here: ESR-TIS Inbound and Outbound flows (MVP)
• A simplified flow and schedule details are shown below:

Where and When

The applications have their own github repositores.

• ESR Serviceis one package with a dependency of TCS persistence and reference client.
• It currently comprises 5 separate jobs (listed in "What it does") which assume they are run in a particular order.
• The jobs are currently duplicated in TCS but are scheduled to run a minute earlier by default

Where and When

SCM:

• TIS-SYNC: https://github.com/Health-Education-England/TIS-ESRSYNC
• ESR ETL: https://github.com/Health-Education-England/TIS-ESR-ETL

The ESR-ETL application is run from a Jenkins cron: 

...

Deployment:

• Currently deployed on

The 5 jobs are triggered by cron schedule on methods in a Java class for each job.  The schedule can be set in the container by defining the cron as an environment variable or startup parameter.

• The default schedules for the cron jobs can be found here: ESR Schedules
• There is a FTP sync process which takes care of syncing files between Azure(TIS owned) and FTP server (ESR owned). 
  • This process is scheduled as a Jenkins job and runs multiple times a day and can be found here: https://build.tis.nhs.uk/jenkins/job/tis-esr-ftpsync-cron-prod/
  • The schedule details are available under: ESR Schedules

What it does

• ESR-ETL receives and sends different files from and to ESR. The details of files and data sets are outlined here: ESR - Datasets
• It receives RMT files in changed (RMC) or full (RMF) mode. It processes and stores POR and POS records from the files.
  • The POR and POS records are persisted in ESR database (mySQL) using ESR service.
• Each of the above files are received per deanery/local office and all the files produced are as well per deanery.
• For each of the deanery posts ESR requests corresponding APP file containing applicant records in placement for the post is generated. The following restriction applies for the selection:
  • Only the future applicant placements are selected.
  • The selected future placements should have a start date between 2 days from that day to 3 months. 
  • It uses TCS service to fetch the post, placement and trainee details.
  • The applicant records are stored in ESR database using ESR service.
• A notification file (DNF - full notifications and DNC - changed notifications) is sent per deanery on a daily basis.
  • The notification records tracks all the events happening in TIS and creates different notification types outlined in the ESR spec. This is a way of updating ESR about the changes happening in TIS.
  • The TIS notification events are tracked and stored in TCS database in the EsrNotification table using the TCS service. This records only the TIS contents.
  • All of the final notification records generated are stored in ESR database using the ESR service.
• An acknowledgement is generated for each RMT file processed and similarly an acknowledgement file (APC) is received from ESR for each APP file sent.
  • These APC records are stored in the ESR database using the ESR service.
• The ESR service and ESR-ETL use profile/keycloak for user authentication and have an ESR specific user configured for the sameETL Timings

What it does

There are 7 Jobs:

• PersonOwnerJob: This triggers a stored procedure that updates the local office for Person records.
• PersonPlacementEmployingBodyTrustJob: After running this job, tcs.PersonTrust table will be truncated and resynchronised with populated EmployingBody Trust data.
• PersonPlacementTrainingBodyTrustJob: Will also synchronise populated trainingBody Trust data into tcs.PersonTrust table.
• PersonRecordStatusJob: Updates the 'training status' (DB field: status) based on a ProgrammeMembership starting today or ending yesterday.
• PostEmployingBodyTrustJob: After running this job, tcs.PostTrust table will be truncated and resynchronised with populated EmployingBody Trust data.
• PostTrainingBodyTrustJob: Will also synchronise populated trainingBody Trust data into tcs.PostTrust table.
• PersonElasticSearchSyncJob:  Will truncate index of persons in ElasticSearch and then create new index and save documents of populated PersonView for ElasticSearch.

Note:

• PersonPlacementTrainingBodyTrustJob should be triggered after PersonPlacementEmployingBodyTrustJob.
• PostTrainingBodyTrustJob should be triggered after PostEmployingBodyTrustJob.