Versions Compared

Old Version 37

changes.mady.by.user Yafang Deng

Saved on

compared with

New Version Current

changes.mady.by.user Joseph (Pepe) Kelly

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Added missing Sync jobs

...

  • ESR-ETL receives and sends different files from and to ESR. The details of files and data sets are outlined here: ESR - Datasets (Historical)
  • It receives RMT files in changed (RMC) or full (RMF) mode. It processes and stores POR and POS records from the files.
    • The POR and POS records are persisted in ESR database (mySQL) using ESR service.
  • Each of the above files are received per deanery/local office and all the files produced are as well per deanery.
  • For each of the deanery posts ESR requests corresponding APP file containing applicant records in placement for the post is generated. The following restriction applies for the selection:
    • Only the future applicant placements are selected.
    • The selected future placements should have a start date between 2 days from that day to 3 months. 
    • It uses TCS service to fetch the post, placement and trainee details.
    • The applicant records are stored in ESR database using ESR service.
  • A notification file (DNF - full notifications and DNC - changed notifications) is sent per deanery on a daily basis.
    • The notification records tracks all the events happening in TIS and creates different notification types outlined in the ESR spec. This is a way of updating ESR about the changes happening in TIS.
    • The TIS notification events are tracked and stored in TCS database in the EsrNotification table using the TCS service. This records only the TIS contents.
    • All of the final notification records generated are stored in ESR database using the ESR service.
  • An acknowledgment is generated for each RMT file processed and similarly an acknowledgment file (APC) is received from ESR for each APP file sent.
    • These APC records are stored in the ESR database using the ESR service.
  • The ESR service and ESR-ETL use profile/keycloak for user authentication and have an ESR specific user configured for the same.

SYNC (Service)

Summary

  • TIS-SYNC is a container for background and long-running 'batch' jobs that synchronise data and update cached values.
  • It currently is one package with a dependency of TCS persistence and reference client.
  • It currently comprises 5 separate jobs (listed in "What it does") which assume they are run in a particular order.
  • The jobs are currently duplicated in TCS but are scheduled to run a minute earlier by default

...

  • The default schedules for the cron jobs can be found here: ETL Timings

What it does

There are 5 7 Jobs:

  • PersonPlacementEmployingBodyTrustJob: Truncates and repopulates the list of connections between trainees and the employingBody trust. It writes data into the database table tcs.PersonTrust.
  • PersonPlacementTrainingBodyTrustJob: Repopulates the list of connections between trainees and the trainingBody trust. It also writes into tcs.PersonTrust, and is triggered after PersonPlacementEmployingBodyTrustJob, so the data wouldn't be truncated at first.
  • PostEmployingBodyTrustJob: Truncates and repopulates the list of connections between posts and the employingBody trust. It writes data into the database tcs.PostTrust.
  • PostTrainingBodyTrustJob: Repopulates the list of connections between posts and the trainingBody trust. It also writes into tcs.PostTrust, and is triggered after PostTrainingBodyTrustJob, so the data wouldn't be truncated at first.
  • PersonElasticSearchSyncJob: Truncates index of persons in ElasticSearch and creates a new index of persons. Then repopulates the documents of PersonViews from TCS-Service and saves these documents in ElasticSearch. PersonOwnerJob: This triggers a stored procedure that updates the local office for Person records.
  • PersonPlacementEmployingBodyTrustJob: After running this job, tcs.PersonTrust table will be truncated and resynchronised with populated EmployingBody Trust data.
  • PersonPlacementTrainingBodyTrustJob: Will also synchronise populated trainingBody Trust data into tcs.PersonTrust table.
  • PersonRecordStatusJob: Updates the 'training status' (DB field: status) based on a ProgrammeMembership starting today or ending yesterday.
  • PostEmployingBodyTrustJob: After running this job, tcs.PostTrust table will be truncated and resynchronised with populated EmployingBody Trust data.
  • PostTrainingBodyTrustJob: Will also synchronise populated trainingBody Trust data into tcs.PostTrust table.
  • PersonElasticSearchSyncJob:  Will truncate index of persons in ElasticSearch and then create new index and save documents of populated PersonView for ElasticSearch.

Note:

  • PersonPlacementTrainingBodyTrustJob should be triggered after PersonPlacementEmployingBodyTrustJob.
  • PostTrainingBodyTrustJob should be triggered after PostEmployingBodyTrustJob.PersonElasticSearchSyncJob should be triggered after PersonPlacementEmployingBodyTrustJob and PersonPlacementTrainingBodyTrustJob are finished.