...
TIS Core Services (TCS)
Summary
What it does
4xP + A
Admin - Reference tables
User management
Document Management
Assessments
Summary
What it does
Generic Upload
Summary
What it does
...
Summary
What it does
| Anchor | ||||
|---|---|---|---|---|
|
Summary
A spring-batch application which runs nightly to extract data from TIS, denormalise to the National Data Warehouse (NDW) schema, and write to an interim database for the NDW to import from.
...
The application is run from a Jenkins cron (00.20 UTC 7 days a week) - here : https://build.tis.nhs.uk/jenkins/view/Prod/job/tis-ndw-etl-cron-prod/
What it does
- The application uses spring-batch framework to work through a series of extract and mapping tasks in discrete chunks.
- It works through each table in the destination database (TIS-Interim, a Microsoft SQL Server DB) in turn and batches the ETL into each table
- When starting on the next table it firsts truncates the table contents in the TIS-Interim
- It extracts data from TIS via direct database calls to the MySQL service databases (as opposed to going through the REST service layer - in order to provide adequate performance)
- It writes data to TIS-Interim using Microsoft SQLServerBulkCopy for MS SQL Server databases and JDBC Batch for other databases
- It accesses TCS, Assessments and Reference databases to assemble the required data
- The mapping is contained in discrete SQL mapping files in https://github.com/Health-Education-England/TIS-NDW-ETL/tree/master/src/main/resources/queries which the application uses to extract data from TIS and normalise it to the single table in the NDW schema
- Note: The destination schema is held in the repository for each destination table but these need to be dropped and recreated manually if they are changed, it's not flyway enabled.
- The service through spring-batch has instrumentation on its progress which it writes to the TIS-Interim database - tables Batch Job Execution and Batch Step Execution can be used to review or debug the application progress through tables and batches.