...
- This is a manual job only
- Can deploy any service in any environment
- Deploys the version in platform.yml
...
- Run by other jenkins jobs
- Creates JAR and Docker inages
- Put them into the registry
...
- This is the main job for esr
- It is triggered by a push into github
- It creates the JAR and Docker image by running service-pipeline
- When FINALTAG is set, it deploys esr-etl into dev
...
Project | Jenkins job | Purpose | SCM triggered | cron | environment | |||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
ANY | service-env-deploy |
| no | Any | ||||||||||||||||||||||||||||||||||
ANY | service-pipeline |
| no | None/Dev | ||||||||||||||||||||||||||||||||||
ESR | esr |
| no | None/Dev | ||||||||||||||||||||||||||||||||||
ESR-ETL | tis-esr-etl |
| no | None/Dev | ||||||||||||||||||||||||||||||||||
tis-esr-parameterized |
| no | Any | |||||||||||||||||||||||||||||||||||
| "yes" | |||||||||||||||||||||||||||||||||||||
tis-esr-etl-param-apc-import |
|
| stage | |||||||||||||||||||||||||||||||||||
tis-esr-etl-param-applicant-load |
|
| stage | |||||||||||||||||||||||||||||||||||
tis-esr-etl-param-applicant-export |
|
| stage | |||||||||||||||||||||||||||||||||||
tis-esr-etl-param-notification-export |
|
| stage | |||||||||||||||||||||||||||||||||||
tis-esr-etl-param-notification-daily-load |
|
| stage | |||||||||||||||||||||||||||||||||||
esr-etl | disabled because tis-esr-etl does the same thing | |||||||||||||||||||||||||||||||||||||
NDW-ETL | ndw-etl |
| no | None/Dev | ||||||||||||||||||||||||||||||||||
tis-ndw-etl-parameterized |
|
| Prod | |||||||||||||||||||||||||||||||||||
chris-tis-ndw |
| |||||||||||||||||||||||||||||||||||||
service-pipeline-ndw-etl | disabled because this was a poor attempt to change service-pipeline | |||||||||||||||||||||||||||||||||||||
Generic Upload | tis-generic-upload-parameterized |
| no | Any | ||||||||||||||||||||||||||||||||||
tis-generic-upload |
| no | None/Dev | |||||||||||||||||||||||||||||||||||
TCS | tcs |
| ||||||||||||||||||||||||||||||||||||
tcs-paramameterized | disabled because this was a poor attempt to change service-pipeline | |||||||||||||||||||||||||||||||||||||
admins-ui | admins-ui |
| ||||||||||||||||||||||||||||||||||||
TIS-ADMINS-UI |
| |||||||||||||||||||||||||||||||||||||
admins-ui-e2e |
| |||||||||||||||||||||||||||||||||||||
admins-ui-uat |
| |||||||||||||||||||||||||||||||||||||
chris-admins-ui |
| |||||||||||||||||||||||||||||||||||||
TIS-ADMINS-UI |
| |||||||||||||||||||||||||||||||||||||
reference | reference |
| ||||||||||||||||||||||||||||||||||||
profile | profile |
| ||||||||||||||||||||||||||||||||||||
profile-parent |
| |||||||||||||||||||||||||||||||||||||
assessments | assessments |
| ||||||||||||||||||||||||||||||||||||
audit | audit-logs-backup-dev |
| ||||||||||||||||||||||||||||||||||||
audit-logs-backup-prod |
| |||||||||||||||||||||||||||||||||||||
revalidation | revalidation |
| yes | |||||||||||||||||||||||||||||||||||
intrepid-reval-etl-dev/stage/uidev |
| no | ||||||||||||||||||||||||||||||||||||
| no | |||||||||||||||||||||||||||||||||||||
intrepid-reval-etl-all-prod |
|
| ||||||||||||||||||||||||||||||||||||
| ||||||||||||||||||||||||||||||||||||||
ci_test | ci_test |
| ||||||||||||||||||||||||||||||||||||
release-tag | release-tag |
| ||||||||||||||||||||||||||||||||||||
fayaz-pl | fayaz-pl |
| ||||||||||||||||||||||||||||||||||||
Trainee-UI | trainee-ui |
| yes | |||||||||||||||||||||||||||||||||||
Notifications | notifications |
| yes | |||||||||||||||||||||||||||||||||||
common | common |
| ||||||||||||||||||||||||||||||||||||
Intrepid | intrepid-load |
| ||||||||||||||||||||||||||||||||||||
intrepid-dr-model |
| yes | ||||||||||||||||||||||||||||||||||||
intrepid-dr-tools |
| yes | no | |||||||||||||||||||||||||||||||||||
intrepid-load-etl |
| no | ||||||||||||||||||||||||||||||||||||
intrepid-anonymizer |
|
| no |
| intrepid-extract-all |
| no | no |
| no |
|
| no | no |
|
Code Block |
---|
Anonymise intrepid data to populate the anonymized mirror |
- This is the main job for intrepid-anonymizer
- It is triggered by a push into TIS-DATA-ANONYMIZER github repo
- It creates the JAR and Docker image by running service-pipeline
- When FINALTAG is set, it deploys generic-upload into dev
- last run 2017 January 9th
- This is the main job for intrepid-core-model
- It is triggered by a push into TIS-INTREPID-CORE-MODEL github repo
- It creates the JAR and Docker image by running mvn2docker
- triggered by intrepid-etl-pipeine
- runs ansible playbook ansible/tasks/intrepid-load.yml in STAGE
Quote:
Code Block |
---|
Runs intrepid-extract and load each morning. Intrepid extract: grabs data from intrepid places in into CSV files (db-to-csv). Intrepid remap: changes column names on CSV files (intrepid-remap-col-structure) - we could remove it Intrepid load: grabs CSV data and places it into the mirror (csv-diff) creates a delta file and then uploads to db Intrepid conosolidated etl: points to the mirror and populates the TIS system
|
- pipeline script from SCM
- SCM: TIS-DEVOPS
- script: jenkins/intrepid-etl-prod.groovy
- jenkins/intrepid-etl-prod.groovy can run intrepid-extract-all, intrepid-remapper-structure, intrepid-load, intrepid-extract-upload, intrepid-compare-stats
- }}
- runs ansible/tasks/intrepid-anonymizer-new.yml in STAGE
- never run
Code Block |
---|
Anonymise intrepid data to populate the anonymized mirror |
- This is the main job for intrepid-anonymizer
- It is triggered by a push into TIS-DATA-ANONYMIZER github repo
- It creates the JAR and Docker image by running service-pipeline
- When FINALTAG is set, it deploys generic-upload into dev
- last run 2017 January 9th
- This is the main job for intrepid-core-model
- It is triggered by a push into TIS-INTREPID-CORE-MODEL github repo
- It creates the JAR and Docker image by running mvn2docker
- triggered by intrepid-etl-pipeine
- runs ansible playbook ansible/tasks/intrepid-load.yml in STAGE
Quote:
Code Block |
---|
Runs intrepid-extract and load each morning. Intrepid extract: grabs data from intrepid places in into CSV files (db-to-csv). Intrepid remap: changes column names on CSV files (intrepid-remap-col-structure) - we could remove it Intrepid load: grabs CSV data and places it into the mirror (csv-diff) creates a delta file and then uploads to db Intrepid conosolidated etl: points to the mirror and populates the TIS system
|
- pipeline script from SCM
- SCM: TIS-DEVOPS
- script: jenkins/intrepid-etl-prod.groovy
- jenkins/intrepid-etl-prod.groovy can run intrepid-extract-all, intrepid-remapper-structure, intrepid-load, intrepid-extract-upload, intrepid-compare-stats
- pipeline script from SCM
- SCM: TIS-DEVOPS
- script: jenkins/intrepid-etl-prod-new.groovy
- jenkins/intrepid-etl-prod-new.groovy runs ansible/tasks/intrepid-extract-new.yml with intrepid_dr_settings=<dr2|mirror-prod> schema=R1 mirror_settings=mirror-prod
batch_size=20
- 2h45
- daily
- runs ansible/tasks/intrepid-extract-all.yml in PROD
- ansible/tasks/intrepid-extract-all.yml: deploys docker image repository.tis.nhs.uk:5000/hee/intrepid-extract:{{ versions['intrepid-extract'] }}
- disabled, last successful run 2017 Nov 30th
- runs ansible/tasks/intrepid-extract-new.yml on ETL hosts in PROD
- midnight
- daily
- This is the main job for notifications
- It is triggered by a push into TIS-INTREPID-DR-ETL github repo
- It creates the JAR and Docker image by running service-pipeline
- When FINALTAG is set, it deploys generic-upload into dev
- triggered by intrepid-etl-pipeine
- runs ansible/tasks/intrepid-compare-stats.yml in PROD
- runs mvn on ETL hosts to create jarfile, then java -jar jarfile
- compares count of items in the prod dr against the mirror
- disabled, last run 2017 Nov 9th
- runs ansible/tasks/intrepid-compare-stats.yml in PROD, with DEVOPS_BASE=/datadrive/data/fayaz/master/devops
- runs ansible/tasks/intrepid-extract-clean.yml --limit 10.140.0.1 in PROD on ETL
- runs automatically
- removes old files from /datadrive/data/intrepid
- midnight
- daily
- runs ansible/tasks/intrepid-reval-etl-all.yml in DEV
- runs ansible/tasks/intrepid-reval-etl-all.yml in STAGE
- runs ansible/tasks/intrepid-reval-etl-all.yml in UIDEV
- It is triggered by a push into TIS-INTREPIDDB-DRTO-ETLCSV github repo
- It creates the JAR and Docker image by running service-pipeline
- When FINALTAG is set, it deploys generic-upload into dev
- triggered by intrepid-etl-pipeine
- runs ansible/tasks/intrepid-compare-stats.yml in PROD
- runs mvn on ETL hosts to create jarfile, then java -jar jarfile
- compares count of items in the prod dr against the mirror
- disabled, last run 2017 Nov 9th
- runs ansible/tasks/intrepid-compare-stats.yml in PROD, with DEVOPS_BASE=/datadrive/data/fayaz/master/devops
- triggers the job intrepid-extract-docker
- It is triggered by a push into TIS-DB-TO-CSV github repo
- It can be triggered by db-to-csv-parent
- It runs mvn clean install -e on TIS-DB-TO-CSV/db-to-csv pom.xml file
- It runs ansible/tasks/spring-boot-build.yml in DEV on localhost
- it runs mvn2docker to create an image named intrepid-extract
- When FINALTAG is set, it deploys generic-upload into dev ????
- triggered by intrepid-etl-pipeine
- compresses and uploads intrepid databases to azure blob storage ???
- runs ansible/tasks/intrepid-extract-cleanupload.yml --limit 10.140.0.1 in PROD on ETL
- runs automatically
- removes old files from ETL hosts
- runs ansible/tasks/intrepid-reval-etl-all.yml in DEV
- runs ansible/tasks/intrepid-reval-etl-all.yml in STAGE
- uses export DEVOPS_BASE=/datadrive/data/intrepid
- midnight
- daily
- fayaz/master/devops in PROD
- last run 2017 November 9th
- deploys docker image repository.tis.nhs.uk:5000/hee/intrepid-extract:{{ versions['intrepid-extract'] }}
- extracts intrepid databases
Code Block |
---|
compares the schema csv exports daily to make sure that the structure hasn't changed. |
- watches intrepid-etl-pipeine as a trigger
- runs ansible/tasks/intrepid-
- schema-
- validation.yml
- It is triggered by a push into TIS-DB-TO-CSV github repo
- triggers the job intrepid-extract-docker
- It is triggered by a push into TIS-DB-TO-CSV github repo
- It can be triggered by db-to-csv-parent
- It runs mvn clean install -e on TIS-DB-TO-CSV/db-to-csv pom.xml file
- It runs ansible/tasks/spring-boot-build.yml in DEV on localhost
- it runs mvn2docker to create an image named intrepid-extract
- When FINALTAG is set, it deploys generic-upload into dev ????
- triggered by intrepid-etl-pipeine
- compresses and uploads intrepid databases to azure blob storage ???
- runs ansible/tasks/intrepid-extract-upload.yml in PROD on ETL hosts
- on Localhost
- runs mvn on ETL hosts to create jarfile, then java -jar jarfile
Code Block |
---|
Runs the second part of the pipeline for intrepid-etl on the mirror stage
Intrepid data anonymizer - takes data from ssh hee-build-azure -> /datadrive/data/intrepid/prod/ (full+delta) anonymises it then places it into /datadrive/data/intrepid/stage/ (full+delta)
Intepid Load Mirror -> takes data from /datadrive/data/intrepid/stage/ , creates a delta file then loads it into mirror stage
Intrepid revalidation stage -> the reval etl on stage |
- watches intrepid-etl-pipeine as a trigger
- pipeline script from SCM
- SCM: TIS-DEVOPS
- script: jenkins/intrepid-etl-stage.groovy
- jenkins/intrepid-etl-stage.groovy runs jenkins jobs intrepid-anonymizer, intrepid-load-stage, intrepid-reval-etl-stage
- runs ansible/tasks/intrepid-remapper-structure.yml with inventory localhost
- ansible/tasks/intrepid-remapper-structure.yml: runs mvn on ETL hosts to create jarfile, then java -jar jarfile
- can be run by intrepid-etl-pipeine
- disabled, last successful run 2017 Nov 9th
- runs ansible/tasks/intrepid-remapper-structure.yml in PROD
- ansible/tasks/intrepid-remapper-structure.yml: runs mvn on ETL hosts to create jarfile, then java -jar jarfile
...