...
- This is a manual job only
- Can deploy any service in any environment
- Deploys the version in platform.yml
...
- Run by other jenkins jobs
- Creates JAR and Docker inages
- Put them into the registry
...
- This is the main job for esr
- It is triggered by a push into github
- It creates the JAR and Docker image by running service-pipeline
- When FINALTAG is set, it deploys esr-etl into dev
...
- This is the main job for esr
- It is triggered by a push into github
- It creates the JAR and Docker image by running service-pipeline
- When FINALTAG is set, it deploys esr-etl into dev
...
- This is for manual run only
- One runs it to pass a parameter like the date or the type of load needed
- It does not create any JAR file or docker image
- It just deploys whatever is in platform.yml into the selected environment
- One can pass any parameter to the job, unlike service-env-deploy
...
...
- This is a cron job in jenkins
- It runs ansible/esr-etl.yml with --extra-vars="LOAD_TYPE=${LOAD_TYPE}"
- Depending on the time, the value of LOAD_TYPE varies
- This deprecates the below tis-esr-param-XXXXX jenkins jobs
...
- This is a cron job in jenkins
- It runs tis-esr-parameterized with LOAD_TYPE=apc-import
- Disabled while tis-esr-etl-param-cron is being tested
...
- 20h00
- daily
...
- This is a cron job in jenkins
- It runs tis-esr-parameterized with LOAD_TYPE=applicant-load
- Disabled while tis-esr-etl-param-cron is being tested
...
- 15h00
- daily
...
- This is a cron job in jenkins
- It runs tis-esr-parameterized with LOAD_TYPE=applicant-export
- Disabled while tis-esr-etl-param-cron is being tested
...
- 17h00
- daily
...
- This is a cron job in jenkins
- It runs tis-esr-parameterized with LOAD_TYPE=notification-export
- Disabled while tis-esr-etl-param-cron is being tested
...
- 17h00
- daily
...
- This is a cron job in jenkins
- It runs tis-esr-parameterized with LOAD_TYPE=notification-export
- Disabled while tis-esr-etl-param-cron is being tested
...
- 02h00
- daily
...
- This is the main job for NDW-ETL
- It is triggered by a push into github
- It creates the JAR and Docker image by running service-pipeline
- When FINALTAG is set, it deploys esr-etl into dev
...
- This is a cron job in Jenkins
- It can take a parameter: the environment
- It will by default run in prod
...
- 07h00
- daily
...
- Uses jenkinsfile
- Can run against all our github repositories
- configured to use tis-ndw-etl github repo
...
- This is for manual run only
- One runs it to pass a DEBUG parameter which allows one to attach to port 5005TCP
- It does not create any JAR file or docker image
- It just deploys whatever is in platform.yml into the selected environment
- One can pass the debug parameter to the job, unlike service-env-deploy
...
- This is the main job for generic-upload
- It is triggered by a push into github
- It creates the JAR and Docker image by running service-pipeline
- When FINALTAG is set, it deploys generic-upload into dev
...
- This is the main job for tcs
- It is triggered by a push into github
- It creates the JAR and Docker image by running service-pipeline
- When FINALTAG is set, it deploys generic-upload into dev
...
- This is the main job for admins-ui
- It is triggered by a push into TIS-ADMINS-UI github repo
- It creates the JAR and Docker image by running service-pipeline
- When FINALTAG is set, it deploys generic-upload into dev
...
- Uses jenkinsfile
- Can run against all our github repositories
- configured to use ndw-etl github repo
...
- runs End to End tests
- active but not a single successfull run
- last issue: access to selenium server http://hub-cloud.browserstack.com/wd/hub
- DECISION REQUIRED
...
- This is a clone of the main job for admins-ui
- It used to be triggered by a push into TIS-ADMINS-UI github repo
- It creates the JAR and Docker image by running service-pipeline
- When FINALTAG is set, it deploys generic-upload into dev
- DECISION REQUIRED - last run was on 2018 March 7th
...
- this is a different implementation of a pipeline
- It uses groovy pipeline directly implemented in the GUI
- It never succeded, last run was on 2018 March 18th
- Last error: java.lang.NoSuchMethodError: No such DSL method 'post' found among steps
- DECISION REQUIRED
...
- Uses jenkinsfile
- Can run against all our github repositories
- configured to use TIS-ADMINS-UI github repo
...
- This is the main job for reference
- It is triggered by a push into TIS-REFERENCE github repo
- It creates the JAR and Docker image by running service-pipeline
- When FINALTAG is set, it deploys generic-upload into dev
...
- This is the main job for profile
- It is triggered by a push into TIS-PROFILE github repo
- It creates the JAR and Docker image by running service-pipeline
- When FINALTAG is set, it deploys generic-upload into dev
...
- This is the main job for profile
- It is triggered by a push into TIS-PROFILE github repo
- It creates the JAR and Docker image by running service-pipeline
- When FINALTAG is set, it deploys generic-upload into dev
- DUPLICATE ???
...
- This is the main job for assessments
- It is triggered by a push into TIS-ASSESSMENTS github repo
- It creates the JAR and Docker image by running service-pipeline
- When FINALTAG is set, it deploys generic-upload into dev
...
- runs ansible task tasks/audit-logs-backup.yml in dev
- backs audt logs to azure storage container heetislogsbackup, container audit-logs-dev
...
- runs ansible task tasks/audit-logs-backup.yml in prod
- backs audt logs to azure storage container heetislogsbackup, container audit-logs-prod
...
- This is the main job for revalidation
- It is triggered by a push into TIS-REVALIDATION github repo
- It creates the JAR and Docker image by running service-pipeline
- When FINALTAG is set, it deploys generic-upload into dev
- triggers ci-test jenkins job
...
- Runs ansible/tasks/intrepid-reval-etl-all.yml in dev, stage, uidev
- creates docker instance intrepid-dr-etl:{{ versions['intrepid-reval-etl'] }}
...
...
- Runs ansible/tasks/intrepid-reval-etl-all.yml in UAT
- creates docker instance intrepid-dr-etl:{{ versions['intrepid-reval-etl'] }}
...
- Runs ansible/tasks/intrepid-reval-etl-all.yml in prod with --limit 10.170.0.137
- creates docker instance intrepid-dr-etl:{{ versions['intrepid-reval-etl'] }}
...
- 5h30
- daily
...
intrepid-reval-etltype-schema-env
...
- takes ETLTYPE and ENVIRONMENT parameters
- Runs ansible/tasks/intrepid-reval-etl-all.yml --extra-vars="etl_type=${ETLTYPE}"
- creates docker instance intrepid-dr-etl:{{ versions['intrepid-reval-etl'] }}
...
- runs mvn verify
- triggers release-tag
...
- is triggered by the likes of ci_test
- takes the following parameters GIT_TAG_URL, GIT_TAG_BRANCH, GIT_TAG_VALUE
- tags the repo GIT_TAG_URL with tag GIT_TAG_VALUE
...
- DECISION REQUIRED - last run 2017 Feb 27
- pipeline script
- takes a git repo, runs maven on it then runs mvn2docker
...
- This is the main job for reference
- It is triggered by a push into TIS-TRAINEE-UI github repo
- It creates the JAR and Docker image by running service-pipeline
- When FINALTAG is set, it deploys generic-upload into dev
...
- This is the main job for notifications
- It is triggered by a push into TIS-NOTIFICATIONS github repo
- It creates the JAR and Docker image by running service-pipeline
- When FINALTAG is set, it deploys generic-upload into dev
...
- This is the main job for common
- It is triggered by a push into TIS-COMMON github repo
- It creates the JAR and Docker image by running service-pipeline
- When FINALTAG is set, it deploys generic-upload into dev
...
Project | Jenkins job | Purpose | SCM triggered | cron | Action | ||
---|---|---|---|---|---|---|---|
ANY | service-env-deploy |
| no | no | KEEP | ||
ANY | service-pipeline |
| no | no | KEEP | ||
ESR | tis-esr |
| yes | no | KEEP | ||
tis-esr-parameterized |
| no | no | ||||
ESR-ETL | tis-esr-etl |
| yes | no | KEEP | ||
tis-esr-etl-cron |
| yes | yes | ||||
tis-esr-etl-parameterized |
| no | no | ||||
tis-esr-etl-full-notification |
| no | no | ||||
TIS-ESR-FTPSYNC | tis-esr-ftpsync-manual |
| no | no | |||
tis-esr-ftpsync-cron-prod |
| no | yes | ||||
tis-esr-ftpsync-cron-stage |
| no | yes | ||||
ESR Not used any more | |||||||
| no | "yes" | KEEP | ||||
tis-esr-etl-jenkinsfile | Test of new pipeline | no | no | ||||
| no | yes | KEEP | ||||
esr-etl |
| yes | no | ||||
tis-esr-etl-param-apc-import |
| no |
| DELETE Deleted on 2018/08 | |||
tis-esr-etl-param-applicant-load |
| no |
| DELETE | |||
tis-esr-etl-param-applicant-export |
| no |
| DELETE | |||
tis-esr-etl-param-notification-export |
| no |
| DELETE | |||
tis-esr-etl-param-notification-daily-load |
| no |
| DELETE | |||
esr-etl |
| yes | no |
| |||
NDW-ETL | tis-ndw-etl |
| yes | no |
| ||
tis-ndw-etl-cron-uat |
| yes | yes | ||||
tis-ndw-etl-cron-prod |
| yes | yes | ||||
tis-ndw-etl-parameterized |
| yes | no | ||||
NDW not used anymore | tis-ndw-etl-parameterized-prod |
| no |
| KEEP | ||
| no |
| KEEP | ||||
chris-tis-ndw |
| no | no |
| |||
service-pipeline-ndw-etl | disabled because this was a poor attempt to change service-pipeline DELETED | no | no | DELETE | |||
Generic Upload | tis-generic-upload-parameterized |
| no | no |
| ||
tis-generic-upload |
| yes | no | KEEP | |||
TCS | tcs |
| yes | no | KEEP | ||
tcs-paramameterized | disabled because this was a poor attempt to change service-pipeline DELETED | DELETE | |||||
admins-ui | admins-ui |
| yes | no | KEEP | ||
TIS-ADMINS-UI |
| no | no |
| |||
admins-ui-e2e |
| yes | no |
| |||
admins-ui-uat |
| no | no |
| |||
chris-admins-ui |
| no | no |
| |||
TIS-ADMINS-UI |
| no | no |
| |||
reference | reference |
| yes | no |
| ||
profile | profile |
| yes | no |
| ||
profile-parent |
| yes | no |
| |||
assessments | assessments |
| yes | no |
| ||
tis-assessment-programme-membership-etl |
| yes | no | ||||
tis-assessment-programme-membership-etl-runner |
| no | no | ||||
audit | audit-logs-backup-dev |
| no |
|
| ||
audit-logs-backup-prod |
| no |
|
| |||
revalidation | revalidation |
| yes | no |
| ||
intrepid | intrepid-reval-etl-dev/stage/uidev |
| no | no |
| ||
| no | no |
| ||||
intrepid-reval-etl-all-prod |
| no |
|
| |||
| no | no |
| ||||
ci_test | ci_test |
| no | no |
| ||
ci_test_tagging |
| yes | no | ||||
ci_tests_parallel |
| yes | no | ||||
release-tag | release-tag |
| no | no |
| ||
fayaz-pl | fayaz-pl |
| no | no |
| ||
Trainee-UI | trainee-ui |
| yes | no |
| ||
Notifications | notifications |
| yes | no |
| ||
common | common |
| yes | no |
| ||
Intrepid | intrepid-load |
| no | no |
| ||
intrepid-dr-model |
| yes | no |
| |||
intrepid-dr-tools |
| yes | no |
| |||
intrepid-load-etl |
| no | no |
| |||
intrepid-anonymizer |
| no | no |
| |||
| no | no |
| ||||
| yes | no |
| ||||
intrepid-core-model |
| yes | no |
| |||
intrepid-load-stage |
| no | no |
| |||
intrepid-etl-pipeine | Quote:
| no | no |
| |||
| no |
|
| ||||
intrepid-extract-all |
| no | no |
| |||
| no |
|
| ||||
| yes | no |
| ||||
| no | no |
| ||||
| no | no |
| ||||
| no |
|
| ||||
| no | no |
| ||||
intrepid-reval-etl-stage |
| no | no |
| |||
intrepid-reval-etl-uidev |
| no | no |
| |||
| yes | no |
| ||||
| yes | no |
| ||||
| no | no |
| ||||
| no | no |
| ||||
| no | no |
| ||||
| no | no |
| ||||
intrepid-etl-pipeline-stage-new |
| no | no |
| |||
| no | no |
| ||||
| no | no |
| ||||
| no | no |
| ||||
intrepid-remapper-structure-etl |
| no | no |
| |||
intrepid-reval-etltype-schema-env |
| no | no |
| |||
| no | no |
| ||||
Consolidated-DR-ETL |
| no | no |
| |||
consolidated-dr-etl-all-parameterised-dev-tisdev4305 |
| no | no |
| |||
consolidated-dr-etl-all-parameterised-stage |
| no | no |
| |||
consolidated-dr-etl-all-parameterised-prod |
| no |
|
| |||
| yes | no |
| ||||
| yes | no |
| ||||
| no | no |
| ||||
| no | no |
| ||||
| no | no |
| ||||
| no | no |
| ||||
| no | no |
| ||||
| no | no |
| ||||
| no | no |
| ||||
consolidated-dr-etl-old-uidev |
| no | no |
| |||
consolidated-dr-etl-schema-env |
| no | no |
| |||
consolidated-dr-etl-all-prod-etl |
| no | no |
| |||
Ansible |
| no |
|
| |||
Misc | bdd_tests_staging | folder /datadrive/data/jenkins/jobs/bdd_tests_staging/ exists but no job in jenkins | no | no |
| ||
browserstack-cucumber |
| no | no |
| |||
CI | ci-stage |
| no | no |
| ||
| no | no |
| ||||
| no | no |
| ||||
| no | no |
| ||||
| no | no |
| ||||
| no | no |
| ||||
| no | no |
| ||||
| no | no |
| ||||
| no | no |
| ||||
| no | no |
| ||||
| yes | no |
| ||||
| no | no |
| ||||
| yes | no |
| ||||
| no | no |
| ||||
| no | no |
| ||||
| yes | no |
| ||||
DOCKER |
| no |
|
| |||
| no |
|
| ||||
| no |
|
| ||||
| no |
|
| ||||
| no |
|
| ||||
| no |
|
| ||||
| no | no |
| ||||
| no | no |
| ||||
| no |
|
| ||||
| no | no |
| ||||
ELK |
| no |
|
| |||
| no |
|
| ||||
| no |
|
| ||||
| no | no |
| ||||
GMC |
| no | no |
| |||
| yes | no |
| ||||
| yes | no |
| ||||
| no | no |
| ||||
| no | no |
| ||||
| no |
|
| ||||
| no | no |
| ||||
| no |
|
| ||||
| no | no |
| ||||
| no |
|
| ||||
| no |
|
| ||||
| no |
|
| ||||
| no | no |
| ||||
| no | no |
| ||||
| yes | no |
| ||||
| no | no |
| ||||
| yes | no |
| ||||
JENKINS |
| no |
|
| |||
| no | no |
| ||||
| yes | no |
| ||||
KEYCLOAK | keycloak-client |
| yes | no |
| ||
| yes | no |
| ||||
| yes | no |
| ||||
| yes | no |
| ||||
london-dr-model |
| yes | no |
| |||
| no |
|
| ||||
| yes | no |
| ||||
| no |
|
| ||||
| no | no |
| ||||
MYSQL |
| no |
|
| |||
| no |
|
| ||||
| no |
|
| ||||
| no | no |
| ||||
| no | no |
| ||||
NODE |
| no | no |
| |||
ORIEL |
| yes | no |
| |||
|
|
|
|
Code Block |
---|
compares the schema csv exports daily to make sure that the structure hasn't changed. |
- watches intrepid-etl-pipeine as a trigger
- runs ansible/tasks/intrepid-schema-validation.yml on Localhost
- runs mvn on ETL hosts to create jarfile, then java -jar jarfile
Code Block |
---|
Runs the second part of the pipeline for intrepid-etl on the mirror stage
Intrepid data anonymizer - takes data from ssh hee-build-azure -> /datadrive/data/intrepid/prod/ (full+delta) anonymises it then places it into /datadrive/data/intrepid/stage/ (full+delta)
Intepid Load Mirror -> takes data from /datadrive/data/intrepid/stage/ , creates a delta file then loads it into mirror stage
Intrepid revalidation stage -> the reval etl on stage |
- watches intrepid-etl-pipeine as a trigger
- pipeline script from SCM
- SCM: TIS-DEVOPS
- script: jenkins/intrepid-etl-stage.groovy
- jenkins/intrepid-etl-stage.groovy runs jenkins jobs intrepid-anonymizer, intrepid-load-stage, intrepid-reval-etl-stage
- disabled, last successful run 2017 Nov 9th
Code Block |
---|
compresses and uploads intrepid databases in azure |
- runs ansible/tasks/intrepid-extract-upload.yml in PROD using DEVOPS_BASE=/datadrive/data/fayaz/master/devops
Code Block |
---|
Remaps the Intrepid DR structure |
- runs ansible/tasks/intrepid-remapper-structure.yml with inventory=localhost on ETL hosts
- runs mvn on ETL hosts to create jarfile, then java -jar jarfile
- runs ansible/tasks/intrepid-remapper-structure.yml with inventory localhost
- ansible/tasks/intrepid-remapper-structure.yml: runs mvn on ETL hosts to create jarfile, then java -jar jarfile
- can be run by intrepid-etl-pipeine
|
- It is triggered by a push into TIS-INTREPID-DR-MODEL github repo
- runs mvn clean install -e
- It is triggered by a push into TIS-INTREPID-DR-TOOLS github repo
- runs mvn clean install -e then mvn2docker
- disabled, last run 2017 Nov 9th
- runs ansible playbook ansible/tasks/intrepid-load.yml in PROD
- runs ansible code ansible/tasks/intrepid-anonymizer.yml against localhost
- deploys docker image repository.tis.nhs.uk:5000/hee/anonymizer:{{ versions['intrepid-anonymizer'] }}
- runs ansible/tasks/intrepid-anonymizer-new.yml in STAGE
- never run
Code Block |
---|
Anonymise intrepid data to populate the anonymized mirror |
- This is the main job for intrepid-anonymizer
- It is triggered by a push into TIS-DATA-ANONYMIZER github repo
- It creates the JAR and Docker image by running service-pipeline
- When FINALTAG is set, it deploys generic-upload into dev
- last run 2017 January 9th
- This is the main job for intrepid-core-model
- It is triggered by a push into TIS-INTREPID-CORE-MODEL github repo
- It creates the JAR and Docker image by running mvn2docker
- triggered by intrepid-etl-pipeine
- runs ansible playbook ansible/tasks/intrepid-load.yml in STAGE
Quote:
Code Block |
---|
Runs intrepid-extract and load each morning. Intrepid extract: grabs data from intrepid places in into CSV files (db-to-csv). Intrepid remap: changes column names on CSV files (intrepid-remap-col-structure) - we could remove it Intrepid load: grabs CSV data and places it into the mirror (csv-diff) creates a delta file and then uploads to db Intrepid conosolidated etl: points to the mirror and populates the TIS system
|
- pipeline script from SCM
- SCM: TIS-DEVOPS
- script: jenkins/intrepid-etl-prod.groovy
- jenkins/intrepid-etl-prod.groovy can run intrepid-extract-all, intrepid-remapper-structure, intrepid-load, intrepid-extract-upload, intrepid-compare-stats
- pipeline script from SCM
- SCM: TIS-DEVOPS
- script: jenkins/intrepid-etl-prod-new.groovy
- jenkins/intrepid-etl-prod-new.groovy runs ansible/tasks/intrepid-extract-new.yml with intrepid_dr_settings=<dr2|mirror-prod> schema=R1 mirror_settings=mirror-prod
batch_size=20
- 2h45
- daily
- runs ansible/tasks/intrepid-extract-all.yml in PROD
- ansible/tasks/intrepid-extract-all.yml: deploys docker image repository.tis.nhs.uk:5000/hee/intrepid-extract:{{ versions['intrepid-extract'] }}
- disabled, last successful run 2017 Nov 30th
- runs ansible/tasks/intrepid-extract-new.yml on ETL hosts in PROD
- midnight
- daily
- This is the main job for notifications
- It is triggered by a push into TIS-INTREPID-DR-ETL github repo
- It creates the JAR and Docker image by running service-pipeline
- When FINALTAG is set, it deploys generic-upload into dev
- triggered by intrepid-etl-pipeine
- runs ansible/tasks/intrepid-compare-stats.yml in PROD
- runs mvn on ETL hosts to create jarfile, then java -jar jarfile
- compares count of items in the prod dr against the mirror
- disabled, last run 2017 Nov 9th
- runs ansible/tasks/intrepid-compare-stats.yml in PROD, with DEVOPS_BASE=/datadrive/data/fayaz/master/devops
- runs ansible/tasks/intrepid-extract-clean.yml --limit 10.140.0.1 in PROD on ETL
- runs automatically
- removes old files from /datadrive/data/intrepid
- midnight
- daily
- runs ansible/tasks/intrepid-reval-etl-all.yml in DEV
- runs ansible/tasks/intrepid-reval-etl-all.yml in STAGE
- runs ansible/tasks/intrepid-reval-etl-all.yml in UIDEV
- It is triggered by a push into TIS-DB-TO-CSV github repo
- triggers the job intrepid-extract-docker
- It is triggered by a push into TIS-DB-TO-CSV github repo
- It can be triggered by db-to-csv-parent
- It runs mvn clean install -e on TIS-DB-TO-CSV/db-to-csv pom.xml file
- It runs ansible/tasks/spring-boot-build.yml in DEV on localhost
- it runs mvn2docker to create an image named intrepid-extract
- When FINALTAG is set, it deploys generic-upload into dev ????
- triggered by intrepid-etl-pipeine
- compresses and uploads intrepid databases to azure blob storage ???
- runs ansible/tasks/intrepid-extract-upload.yml in PROD on ETL hosts
- uses export DEVOPS_BASE=/datadrive/data/fayaz/master/devops in PROD
- last run 2017 November 9th
- deploys docker image repository.tis.nhs.uk:5000/hee/intrepid-extract:{{ versions['intrepid-extract'] }}
- extracts intrepid databases
no | no |
| |||||||||||
OWASP |
| no | no |
| |||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| no | no |
| ||||||||||
person-owner-post-view-rebuild |
| no |
|
| |||||||||
| no | no |
| ||||||||||
| no |
|
| ||||||||||
| no | no |
| ||||||||||
RELEASE |
| no | no |
| |||||||||
| no | no | KEEP | ||||||||||
| no | no |
| ||||||||||
| no | no |
| ||||||||||
| no | no |
| ||||||||||
| no | no |
| ||||||||||
| no | no |
| ||||||||||
| no | no |
| ||||||||||
SANDBOX |
| no |
|
| |||||||||
| no |
|
| ||||||||||
SERVICE |
| no | no |
| |||||||||
| no | no |
| ||||||||||
| no | no |
| ||||||||||
| no | no |
| ||||||||||
service-pipeline-bkp |
| no | no |
| |||||||||
service-pipeline-ndw-etl |
| no | no |
| |||||||||
| no | no |
| ||||||||||
| no |
|
| ||||||||||
| no | no |
| ||||||||||
SITE |
| no |
|
| |||||||||
| no | no |
| ||||||||||
| no | no |
| ||||||||||
| no | no |
| ||||||||||
| no | no |
| ||||||||||
| no | no |
| ||||||||||
ssh |
| no | no |
| |||||||||
|
| ||||||||||||
| no | no |
| ||||||||||
|
|
| |||||||||||
| no | no |
| ||||||||||
| yes | no |
| ||||||||||
| yes | no |
| ||||||||||
| yes | no |
| ||||||||||
| no |
|
| ||||||||||
| no | no |
| ||||||||||
| no | no |
|