Page Comparison

• This is a manual job only
• Can deploy any service in any environment
• Deploys the version in platform.yml

• Run by other jenkins jobs
• Creates JAR and Docker inages
• Put them into the registry

• This is the main job for esr
• It is triggered by a push into github
• It creates the JAR and Docker image by running service-pipeline
• When FINALTAG is set, it deploys esr-etl into dev

• This is the main job for esr
• It is triggered by a push into github
• It creates the JAR and Docker image by running service-pipeline
• When FINALTAG is set, it deploys esr-etl into dev

• This is for manual run only
• One runs it to pass a parameter like the date or the type of load needed
• It does not create any JAR file or docker image
• It just deploys whatever is in platform.yml into the selected environment
• One can pass any parameter to the job, unlike service-env-deploy

tis-esr-etl-param-cron

• This is a cron job in jenkins
• It runs ansible/esr-etl.yml with --extra-vars="LOAD_TYPE=${LOAD_TYPE}"
• Depending on the time, the value of LOAD_TYPE varies
• This deprecates the below tis-esr-param-XXXXX jenkins jobs

• This is a cron job in jenkins
• It runs tis-esr-parameterized with LOAD_TYPE=apc-import
• Disabled while tis-esr-etl-param-cron is being tested

• 20h00
• daily

• This is a cron job in jenkins
• It runs tis-esr-parameterized with LOAD_TYPE=applicant-load
• Disabled while tis-esr-etl-param-cron is being tested

• 15h00
• daily

• This is a cron job in jenkins
• It runs tis-esr-parameterized with LOAD_TYPE=applicant-export
• Disabled while tis-esr-etl-param-cron is being tested

• 17h00
• daily

• This is a cron job in jenkins
• It runs tis-esr-parameterized with LOAD_TYPE=notification-export
• Disabled while tis-esr-etl-param-cron is being tested

• 17h00
• daily

• This is a cron job in jenkins
• It runs tis-esr-parameterized with LOAD_TYPE=notification-export
• Disabled while tis-esr-etl-param-cron is being tested

• 02h00
• daily

• This is the main job for NDW-ETL
• It is triggered by a push into github
• It creates the JAR and Docker image by running service-pipeline
• When FINALTAG is set, it deploys esr-etl into dev

• This is a cron job in Jenkins
• It can take a parameter: the environment
• It will by default run in prod

• 07h00
• daily

• Uses jenkinsfile
• Can run against all our github repositories
• configured to use tis-ndw-etl github repo

• This is for manual run only
• One runs it to pass a DEBUG parameter which allows one to attach to port 5005TCP
• It does not create any JAR file or docker image
• It just deploys whatever is in platform.yml into the selected environment
• One can pass the debug parameter to the job, unlike service-env-deploy

• This is the main job for generic-upload
• It is triggered by a push into github
• It creates the JAR and Docker image by running service-pipeline
• When FINALTAG is set, it deploys generic-upload into dev

• This is the main job for tcs
• It is triggered by a push into github
• It creates the JAR and Docker image by running service-pipeline
• When FINALTAG is set, it deploys generic-upload into dev

• This is the main job for admins-ui
• It is triggered by a push into TIS-ADMINS-UI github repo
• It creates the JAR and Docker image by running service-pipeline
• When FINALTAG is set, it deploys generic-upload into dev

• Uses jenkinsfile
• Can run against all our github repositories
• configured to use ndw-etl github repo

• runs End to End tests
• active but not a single successfull run
• last issue: access to selenium server http://hub-cloud.browserstack.com/wd/hub
• DECISION REQUIRED

• This is a clone of the main job for admins-ui
• It used to be triggered by a push into TIS-ADMINS-UI github repo
• It creates the JAR and Docker image by running service-pipeline
• When FINALTAG is set, it deploys generic-upload into dev
• DECISION REQUIRED - last run was on 2018 March 7th

• this is a different implementation of a pipeline
• It uses groovy pipeline directly implemented in the GUI
• It never succeded, last run was on 2018 March 18th
• Last error: java.lang.NoSuchMethodError: No such DSL method 'post' found among steps
• DECISION REQUIRED

• Uses jenkinsfile
• Can run against all our github repositories
• configured to use TIS-ADMINS-UI github repo

• This is the main job for reference
• It is triggered by a push into TIS-REFERENCE github repo
• It creates the JAR and Docker image by running service-pipeline
• When FINALTAG is set, it deploys generic-upload into dev

• This is the main job for profile
• It is triggered by a push into TIS-PROFILE github repo
• It creates the JAR and Docker image by running service-pipeline
• When FINALTAG is set, it deploys generic-upload into dev

• This is the main job for profile
• It is triggered by a push into TIS-PROFILE github repo
• It creates the JAR and Docker image by running service-pipeline
• When FINALTAG is set, it deploys generic-upload into dev
• DUPLICATE ???

• This is the main job for assessments
• It is triggered by a push into TIS-ASSESSMENTS github repo
• It creates the JAR and Docker image by running service-pipeline
• When FINALTAG is set, it deploys generic-upload into dev

• runs ansible task tasks/audit-logs-backup.yml in dev
• backs audt logs to azure storage container heetislogsbackup, container audit-logs-dev

• runs ansible task tasks/audit-logs-backup.yml in prod
• backs audt logs to azure storage container heetislogsbackup, container audit-logs-prod

• This is the main job for revalidation
• It is triggered by a push into TIS-REVALIDATION github repo
• It creates the JAR and Docker image by running service-pipeline
• When FINALTAG is set, it deploys generic-upload into dev
• triggers ci-test jenkins job 

• Runs ansible/tasks/intrepid-reval-etl-all.yml in dev, stage, uidev
• creates docker instance intrepid-dr-etl:{{ versions['intrepid-reval-etl'] }}

intrepid-reval-etl-all-uat

• Runs ansible/tasks/intrepid-reval-etl-all.yml in UAT
• creates docker instance intrepid-dr-etl:{{ versions['intrepid-reval-etl'] }}

• Runs ansible/tasks/intrepid-reval-etl-all.yml in prod with --limit 10.170.0.137
• creates docker instance intrepid-dr-etl:{{ versions['intrepid-reval-etl'] }}

• 5h30
• daily

intrepid-reval-etltype-schema-env

• takes ETLTYPE and ENVIRONMENT parameters
• Runs ansible/tasks/intrepid-reval-etl-all.yml --extra-vars="etl_type=${ETLTYPE}"
• creates docker instance intrepid-dr-etl:{{ versions['intrepid-reval-etl'] }}

• runs mvn verify
• triggers release-tag

• is triggered by the likes of ci_test
• takes the following parameters GIT_TAG_URL, GIT_TAG_BRANCH, GIT_TAG_VALUE
• tags the repo GIT_TAG_URL with tag GIT_TAG_VALUE

• DECISION REQUIRED -  last run 2017 Feb 27
• pipeline script
• takes a git repo, runs maven on it then runs mvn2docker

• This is the main job for reference
• It is triggered by a push into TIS-TRAINEE-UI github repo
• It creates the JAR and Docker image by running service-pipeline
• When FINALTAG is set, it deploys generic-upload into dev

• This is the main job for notifications
• It is triggered by a push into TIS-NOTIFICATIONS github repo
• It creates the JAR and Docker image by running service-pipeline
• When FINALTAG is set, it deploys generic-upload into dev

• This is the main job for common
• It is triggered by a push into TIS-COMMON github repo
• It creates the JAR and Docker image by running service-pipeline
• When FINALTAG is set, it deploys generic-upload into dev

• runs in PROD
• runs ansible code ansible/tasks/intrepid-load.yml
• does not use any git repo, apat from tis-devops
• ansible/tasks/intrepid-load.yml: deploys docker image repository.tis.nhs.uk:5000/hee/csv-diff:{{ versions['intrepid-load'] }}

• It is triggered by a push into TIS-INTREPID-DR-MODEL github repo
• runs mvn clean install -e

• It is triggered by a push into TIS-INTREPID-DR-TOOLS github repo
• runs mvn clean install -e then mvn2docker

• disabled, last run 2017 Nov 9th
• runs ansible playbook ansible/tasks/intrepid-load.yml in PROD

• runs ansible code ansible/tasks/intrepid-anonymizer.yml against localhost
• deploys docker image repository.tis.nhs.uk:5000/hee/anonymizer:{{ versions['intrepid-anonymizer'] }}
  

intrepid-anonymizer-new

• runs ansible/tasks/intrepid-anonymizer-new.yml in STAGE
• never run

intrepid-anonymizer-build

Anonymise intrepid data to populate the anonymized mirror

• This is the main job for intrepid-anonymizer
• It is triggered by a push into TIS-DATA-ANONYMIZER github repo
• It creates the JAR and Docker image by running service-pipeline
• When FINALTAG is set, it deploys generic-upload into dev

• last run 2017 January 9th
• This is the main job for intrepid-core-model
• It is triggered by a push into TIS-INTREPID-CORE-MODEL github repo
• It creates the JAR and Docker image by running mvn2docker

• triggered by intrepid-etl-pipeine
• runs ansible playbook ansible/tasks/intrepid-load.yml in STAGE

intrepid-etl-pipeline-new

• pipeline script from SCM
• SCM: TIS-DEVOPS
• script: jenkins/intrepid-etl-prod-new.groovy
• jenkins/intrepid-etl-prod

• -new.groovy runs ansible/tasks/intrepid-extract-new.yml with intrepid_dr_settings=<dr2|mirror-prod> schema=R1 mirror_settings=mirror-prod
  batch_size=20
  

• 2h45
• daily

• runs ansible/tasks/intrepid-extract-all

• .yml in PROD
• ansible/tasks/intrepid-extract-all.yml: deploys docker image repository.tis.nhs.uk:5000/hee/intrepid-extract:{{ versions['intrepid-extract'] }}

intrepid-extract-

dr2

• disabled, last successful run 2017 Nov 30th
• runs ansible/tasks/intrepid-extract-new.yml on ETL hosts in PROD

• midnight
• daily

intrepid-etl-pipeline-new

dr-etl-build

• This is the main job for notifications
• It is triggered by a push into TIS-INTREPID-DR-ETL github repo
• It creates the JAR and Docker image by running service-pipeline
• When FINALTAG is set, it deploys generic-upload into dev

intrepid-compare-stats

• 2h45
• daily

intrepid-compare-stats-etl

• disabled, last run 2017 Nov 9th
• runs ansible/tasks/intrepid-compare-stats.yml in PROD, with DEVOPS_BASE=/datadrive/data/fayaz/master/devops

intrepid-extract-clean

• runs ansible/tasks/intrepid-extract-allclean.yml --limit 10.140.0.1 in PROD ansibleon ETL 
• runs automatically
• removes old files from /datadrive/data/intrepid
  

• midnight
• daily

intrepid-reval-etl-dev

• runs ansible/tasks/intrepid-extractreval-etl-all.yml : deploys docker image repository.tis.nhs.uk:5000/hee/intrepid-extract:{{ versions['intrepid-extract'] }}in DEV

• runs ansible/tasks/intrepid-reval-etl-all.yml in STAGE

• runs ansible/tasks/intrepid-extractreval-etl-newall.yml on ETL hosts in PRODUIDEV

intrepiddb-drto-etlcsv-build

parent

• It is triggered by a push into TIS-INTREPIDDB-DRTO-ETLCSV github repo
• It creates the JAR and Docker image by running service-pipeline
• When FINALTAG is set, it deploys generic-upload into dev

intrepid-compare-stats

intrepid-compare-stats-etl

• disabled, last run 2017 Nov 9thtriggers the job intrepid-extract-docker

intrepid-extract-docker

• It is triggered by a push into TIS-DB-TO-CSV github repo
• It can be triggered by db-to-csv-parent
• It runs mvn clean install -e on TIS-DB-TO-CSV/db-to-csv pom.xml file
• It runs ansible/tasks/spring-boot-build.yml in DEV on localhost
• it runs mvn2docker to create an image named intrepid-extract
• When FINALTAG is set, it deploys generic-upload into dev ????

intrepid-extract-upload

• triggered by intrepid-etl-pipeine
• compresses and uploads intrepid databases to azure blob storage ???
• runs ansible/tasks/intrepid-compareextract-statsupload.yml in PROD , with DEVOPSon ETL hosts

intrepid-extract-all-etl

• uses export DEVOPS_BASE=/datadrive/data/fayaz/master/devops

intrepid-extract-clean

• runs ansible/tasks/intrepid-extract-clean.yml --limit 10.140.0.1 in PROD on ETL 
• runs automatically
• removes old files from /datadrive/data/intrepid
  

• midnight
• daily

intrepid-reval-etl-dev

• runs ansible/tasks/intrepid-reval-etl-all.yml in DEV

• in PROD
• last run 2017 November 9th
• deploys docker image repository.tis.nhs.uk:5000/hee/intrepid-extract:{{ versions['intrepid-extract'] }}
  
• extracts intrepid databases

intrepid-schema-validation

intrepid-

etl-

pipeline-

• runs ansible/tasks/intrepid-reval-etl-all.yml in UIDEV

db-to-csv-parent

• It is triggered by a push into TIS-DB-TO-CSV github repo
• triggers the job intrepid-extract-docker

intrepid-extract-docker

• It is triggered by a push into TIS-DB-TO-CSV github repo
• It can be triggered by db-to-csv-parent
• It runs mvn clean install -e on TIS-DB-TO-CSV/db-to-csv pom.xml file
• It runs ansible/tasks/spring-boot-build.yml in DEV on localhost
• it runs mvn2docker to create an image named intrepid-extract
• When FINALTAG is set, it deploys generic-upload into dev ????

intrepid-extract-upload

• triggered by intrepid-etl-pipeine
• compresses and uploads intrepid databases to azure blob storage ???
• runs ansible/tasks/intrepid-extract-upload.yml in PROD on ETL hosts

• uses export DEVOPS_BASE=/datadrive/data/fayaz/master/devops in PROD
• last run 2017 November 9th
• deploys docker image repository.tis.nhs.uk:5000/hee/intrepid-extract:{{ versions['intrepid-extract'] }}
  
• extracts intrepid databases

intrepid-schema-validation

intrepid-etl-pipeline-stage-new

Runs the second part of the pipeline for intrepid-etl on the mirror stage
Intrepid data anonymizer - takes data from ssh hee-build-azure -> /datadrive/data/intrepid/prod/ (full+delta) anonymises it then places it into /datadrive/data/intrepid/stage/ (full+delta)
Intepid Load Mirror -> takes data from /datadrive/data/intrepid/stage/ , creates a delta file then loads it into mirror stage
Intrepid revalidation stage -> the reval etl on stage

Runs the second part of the pipeline for intrepid-etl on the mirror stage
Intrepid data anonymizer - takes data from ssh hee-build-azure -> /datadrive/data/intrepid/prod/ (full+delta) anonymises it then places it into /datadrive/data/intrepid/stage/ (full+delta)
Intepid Load Mirror -> takes data from /datadrive/data/intrepid/stage/ , creates a delta file then loads it into mirror stage
Intrepid revalidation stage -> the reval etl on stage

• watches intrepid-etl-pipeine as a trigger
• pipeline script from SCM
• SCM: TIS-DEVOPS
• script: jenkins/intrepid-etl-stage.groovy
• jenkins/intrepid-etl-stage.groovy runs jenkins jobs intrepid-anonymizer, intrepid-load-stage, intrepid-reval-etl-stage
  

intrepid-etl-pipeline-stage-new

Runs the second part of the pipeline for intrepid-etl on the mirror stage
Intrepid data anonymizer - takes data from ssh hee-build-azure -> /datadrive/data/intrepid/prod/ (full+delta) anonymises it then places it into /datadrive/data/intrepid/stage/ (full+delta)
Intepid Load Mirror -> takes data from /datadrive/data/intrepid/stage/ , creates a delta file then loads it into mirror stage
Intrepid revalidation stage -> the reval etl on stage

intrepid-extract-upload-etl

intrepid-remapper-structure

intrepid-remapper-structure

• runs ansible/tasks/intrepid-remapper-structure.yml with inventory localhost
• ansible/tasks/intrepid-remapper-structure.yml: runs mvn on ETL hosts to create jarfile, then java -jar jarfile
• can be run by  intrepid-etl-pipeine

intrepid-extract-consolidated

• runs ansible/tasks/intrepid-extract-consolidated.yml in PROD
• ansible/tasks/intrepid-extract-consolidated.yml does not exist
• last successfull run: 2017-07-23

intrepid-remapper-structure-etl

• disabled, last run 2017 Nov 9th
• runs ansible/tasks/intrepid-remapper-structure.yml with DEVOPS_BASE=/datadrive/data/fayaz/master/devops

intrepid-reval-etltype-schema-env

• runs ansible/tasks/intrepid-reval-etl-all.yml --extra-vars="etl_type=${ETLTYPE}"
• ETLTYPE= <arcp|connectionDiscrepancies|contactDetails|placements|revalidation|selfDeclaration|traineeCard>
• ansible/tasks/intrepid-reval-etl-all.yml deploys repository.tis.nhs.uk:5000/hee/intrepid-dr-etl:{{ versions['intrepid-reval-etl'] }}
  

test-intrepid-connection

• disabled, last run 2017 March 13th
• runs /opt/mssql-tools/bin/sqlcmd

consolidated-dr-etl-all-parameterised-dev

• pipeline script from SCM
• SCM: TIS-DEVOPS
• script: jenkins/

• consolidated-dr-etl

• .groovy
• jenkins/

• consolidated-dr-etl

• .groovy:

• runs ansible/tasks/

• consolidated-dr-

• etl-

• old.yml or ansible/tasks/consolidated-dr-

intrepid-extract-upload-etl

intrepid-remapper-structure