Jenkins jobs

• This is a manual job only
• Can deploy any service in any environment
• Deploys the version in platform.yml

• Run by other jenkins jobs
• Creates JAR and Docker inages
• Put them into the registry

• This is the main job for esr
• It is triggered by a push into github
• It creates the JAR and Docker image by running service-pipeline
• When FINALTAG is set, it deploys esr-etl into dev

• This is the main job for esr
• It is triggered by a push into github
• It creates the JAR and Docker image by running service-pipeline
• When FINALTAG is set, it deploys esr-etl into dev

• This is for manual run only
• One runs it to pass a parameter like the date or the type of load needed
• It does not create any JAR file or docker image
• It just deploys whatever is in platform.yml into the selected environment
• One can pass any parameter to the job, unlike service-env-deploy

tis-esr-etl-param-cron

• This is a cron job in jenkins
• It runs ansible/esr-etl.yml with --extra-vars="LOAD_TYPE=${LOAD_TYPE}"
• Depending on the time, the value of LOAD_TYPE varies
• This deprecates the below tis-esr-param-XXXXX jenkins jobs

• This is a cron job in jenkins
• It runs tis-esr-parameterized with LOAD_TYPE=apc-import
• Disabled while tis-esr-etl-param-cron is being tested

• 20h00
• daily

• This is a cron job in jenkins
• It runs tis-esr-parameterized with LOAD_TYPE=applicant-load
• Disabled while tis-esr-etl-param-cron is being tested

• 15h00
• daily

• This is a cron job in jenkins
• It runs tis-esr-parameterized with LOAD_TYPE=applicant-export
• Disabled while tis-esr-etl-param-cron is being tested

• 17h00
• daily

• This is a cron job in jenkins
• It runs tis-esr-parameterized with LOAD_TYPE=notification-export
• Disabled while tis-esr-etl-param-cron is being tested

• 17h00
• daily

• This is a cron job in jenkins
• It runs tis-esr-parameterized with LOAD_TYPE=notification-export
• Disabled while tis-esr-etl-param-cron is being tested

• 02h00
• daily

• This is the main job for NDW-ETL
• It is triggered by a push into github
• It creates the JAR and Docker image by running service-pipeline
• When FINALTAG is set, it deploys esr-etl into dev

• This is a cron job in Jenkins
• It can take a parameter: the environment
• It will by default run in prod

• 07h00
• daily

• Uses jenkinsfile
• Can run against all our github repositories
• configured to use tis-ndw-etl github repo

• This is for manual run only
• One runs it to pass a DEBUG parameter which allows one to attach to port 5005TCP
• It does not create any JAR file or docker image
• It just deploys whatever is in platform.yml into the selected environment
• One can pass the debug parameter to the job, unlike service-env-deploy

• This is the main job for generic-upload
• It is triggered by a push into github
• It creates the JAR and Docker image by running service-pipeline
• When FINALTAG is set, it deploys generic-upload into dev

• This is the main job for tcs
• It is triggered by a push into github
• It creates the JAR and Docker image by running service-pipeline
• When FINALTAG is set, it deploys generic-upload into dev

• This is the main job for admins-ui
• It is triggered by a push into TIS-ADMINS-UI github repo
• It creates the JAR and Docker image by running service-pipeline
• When FINALTAG is set, it deploys generic-upload into dev

• Uses jenkinsfile
• Can run against all our github repositories
• configured to use ndw-etl github repo

• runs End to End tests
• active but not a single successfull run
• last issue: access to selenium server http://hub-cloud.browserstack.com/wd/hub
• DECISION REQUIRED

• This is a clone of the main job for admins-ui
• It used to be triggered by a push into TIS-ADMINS-UI github repo
• It creates the JAR and Docker image by running service-pipeline
• When FINALTAG is set, it deploys generic-upload into dev
• DECISION REQUIRED - last run was on 2018 March 7th

• this is a different implementation of a pipeline
• It uses groovy pipeline directly implemented in the GUI
• It never succeded, last run was on 2018 March 18th
• Last error: java.lang.NoSuchMethodError: No such DSL method 'post' found among steps
• DECISION REQUIRED

• Uses jenkinsfile
• Can run against all our github repositories
• configured to use TIS-ADMINS-UI github repo

• This is the main job for reference
• It is triggered by a push into TIS-REFERENCE github repo
• It creates the JAR and Docker image by running service-pipeline
• When FINALTAG is set, it deploys generic-upload into dev

• This is the main job for profile
• It is triggered by a push into TIS-PROFILE github repo
• It creates the JAR and Docker image by running service-pipeline
• When FINALTAG is set, it deploys generic-upload into dev

• This is the main job for profile
• It is triggered by a push into TIS-PROFILE github repo
• It creates the JAR and Docker image by running service-pipeline
• When FINALTAG is set, it deploys generic-upload into dev
• DUPLICATE ???

• This is the main job for assessments
• It is triggered by a push into TIS-ASSESSMENTS github repo
• It creates the JAR and Docker image by running service-pipeline
• When FINALTAG is set, it deploys generic-upload into dev

• runs ansible task tasks/audit-logs-backup.yml in dev
• backs audt logs to azure storage container heetislogsbackup, container audit-logs-dev

• runs ansible task tasks/audit-logs-backup.yml in prod
• backs audt logs to azure storage container heetislogsbackup, container audit-logs-prod

• This is the main job for revalidation
• It is triggered by a push into TIS-REVALIDATION github repo
• It creates the JAR and Docker image by running service-pipeline
• When FINALTAG is set, it deploys generic-upload into dev
• triggers ci-test jenkins job 

• Runs ansible/tasks/intrepid-reval-etl-all.yml in dev, stage, uidev
• creates docker instance intrepid-dr-etl:{{ versions['intrepid-reval-etl'] }}

intrepid-reval-etl-all-uat

• Runs ansible/tasks/intrepid-reval-etl-all.yml in UAT
• creates docker instance intrepid-dr-etl:{{ versions['intrepid-reval-etl'] }}

• Runs ansible/tasks/intrepid-reval-etl-all.yml in prod with --limit 10.170.0.137
• creates docker instance intrepid-dr-etl:{{ versions['intrepid-reval-etl'] }}

• 5h30
• daily

intrepid-reval-etltype-schema-env

• takes ETLTYPE and ENVIRONMENT parameters
• Runs ansible/tasks/intrepid-reval-etl-all.yml --extra-vars="etl_type=${ETLTYPE}"
• creates docker instance intrepid-dr-etl:{{ versions['intrepid-reval-etl'] }}

• runs mvn verify
• triggers release-tag

• is triggered by the likes of ci_test
• takes the following parameters GIT_TAG_URL, GIT_TAG_BRANCH, GIT_TAG_VALUE
• tags the repo GIT_TAG_URL with tag GIT_TAG_VALUE

• DECISION REQUIRED -  last run 2017 Feb 27
• pipeline script
• takes a git repo, runs maven on it then runs mvn2docker

• This is the main job for reference
• It is triggered by a push into TIS-TRAINEE-UI github repo
• It creates the JAR and Docker image by running service-pipeline
• When FINALTAG is set, it deploys generic-upload into dev

• This is the main job for notifications
• It is triggered by a push into TIS-NOTIFICATIONS github repo
• It creates the JAR and Docker image by running service-pipeline
• When FINALTAG is set, it deploys generic-upload into dev

• This is the main job for common
• It is triggered by a push into TIS-COMMON github repo
• It creates the JAR and Docker image by running service-pipeline
• When FINALTAG is set, it deploys generic-upload into dev

• runs in PROD
• runs ansible code ansible/tasks/intrepid-load.yml
• does not use any git repo, apat from tis-devops
• ansible/tasks/intrepid-load.yml: deploys docker image repository.tis.nhs.uk:5000/hee/csv-diff:{{ versions['intrepid-load'] }}

• It is triggered by a push into TIS-INTREPID-DR-MODEL github repo
• runs mvn clean install -e

• It is triggered by a push into TIS-INTREPID-DR-TOOLS github repo
• runs mvn clean install -e then mvn2docker

• disabled, last run 2017 Nov 9th
• runs ansible playbook ansible/tasks/intrepid-load.yml in PROD

• runs ansible code ansible/tasks/intrepid-anonymizer.yml against localhost
• deploys docker image repository.tis.nhs.uk:5000/hee/anonymizer:{{ versions['intrepid-anonymizer'] }}
  

intrepid-anonymizer-new

• runs ansible/tasks/intrepid-anonymizer-new.yml in STAGE
• never run

intrepid-anonymizer-build

Anonymise intrepid data to populate the anonymized mirror

• This is the main job for intrepid-anonymizer
• It is triggered by a push into TIS-DATA-ANONYMIZER github repo
• It creates the JAR and Docker image by running service-pipeline
• When FINALTAG is set, it deploys generic-upload into dev

• last run 2017 January 9th
• This is the main job for intrepid-core-model
• It is triggered by a push into TIS-INTREPID-CORE-MODEL github repo
• It creates the JAR and Docker image by running mvn2docker

• triggered by intrepid-etl-pipeine
• runs ansible playbook ansible/tasks/intrepid-load.yml in STAGE

intrepid-etl-pipeline-new

• pipeline script from SCM
• SCM: TIS-DEVOPS
• script: jenkins/intrepid-etl-prod-new.groovy
• jenkins/intrepid-etl-prod-new.groovy runs ansible/tasks/intrepid-extract-new.yml with intrepid_dr_settings=<dr2|mirror-prod> schema=R1 mirror_settings=mirror-prod
  batch_size=20
  

• 2h45
• daily

• runs ansible/tasks/intrepid-extract-all.yml in PROD
• ansible/tasks/intrepid-extract-all.yml: deploys docker image repository.tis.nhs.uk:5000/hee/intrepid-extract:{{ versions['intrepid-extract'] }}

intrepid-extract-dr2

• disabled, last successful run 2017 Nov 30th
• runs ansible/tasks/intrepid-extract-new.yml on ETL hosts in PROD

• midnight
• daily

intrepid-dr-etl-build

• This is the main job for notifications
• It is triggered by a push into TIS-INTREPID-DR-ETL github repo
• It creates the JAR and Docker image by running service-pipeline
• When FINALTAG is set, it deploys generic-upload into dev

intrepid-compare-stats

intrepid-compare-stats-etl

• disabled, last run 2017 Nov 9th
• runs ansible/tasks/intrepid-compare-stats.yml in PROD, with DEVOPS_BASE=/datadrive/data/fayaz/master/devops

intrepid-extract-clean

• runs ansible/tasks/intrepid-extract-clean.yml --limit 10.140.0.1 in PROD on ETL 
• runs automatically
• removes old files from /datadrive/data/intrepid
  

• midnight
• daily

intrepid-reval-etl-dev

• runs ansible/tasks/intrepid-reval-etl-all.yml in DEV

• runs ansible/tasks/intrepid-reval-etl-all.yml in STAGE

• runs ansible/tasks/intrepid-reval-etl-all.yml in UIDEV

db-to-csv-parent

• It is triggered by a push into TIS-DB-TO-CSV github repo
• triggers the job intrepid-extract-docker

intrepid-extract-docker

• It is triggered by a push into TIS-DB-TO-CSV github repo
• It can be triggered by db-to-csv-parent
• It runs mvn clean install -e on TIS-DB-TO-CSV/db-to-csv pom.xml file
• It runs ansible/tasks/spring-boot-build.yml in DEV on localhost
• it runs mvn2docker to create an image named intrepid-extract
• When FINALTAG is set, it deploys generic-upload into dev ????

intrepid-extract-upload

• triggered by intrepid-etl-pipeine
• compresses and uploads intrepid databases to azure blob storage ???
• runs ansible/tasks/intrepid-extract-upload.yml in PROD on ETL hosts

intrepid-extract-all-etl

• uses export DEVOPS_BASE=/datadrive/data/fayaz/master/devops in PROD
• last run 2017 November 9th
• deploys docker image repository.tis.nhs.uk:5000/hee/intrepid-extract:{{ versions['intrepid-extract'] }}
  
• extracts intrepid databases

intrepid-schema-validation

intrepid-etl-pipeline-stage

intrepid-etl-pipeline-stage-new

intrepid-extract-upload-etl

intrepid-remapper-structure

intrepid-remapper-structure

• runs ansible/tasks/intrepid-remapper-structure.yml with inventory localhost
• ansible/tasks/intrepid-remapper-structure.yml: runs mvn on ETL hosts to create jarfile, then java -jar jarfile
• can be run by  intrepid-etl-pipeine

intrepid-extract-consolidated

• runs ansible/tasks/intrepid-extract-consolidated.yml in PROD
• ansible/tasks/intrepid-extract-consolidated.yml does not exist
• last successfull run: 2017-07-23

intrepid-remapper-structure-etl

• disabled, last run 2017 Nov 9th
• runs ansible/tasks/intrepid-remapper-structure.yml with DEVOPS_BASE=/datadrive/data/fayaz/master/devops

intrepid-reval-etltype-schema-env

• runs ansible/tasks/intrepid-reval-etl-all.yml --extra-vars="etl_type=${ETLTYPE}"
• ETLTYPE= <arcp|connectionDiscrepancies|contactDetails|placements|revalidation|selfDeclaration|traineeCard>
• ansible/tasks/intrepid-reval-etl-all.yml deploys repository.tis.nhs.uk:5000/hee/intrepid-dr-etl:{{ versions['intrepid-reval-etl'] }}
  

test-intrepid-connection

• disabled, last run 2017 March 13th
• runs /opt/mssql-tools/bin/sqlcmd

consolidated-dr-etl-all-parameterised-dev

• pipeline script from SCM
• SCM: TIS-DEVOPS
• script: jenkins/consolidated-dr-etl.groovy
• jenkins/consolidated-dr-etl.groovy: runs ansible/tasks/consolidated-dr-etl-old.yml or ansible/tasks/consolidated-dr-etl-all.yml
  
  

• disabled

• This is to test a new parameter for consolidated-dr-etl-all-parameterised-dev 
  The new parameter is POST_PROGRAMME_STEP_ENABLED 
  We use a new branch named TISDEV-4305-POST_PROGRAMME_STEP_ENABLED of TIS-DEVOPS and a modified version of jenkins/consolidated-dr-etl.groovy
  

• collects lots of parameters
• runs jenkins/consolidated-dr-etl.groovy
• jenkins/consolidated-dr-etl.groovy: runs ansible/tasks/consolidated-dr-etl-old.yml or ansible/tasks/consolidated-dr-etl-all.yml
• by default, ENVIRONMENT=prod

consolidated-dr-etl-all-parameterised-prod

• collects lots of parameters
• runs jenkins/consolidated-dr-etl.groovy
• jenkins/consolidated-dr-etl.groovy: runs ansible/tasks/consolidated-dr-etl-old.yml or ansible/tasks/consolidated-dr-etl-all.yml
• by default, ENVIRONMENT=prod

• 05h30
• daily

consolidated-dr-etl

• This is the main job for intrepid-core-model
• It is triggered by a push into TIS-CONSOLIDATED-DR-ETL github repo
• It creates the JAR and Docker image by running mvn2docker

consolidated-dr-etl-old

• This is the main job for intrepid-core-model
• It is triggered by a push into TIS-CONSOLIDATED-DR-ETL-OLD github repo
• It creates the JAR and Docker image by running mvn2docker

consolidated-dr-etl-all-dev

• runs ansible/tasks/consolidated-dr-etl-all.yml in DEV

consolidated-dr-etl-old-dev

• runs ansible/tasks/consolidated-dr-etl-old.yml  --limit 10.150.0.137 in DEV 

consolidated-dr-etl-all-prod

• runs ansible/tasks/consolidated-dr-etl-all.yml  in PROD
• deploys repository.tis.nhs.uk:5000/hee/consolidated-dr-etl:{{ versions['consolidated-dr-etl'] }}
  

consolidated-dr-etl-old-prod

consolidated-dr-etl-all-stage

• runs ansible/tasks/consolidated-dr-etl-all.yml  in STAGE
• ansible/tasks/consolidated-dr-etl-all.yml: deploys repository.tis.nhs.uk:5000/hee/consolidated-dr-etl:{{ versions['consolidated-dr-etl'] }}
  

consolidated-dr-etl-all-uidev

• runs ansible/tasks/consolidated-dr-etl-all.yml in UIDEV
• ansible/tasks/consolidated-dr-etl-all.yml: deploys repository.tis.nhs.uk:5000/hee/consolidated-dr-etl:{{ versions['consolidated-dr-etl'] }}

consolidated-dr-etl-old-stage

• ANY environment
• runs ansible/tasks/consolidated-dr-etl-all.yml --extra-vars="schema=${SCHEMA}"
• deploys repository.tis.nhs.uk:5000/hee/consolidated-dr-etl:{{ versions['consolidated-dr-etl'] }}
  

• DISABLED
• uses /datadrive/data/fayaz/master/devops to run ansible/tasks/consolidated-dr-etl-all.yml in PROD

ansible-cmd

• runs /usr/local/bin/ansible -i /home/jenkins/data/devops/ansible/inventory/all -m setup --tree /datadrive/data/ansible-cmdb/ all

• 7h15
• daily

bdd_tests_staging

• last run on Nov 29, 2016 5:57:22 PM
• uses TIS-REVALIDATION repo
• runs "npm install && gulp bs" in DEV ???

• last run Jun 19, 2017 4:02 PM
• uses TIS-REVALIDATION repo
• runs the  ci-stage-pipeline pipeline

ci-stage-pipeline

• last run Jun 19, 2017 4:02:41 PM
• runs a pipeline script which runs a "mvn verify" on REVALIDATION, CONCERNS and NOTIFICATIONS

ci-staging

• last run Feb 20, 2017 4:46:10 PM
• watches site-stage
• triggers intrepid-reval-etl-stage

ci_test

• uses TIS-REVALIDATION repo
• run "mvn verify"
• triggers release-tag jenkins job

ci-tests-concerns

• last run Feb 21, 2017 2:42:59 PM
• uses TIS-CONCERNS git repo
• runs "mvn verify"

ci_tests_parallel

• DISABLED
• last run Sep 27, 2016 8:48:22 AM
• uses TIS-REVALIDATIONS git repo
• watches jenkins job revalidation-dev-deploy which does not exist

ci_tests_tagging

• DISABLED
• Never run
• uses TIS-REVALIDATIONS git repo
• runs selenium
• triggers non existing git-tag jenkins job

csv-diff

• uses db-to-csv git repo
• runs ansible/tasks/spring-boot-build.yml ansible playbook with ARTIFACT_ID="intrepid-load"
• ansible/tasks/spring-boot-build.yml runs mvn2docker
  
• last run Nov 15, 2017 10:15:45 AM

csv-to-db

• uses csv-to-db git repo
• runs ansible/tasks/spring-boot-build.yml ansible playbook with ARTIFACT_ID="intrepid-load"
• ansible/tasks/spring-boot-build.yml runs mvn2docker
  
• last run Nov 15, 2017 10:15:45 AM

curator-docker

• last run Apr 18, 2017 10:06:07 AM
• Rebuilds the Curator docker Image
• uses TIS-CURATOR git repo
• runs /home/jenkins/data/devops/bin/docker-build-push.sh which runs docker build|push|tag|rmi
  

db-to-csv

• last run Nov 15, 2017 10:15:45 AM
• uses db-to-csv git repo
• runs ansible/tasks/spring-boot-build.yml ansible playbook with ARTIFACT_ID="intrepid-load"
• ansible/tasks/spring-boot-build.yml runs mvn2docker

db-to-csv-archive

• DISABLED
• uses db-to-csv git repo
• last run Mar 16, 2017 12:54:04 PM

db-to-csv-parent

• last run Nov 15, 2017 10:15:45 AM
• uses TIS-DB-TO-CSV git repo
• triggers intrepid-extract-docker jenkins job

db-to-csv-test-archive

• DISABLED
• last run Mar 20, 2017 1:51:27 PM
• uses db-to-csv git repo
• runs /tmp/fayaz/TIS-DEVOPS/ansible/tasks/spring-boot-build.yml

dev-api-load-test

• never run
• uses TIS-JMETER git repo
• watches ci-staging
• runs "./jmeter.sh -n -t load.jmx -l testresults.jtl"

devops

• uses TIS-DEVOPS git repo
• runs python "${WORKSPACE}/ansible/inventory/tests.py"
• and then runs rsync -av --delete "${WORKSPACE}/" --exclude ".git" /home/jenkins/data/devops/
• and rsync -ave 'ssh' --delete "${WORKSPACE}/" --exclude ".git" $(/home/jenkins/data/devops/bin/ip etl-stage):/home/heetis/data/devops/

docker-alpine-java

• DISABLED
• last run Dec 16, 2017 12:41:00 AM
• uses docker-alpine-java git repo
• runs /home/jenkins/data/devops/bin/docker-build-push.sh
• /home/jenkins/data/devops/bin/docker-build-push.sh runs docker build|push|tag|rmi

• midnight
• daily

docker-clean-build

• 23h00
• friday

docker-clean-dev

Runs regularly to tidy up stray docker artifacts such as "<none>" images, old stopped containers and anonymous volumes

• uses DEV ansible inventory
• runs ansible/tasks/docker-clean.yml playbook
• ansible/tasks/docker-clean.yml: runs docker system prune -af

• 23h00
• friday

docker-clean-prod

• Runs regularly to tidy up stray docker artifacts such as "<none>" images, old stopped containers and anonymous volumes

• uses PROD ansible inventory
• runs ansible/tasks/docker-clean.yml playbook
• ansible/tasks/docker-clean.yml: runs docker system prune -af

• 23h00
• daily

docker-clean-stage

• Runs regularly to tidy up stray docker artifacts such as "<none>" images, old stopped containers and anonymous volumes

  • uses STAGE ansible inventory
  • runs ansible/tasks/docker-clean.yml playbook
  • ansible/tasks/docker-clean.yml: runs docker system prune -af

• 23h00
• daily

docker-clean-uidev

• 23h00
• daily

docker-freeze-dev

docker-freeze-prod

• Freezes docker images on a given platform

• uses the DEV ansible inventory
• runs ansible/tasks/docker-freeze.yml -vvv
• ansible/tasks/docker-freeze.yml runs bin/docker-freeze.py --tag {{ platform }}"
  
• bin/docker-freeze.py  runs docker push|tag

docker-registryui-cert-renew

• 6h06
• sunday