2017-03-21 ETL prod failure
- Fayaz Abdul (Unlicensed)
- Reuben Noot (Deactivated)
Owned by Fayaz Abdul (Unlicensed)
| Date |
|
| Authors | |
| Status | Complete |
| Summary | Jenkins etl-prod jobs didn't executed . Jenkins process was down on the server due to intrepid-extraxt jenkins job |
| Impact | stale data was being used in the Revalidation app from to |
Root Cause
ETL of gmcsync and core didn't executed as Jenkins process was down.
Trigger
etl-prod jenkins jobs run daily
Resolution
Noticed the other intrepid-extract job is taking up the resources(memory) while running and therefore jenkins java process was being killed. Limited the memory usage at the java heap level for intrepid-extract job.
Detection
Jenkins was not reachable and etl jobs didn't executed
Action Items
| Action Item | Type | Owner | Issue |
|---|---|---|---|
| Improve Monitoring | Prevent | Fayaz Abdul (Unlicensed) | TISDEV-895 |
Timeline
- Jenkins was down and the etl jobs didn't executed.
- Fix to restrict memory usage went live and the etl jobs executed fine on
Supporting Information
None.
Slack: https://hee-nhs-tis.slack.com/
Jira issues: https://hee-tis.atlassian.net/issues/?filter=14213