/
2020-08-03 TIS NDW ETLs didn't run

2020-08-03 TIS NDW ETLs didn't run

 

Date

Aug 3, 2020

Authors

@Andy Nash (Unlicensed)

Status

Complete

Summary

TIS NDW ETLs didn't run on Stage or Prod

Impact

Downstream NDW processes based on ‘stale’ data

 

  • Root Cause(s)

  • Trigger

  • Resolution

  • Detection

  • Action Items

Root Cause(s)

  • The scheduled task didn’t execute

  • The build server (with the schedule) didn’t get triggered in the expected timeframe

  • The build server was unavailable

  • Automated change requests caused a massive spike in load

Trigger

  • Unable to run the build on schedule

Resolution

  • Restarted Jenkins.

  • Manually triggered the tis-ndw-etl-cron-prod-to-prod, tis-ndw-etl-cron-stage-to-uat and tis-esr-etl-cron-notification-daily-load jobs.

  • Notified NDW team via. slack channel.

Detection

  • Noticed while checking weekly ETL run

Action Items

Action Item

Owner

Issue

Action Item

Owner

Issue

Trigger scheduled tasks differently.

 

 

Related content

2021-07-23 TIS-NDW ETL failure
2021-07-23 TIS-NDW ETL failure
More like this
2023-04-11 ndw-etl-prod task was not run
2023-04-11 ndw-etl-prod task was not run
More like this
2020-05-21 TIS NDW Failure
2020-05-21 TIS NDW Failure
More like this
2021-05-13 NDW and GMC ETLs didn't run (Jenkins Down)
2021-05-13 NDW and GMC ETLs didn't run (Jenkins Down)
More like this
2020-07-08 TIS NDW Failure
2020-07-08 TIS NDW Failure
More like this
2018-07-02 TIS-NDW ETL failed to run Sat/Sun/Mon morning
2018-07-02 TIS-NDW ETL failed to run Sat/Sun/Mon morning
More like this