2020-08-03 TIS NDW ETLs didn't run

 

Date

Aug 3, 2020

Authors

@Andy Nash (Unlicensed)

Status

Complete

Summary

TIS NDW ETLs didn't run on Stage or Prod

Impact

Downstream NDW processes based on ‘stale’ data

 

  • Root Cause(s)

  • Trigger

  • Resolution

  • Detection

  • Action Items

Root Cause(s)

  • The scheduled task didn’t execute

  • The build server (with the schedule) didn’t get triggered in the expected timeframe

  • The build server was unavailable

  • Automated change requests caused a massive spike in load

Trigger

  • Unable to run the build on schedule

Resolution

  • Restarted Jenkins.

  • Manually triggered the tis-ndw-etl-cron-prod-to-prod, tis-ndw-etl-cron-stage-to-uat and tis-esr-etl-cron-notification-daily-load jobs.

  • Notified NDW team via. slack channel.

Detection

  • Noticed while checking weekly ETL run

Action Items

Action Item

Owner

Issue

Action Item

Owner

Issue

Trigger scheduled tasks differently.