Versions Compared

Old Version 1

changes.mady.by.user Andy Nash

Saved on

compared with

New Version Current

changes.mady.by.user Joseph (Pepe) Kelly

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Date

Authors

Andy Nash (Unlicensed)

Status

Complete

Summary

TIS NDW ETLs didn't run on Stage or Prod

Impact

Downstream NDW processes could not startbased on ‘stale’ data

  • Root Cause(s)

  • Trigger

  • Resolution

  • Detection

  • Action Items

...

  • The scheduled task didn’t execute

  • The build server (with the schedule) didn’t get triggered in the expected timeframe

  • Each new ‘ETL’ killed the previous one;

  • The build server was unavailable

  • Automated change requests caused a massive spike in load

...

  • Restarted Jenkins.

  • Manually triggered the tis-esrndw-etl-cron-prod-to-prod, tis-ndw-etl-cron-stage-applicantto-export job uat and then the tis-esr-etl-cron-notification-daily-export job.Moved files to Monday’s 2020-08-03-outbound containerload jobs.

  • Notified NDW team via. slack channel.

Detection

  • Noticed while checking weekly ETL run

...