2020-06-02 NDW ETL didn't complete in time

Date

Jun 2, 2020

Authors

@Joseph (Pepe) Kelly

Status

Stage and possibly Prod NDW populated late

Summary

A connection issue caused the transfer to the NDW to ‘hang’ on the stage/uat

Impact

Imformation may have been incomplete/1 day out of date

 

Root Cause(s)

  • Connection unavailable

  • Stage was unable to re-establish connection although the prod environment did.

  • NDW database unreachable part way through ETL.

Trigger

  • Notification via. Slack

Resolution

  • Re-ran job & notified NDW team

Timeline

  • 04:22 - Connections dropped

  • ? - log showing PROD transfer resumed and completed

  • 08:34 - Reported via. Slack

  • 10:05 - Re-runnig Prod transfer completes

  • 14:53 - NDW jobs complete