Non-technical Description

There was an interruption that stopped TIS data being sent to Hicom via the NDW.

Trigger

• Connection termination, probably due to NDW maintenance

Detection

• Alerting in our monitoring channel

• 

Resolution

• Reran the ETLs (resolved some downstream impact)

• Intrepid Leave Manager

Timeline

• 10 Mar 2021 - 04:41 & 05:02 - Alerts in the NDW monitoring channel

• 10 Mar 2021 - 08:30 - Notification to users on slack

• 10 Mar 2021 - 09:06 - NDW succesfuly re-run on stage and prod

• 10 Mar 2021 - 11:45 - NDW team confirm downstream

• 10 Mar 2021 - ~midday - Agreed with HICOM less disruption by waiting for tomorrow rather than re-running jobs

• 11 Mar 2021 - 10:50 - Confirmed that Leave Manager looks correct; standard checks are displaying normal results.

Root Cause(s)

• The Ansible job timed out.

• The ETL kept retrying a failed chunk. Retries all failed as well:

• All connections in the pool to NDW got closed and didn’t get re-created.

Action Items

Lessons Learned