Versions Compared

Old Version 2

changes.mady.by.user Philip Wilsdon

Saved on

compared with

New Version Current

changes.mady.by.user Joseph (Pepe) Kelly

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Date

Authors

Philip Wilsdon (Unlicensed)

Status

DocumentingDone

Summary

https://hee-tis.atlassian.net/browse/TIS21-1300

Impact

  • An increase in the number of helpdesk calls as people could not see trainees in Intrepid Leave Manager.   Some tableau reports would have been inaccurate in the morning.

Table of Contents

Non-technical Description

There was an interruption that stopped TIS data being sent to Hicom via the NDW.

...

Trigger

  • Connection termination, probably due to NDW maintenance

...

Detection

  • Alerting in our monitoring channel

...

Resolution

  • Reran the ETLs (resolved some downstream impact)

  • Intrepid Leave Manager

...

Timeline

  • - 04:41 & 05:02 - Alerts in the NDW monitoring channel

  • - 08:30 - Notification to users on slack

  • - 09:06 - NDW succesfuly re-run on stage and prod

  • - 11:45 - NDW team confirm downstream

  • - ~midday - Agreed with HICOM less disruption by waiting for tomorrow rather than re-running jobs

  • - 10:50 - Confirmed that Leave Manager looks correct; standard checks are displaying normal results.

...

Root Cause(s)

...

  • The Ansible job timed out.

  • The ETL kept retrying a failed chunk. Retries all failed as well:

  • All connections in the pool to NDW got closed and didn’t get re-created.

...

Action Items

Action Items

Owner

Ansible retries (Park this for now?)

John Simmons (Deactivated)

https://hee-tis.atlassian.net/browse/TIS21-1316

Lessons Learned

...

Marcello Fabbri (Unlicensed) Edward Barclay

https://hee-tis.atlassian.net/browse/TIS21-1318

Edward Barclay Marcello Fabbri (Unlicensed)

https://hee-tis.atlassian.net/browse/TIS21-1319

Andy Dingley

...

Lessons Learned