Non-technical Description

The overnight sync procedure for NI TIS was interrupted and failed to complete.

Data between tables within the NI TIS environment did not sync properly meaning some records will not display correct current information.

The team created a secondary service for the procedure to use which has resolved the issue. And a support ticket has been raised with the hosting company that will help the team decide whether to fix and switch back to the old service, or remain with the new one.

Trigger

Detection

• Detected when slack notifications in the #monitoring-prod channel failed to appear

Resolution

• The sync jobs run fine when manually triggered however this is a temporary solution

Timeline

• 08 Feb 2021 01:23 - Slack Alert in the #monitoring-prod channel

• 08 Feb 2021 08:08 - We let NI users know of the problem and this was acknowledged at 08:23

• 08 Feb 2021 08:58 - Let them know all backup and running

Root Cause(s)

Action Items

Lessons Learned