Non-technical Description

The overnight sync procedure for NI TIS was interrupted and failed to complete.

The “quick access” copy of data did not sync properly meaning the Person search page was not operating properly.

The team created a secondary copy to use which has resolved the issue. And a support ticket has been raised with the hosting company that will help the team decide whether to fix and switch back to the old service, or remain with the new one.

Trigger

Detection

• Detected when slack notifications in the #monitoring-prod channel failed to appear

Resolution

• The sync jobs run fine when manually triggered however this is a temporary solution

Timeline

• 06 Feb 2021 01:23 - Slack Alert in the #monitoring-prod channel.

• 08 Feb 2021 OOH support raised with AWS.

• 08 Feb 2021 08:08 - We let NI users know of the problem and this was acknowledged at 08:23.

• 08 Feb 2021 Created a replacement resource.

• 08 Feb 2021 08:58 - Let them know all backup and running.

Root Cause(s)

• Under Investigation AWS Case ID 7967368241

Action Items

Lessons Learned