2017-01-23 Intrepid DR restored to day -1

Date

 

AuthorsReuben Noot (Deactivated)
StatusComplete
SummaryHicom detected errors with their internal DR update on the 18/01/17, on 23/01/17 they restored a backup from 21/01/17 and emailed users
ImpactNo impact as our ETL on Monday 23/01/17 ran ok and after checked we have confirmed it brought over the expected data

Root Cause

Email from Martin at 8.50 on Monday 23/01/17:

There were errors during the refresh process for the Intrepid Data Repository on 18/01/2017 at 15:33

The Data Repository was restored from the latest successful run on the night of 21/01/2017, and therefore data is now over 24 hours old.

We expect resumption of normal service, and a successful refresh to occur tonight. Further communication will follow if this changes.

Given that minimal changes will have occurred on a Sunday, we do not anticipate an issue for the majority of DR users. However extracts taken overnight will be incomplete, and therefore we would advise re-running these extracts.

Note that we had to kill active sessions for users logged in at the point of the restore.

Please accept our apologies for any inconvenience caused.

Thanks,
Martin

Martin Stanford
Delivery Manager
Hicom, Red House, Cemetery Pales, Brookwood, Surrey, GU24 0BL

Trigger

Nightly refresh of the Intrepid Data Repository (DR)

Resolution

Checked that our ETL that ran in the early hours of  was successful and brought all the required data across, it did, so no further action required.

Detection

Email from Martin at Hicom

Action Items

Action ItemTypeOwnerIssue




Timeline

8.50am - Email from Martin at Hicom

9.15am - Checked with Yvonne in the #Intrepid_Integration channel and she agreed to enable the service account's access briefly if required for a ETL re-run

9.20am - Alex checked the ETL and data extracted, no issues found, no re-run required.

Supporting Information

e.g. monitoring dashboards