Root Cause

Hicom's nightly refresh failed when trying to populate the 1.1.6 schema.

Trigger

Their nightly scheduled update.

Resolution

Hicom reran their ETLs and we reran ours shortly after.

Detection

Our nightly checks detected changes between the DR and mirror.

Action Items

Timeline

8:08 Russell Steel (Unlicensed) notified Yvonne on Slack to let her know that we were seeing problems

9:11 Yvonne informed us of the problem on their side

9:45 Martin at Hicom emailed Ben Whitton to tell him about the problem

10:04 Ben Whitton emailed the TIS team to let us know

Supporting Information

Slack Transcript

russ [8:08 AM]
@yvonne morning, we were expecting the NE DR to be upgraded on the 18th, but this schema is still showing in the old DR;

TABLE_CATALOG table_schema (No column name)
Intrepid.DataRepository HEEMD 70
Intrepid.DataRepository HELASE 70
Intrepid.DataRepository HENE 70

Please could you confirm when this environment is expected to be upgraded ?

yvonne [9:11 AM]
HENE and HEEMD were upgraded and moved to the HEE DR, but the old DR then failed to run, so by default it does not refresh and continues to display the old data. The latest data for HENE and HEEMD is in HEE DR. We are re-running the old DR now so we think it will have refreshed by about 11am.

Email from Martin to Ben

We experienced an issue over the weekend with the V10.8 Data Repository (Intrepid.DataRepository). This was caused by the migration of HENE and HEEM from Intrepid.DataRepository to Intrepid.DataRepository.HEE.

Intrepid.DataRepository.HEE refreshed successfully with the two new datasets, but Intrepid.DataRepository did not. This means that the data currently available on Intrepid.DataRepository does not contain any updates made to live data on Friday.

We are re-running the refresh, and expect to have up-to-date data available by 1200. At this point system accounts will be enabled to allow a re-run of any extracts needed.