Date	2017-08-10
Authors	Graham O'Regan (Unlicensed)
Status	Complete
Summary	Hicom's DR refresh failed so the TIS ETLs only had access to the previous day's data.
Impact	Revalidation nationally had stale data.

Root Cause

Hicom have a problem with a lock on a mutex.

Trigger

Their nightly scheduled update.

Resolution

They reran their update

Detection

They detected the failure and notified us.

Action Items

Action Item	Type	Owner	Issue
	mitigate/prevent

Timeline

9:42 Email from Martin at Hicom

"Unfortunately the Data Repository suffered a repeat of Monday night’s issue last night.

While a successful refresh was still achieved, this did not complete until after 0530, and therefore any extracts run at this time will have picked up data from yesterday rather than today.

I have therefore enabled the system accounts to allow extracts to be rerun.

My apologies for the further inconvenience caused.

We have now identified the root cause of this issue, and we are working to resolve this for tonight’s refresh.

Supporting Information

e.g. monitoring dashboards

TIS21 Confluence Space

2017-08-10 Hicom DR refresh failed