...
An issue was discovered (on 11th Oct 2023) where TIS Data Extracts' refresh were failing since 2 days prior. The impact of this was the data Data in some of the TIS reports were stale and reflected what was on TIS from the last successful refresh.
Trigger
Refer to timeline point:
09:42
...
Detection
Email notifications in Site Owner’s inbox
...
BST unless otherwise stated
DateTime | Activty | Supporting documentation (if any) | |
|---|---|---|---|
| 1 | 09:42 | Site Owner raises the alarm on the number of email notifications in own inbox about failed tableau refreshes on TIS PID Data site | Sample of one of the email notifications  |
| 2 | 10:15 |
...
PO and team informed. Brief team discussion on priority and delegation of the ‘coordinator’ role for the incident | ||
| 3 | 11:00 |
...
Quick analysis on number of data sources with failed refreshes and dates of last successful refresh established. | ||
| 4 | 11:56 |
...
Tableau Administrator/Data Service team got notified |  | ||
| 5 | 13:00 | Further analysis to establish scope/coverage of the problem and extent of stale data |  |
| 6 | 14:09 |
...
Tableau Administrator/Data Service team confirms identification of the root cause of the problem |
...
 | |
| 7 | 07:21 |
...
Requested progress update on permanent fix from Tableau Administrator/Data Service team |  | |
| 8 | 07:29 |
...
Updated PO on state of affairs at the time via an MS teams call | ||
| 9 | 09:33 |
...
Received confrmation of permanent fix in progres from Tableau Administrator/Data Service team |  | |
| 10 | 11:35 |
...
Communication sent out on MS Teams channels to stakeholders. This was sent out on the following channels:
|  | |
| 11 | 14:23 |
...
Received notification of completion of permanent fix. Verified data sources now refreshed |  | |
| 12 | 14:38 |
...
Updated stakeholders on MS Teams channels |  |
Root Cause(s)
Modifications made to the permissions on the NDW service account used to refresh data extracts on Tableau Server.
Refer to timeline point:
14:09
...
Action Items
Action Items | Owner | Status |
|---|---|---|
Suggest to Tableau Administrator/Data Service the possible update of distribution list to include more recipients of error/failure notifiations. Potential inclusions are: PO, Lead developer, one analyst from each team | ||
Reduce frequency of refreshes for non-critical data where possible and adjdust refresh schedule accordingly for spread |
...