Date |
|
Authors | |
Status | Documenting |
Summary | TIS21-5196 - Getting issue details... STATUS |
Impact | Data in some of the TIS reports were stale and reflected what was on TIS from the last successful refresh |
Non-technical Description
An issue was discovered (on 11th Oct 2023) where TIS Data Extracts' refresh were failing since 2 days prior. The impact of this was the data Data in some of the TIS reports were stale and reflected what was on TIS from the last successful refresh.
Trigger
Detection
Email notifications in Site Owner’s inbox
Resolution
Re-application of missing permisssions on the NDW service account used to refresh data extracts on Tableau Server.
Re-run refresh post-fix
Timeline
BST unless otherwise stated
09:42 Site Owner raises the alarm on the number of email notifications in own inbox about failed tableau refreshes on TIS PID Data site
10:15 - PO and team informed. Brief team discussion on priority and delegation of the ‘coordinator’ role for the incident
11:00 - Quick analysis on number of data sources with failed refreshes and dates of last successful refresh established
11:56 - Tableau Administrator/Data Service team got notified
14:09 - Tableau Administrator/Data Service team confirms identification of the root cause of the problem
13:00 - Further analysis to establish scope/coverage of the problem and extent of stale data
07:21 - Requested progress update on permanent fix from Tableau Administrator/Data Service team
07:29 - Updated PO on state of affairs at the time
09:33 - Received confrmation of permanent fix in progres from Tableau Administrator/Data Service team
11:35 - Communication sent out on MS Teams channels to stakeholders
14:23 - Received notification of completion of permanent fix. Verified data sources now refreshed
14:38 - Updated stakeholders on MS Teams channels
Root Cause(s)
Modifications made to the permissions on the NDW service account used to refresh data extracts on Tableau Server
Action Items
Action Items | Owner | Status |
|---|---|---|
Suggest to Tableau Administrator/Data Service the possible update of distribution list to include more recipients of error/failure notifiations. Potential inclusions are: PO, Lead developer, one analyst from each team | Tableau Administrator/Data Service | |
Reduce frequency of refreshes for non-critical data where possible and adjdust refresh schedule accordingly for spread |
Lessons Learned
We were fortunate that the Site Owner was working.
Possible update of distribution list to include more recipients of error/failure notifiations. Suggestions for potential inclusions are: PO, Lead developer, one analyst from each team
Reduce frequency of refreshes for non-critical data where possible and adjdust refresh schedule accordingly for spread
Add Comment