...
| Date | |
| Authors | Jayanta Saha, John Simmons (Deactivated), Chris Mills (Unlicensed) |
| Status | Resolved |
| Summary | TIS Prod → NDW Prod ETL created duplicate entries in the NDW D/B. This occurred as a recent code change resulted in the ETL running from both Blue and Green (which aren't synced, so caused dupes) |
| Impact | NDW Prod D/B had every record duplicated. |
| Table of Contents |
|---|
Jira reference
n/a.
Impact
Impact
- When the TIS Prod → NDW Prod ETL started, data from both Blue and Green was transferred, resulting in a duplicate of each record in NDW. All reporting then required some post-processing to account for removal of duplicates.
Root Causes
- Recent TIS-DEVOPS deployment code changes which were kicking off the TIS-NDW-ETL running on both Blue and Green.
Trigger
- NDW databases were getting duplicates in different tables. NDW team, or one of their downstream users noticed the problem and alerted TIS on Slack #tis-ndw-etl channel.
Resolution
- Modification of the TIS-DEVOPS deployment code so that it runs only on Green.
Detection / Timeline
- .
20-12-2018 - NDW Team reported the duplicate issues and DEVOPS started looking at the docker logs and pull them in a file
- 21-12-2018 - Similar issues happened today and Dev team checked the docker logs, run queries to check if the row count matches in the TCS and NDW db tables. Row count did not match as expected. DEVOPs team checked the version of etls on both Stage and Prod and found they are same. Finally they checked the TIS-DEVOPS code base found the issue.
Action Items
- .
Lessons Learned
...
Where we got lucky
- .
Supporting information
. To check docker logs use the following command docker logs ndw-etl_ndw-etl_1 --tail 3000