Date	21 Dec 2018
Authors	Jayanta Saha, John Simmons (Deactivated), Chris Mills (Unlicensed)
Status	Resolved
Summary	TIS Prod → NDW Prod ETL created duplicate entries in the NDW D/B. This occurred as a recent code change resulted in the ETL running from both Blue and Green (which aren't synced, so caused dupes)
Impact	NDW Prod D/B had every record duplicated.

Impact

When the TIS Prod → NDW Prod ETL started, data from both Blue and Green was transferred, resulting in a duplicate of each record in NDW. All reporting then required some NDW post-processing to account for removal of duplicates.

Root Causes

Recent TIS-DEVOPS deployment code changes which were kicking off the TIS-NDW-ETL running on both Blue and Green servers.

NDW databases were getting duplicates in different tables. NDW team, or one of their downstream users noticed the problem and alerted TIS on Slack #tis-ndw-etl channel.

20-12-2018

User queried the NDW Team via #tis-ndw-etl channel regarding duplicate issues on the NDW D/B. DevOps started looking at the docker logs and pulled them in a file (as all Devs were on the visit to King's Hospital)

21-12-2018

Removed from Apps, so behaviour of TIS Prod –> NDW Prod ETL should match that of TIS Stage → NDW UAT ETL and not duplicate entries.
Ticket up fixing the ETL properly to address tech debt that makes the ETL far from optimal.
Follow up with NDW team to confirm problem was introduced 10 days ago, and to understand knock on implications of this in order to determine whether any further adjustments are required (e.g. for people who have put reports together based on double the records).
Understand precisely why TIS Stage → NDW UAT is not exhibiting the same problem - is Blue/Green synced correctly on Stage, and not on Prod?

Issue could have been picked up during PR of https://github.com/Health-Education-England/TIS-DEVOPS/pull/391 but was missed - less haste, more speed with future PRs.
Addressing tech debt will make incident resolution simpler as the problem will be more obvious.

Relatively quickly identified the problem.
Relatively quickly identified a quick fix to prevent problem occurring again this evening.
NDW also identified a quick fix at their end and applied it to fix problem today.

Speedy PR approval missed the underlying problem.
No alerts to indicate the problem had happened (or that it actually started happening 10 days ago).

To check docker logs use the following command docker logs ndw-etl_ndw-etl_1 --tail 3000
#incident-ndw-20181221 channel in Slack