Content Comparison

...

• Similar issues happened today so:
  1. Dev team checked the docker logs.
  2. Then ran queries to check if the row count matches in the TCS and NDW D/B tables. Row count did not match as expected.
  3. DevOps checked the tis_ndw_etl code version in the Docker of of both Stage and Prod (they were the same).
  4. Then checked the TIS-DEVOPS code base and found is running on both Blue and Green (without locking or checking so just repeats everything twice).
  5. Checked whether this was also the case on Stage (because NDW UAT that receives data from TIS Stage was not suffering the duplicate issue). It wasn't.
  6. Identified that major improvements needed making to the ETL in future (apart from the fixes for this specific issue). Jira ticket to follow.
  7. Missed removing
  if
  1. it (what?) from Apps when resolving an array problem 10 days ago - indicating the problem has been around for 10 days but only picked up two days ago.
  2. Duplicates still exist, but are legacy duplicates inherited from Intrepid, so should be handled separately.

Action Items

• Removed from Apps, so behaviour of TIS Prod –> NDW Prod ETL should match that of TIS Stage → NDW UAT ETL and not duplicate entries.
• Ticket up fixing the ETL properly to address tech debt that makes the ETL far from optimal.
• Follow up with NDW team to confirm problem was introduced 10 days ago, and to understand knock on implications of this in order to determine whether any further adjustments are required (e.g. for people who have put reports together based on double the records).
• Understand precisely why TIS Stage → NDW UAT is not exhibiting the same problem - is Blue/Green synced correctly on Stage, and not on Prod?

...