Non-technical Description

• DMS was not running on either prod and preprod. Changes in the TIS databases meant to be captured and synchronized to TSS must not have been captured while DMS was down.

Trigger

• Mysql server not being updated to anticipate DMS’s change of address

Detection

• Andy Dingley noticed the tasks were not running on both preprod and prod

Resolution

• Whitelisting of DMS’s new address on the Mysql server

Timeline

• 15 Oct 2021 (approximate) - DMS tasks stop working

• 09 Dec 2021 14:10 GMT - Andy finds the the DMS tasks are not running

• 09 Dec 2021 15:30 GMT - Tasks restarted successfully after the whitelisting of DMS addresses

• 23 Dec 2021 14:20 GMT - Ticket opened with AWS Support to get information on why the DMS addresses were changed

• 23 Dec 2021 18:50 GMT - Response from AWS Support

Root Cause(s)

• DMS’s address change

  • Why did the DMS address change (& when might it happen again?)

• Does this mean it’s a firewall thing? MySQL user too? Is there a more dynamic way that we can set this?

  • Use AWS Secrets Manager instead?

• AWS Support mentioned that a host replacement occurred on both preprod/prod Replication Instances on 13 Dec 2021 - AWS Support was unable to get access to the records related to our DMS service issues back in October as the process logs are only kept for a limited time however there is a good chance that a host replacement also occurred in October. The public IPs would have been changed when the hosts were replaced.

Action Items

Lessons Learned