Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Date

Authors

Status

Documenting

Summary

DMS tasks have been down for a couple of months (since mid October) and were brought back up only today

Impact

No data changes synchronized over to TSS in those two months

Non-technical Description

  • DMS was not running on either prod and preprod. Changes in the TIS databases meant to be captured and synchronized to TSS must not have been captured while DMS was down.

...

Trigger

  • Mysql server not being updated to anticipate DMS’s change of address

...

Detection

Resolution

TODO: .

Timeline

  • -

  • Andy Dingley noticed the tasks were not running on both preprod and prod

...

Resolution

  • Whitelisting of DMS’s new address on the Mysql server

...

Timeline

  • (approximate) - DMS tasks stop working

  • 14:10 GMT - Andy finds the the DMS tasks are not running

  • 15:30 GMT - Tasks restarted successfully after the whitelisting of DMS’s address

...

Root Cause(s)

  • DMS’s address change

...

Action Items

Action Items

Owner

Add monitoring to DMS

Mitigate to prevent this from happening in the future

...

Lessons Learned