Non-technical Description

The reference service for NIMDTA was stuck in a cycle attempting to start-up due to an issue with our database migration tool. An upgrade to the migration tool changed the behaviour for missing migration steps, causing a failed start-up instead of ignoring the missing steps as it previously did.

Trigger

• Upgraded Flyway version deployed to production.

Detection

• NIMDTA user reported having an issue loading TIS.

Resolution

• Delete the schema history entries for the seed data loaded by the Consolidated DR ETL.

• Allow the reference service to restart and re-run Flyway validation.
  
  

  Open

  

Timeline

• May 26, 2021: 12:39 BST - Upgraded Flyway deployed to production

• May 26, 2021: 12:42 BST - Notification in #monitoring-prod, thought to be due to an undeployed ops change.

• May 26, 2021: 12:45 BST - Ops change merged and applied.

• May 26, 2021: 14:22 BST - NIMDTA user reported being unable to use the application

• May 26, 2021: 15:02 BST - Fix deployed.

Root Cause(s)

• Flyway migration validation failed after an upgraded version was deployed.

• There were migrations applied that were missing from the reference service.

• The Consolidated DR ETL used Flyway to load seed data, those scripts were not available to the reference service.

• The combination of behaviour change in latest Flyway versions and the way the migrations were ran caused the validation failure. The validation behaviour is correct, but was not foreseen in this case.

Action Items

Lessons Learned