Non-technical Description

The reference service for NIMDTA was stuck in a cycle attempting to start-up due to an issue with our database migration tool. An upgrade to the migration tool changed the behaviour for missing migration steps, causing a failed start-up instead of ignoring the missing steps as it previously did.

Trigger

• Upgraded Flyway version deployed to production.

Detection

• NIMDTA user reported having an issue loading TIS.

Resolution

• Delete the schema history entries for the seed data loaded by the Consolidated DR ETL.

• Allow the reference service to restart and re-run Flyway validation.
  

  

Timeline

• 26 May 2021: 12:39 BST - Upgraded Flyway deployed to production

• 26 May 2021: 12:42 BST - Notification in #monitoring-prod, thought to be due to an undeployed ops change.

• 26 May 2021: 12:45 BST - Ops change merged and applied.

• 26 May 2021: 14:22 BST - NIMDTA user reported being unable to use the application

• 26 May 2021: 15:02 BST - Fix deployed.

Root Cause(s)

• Flyway migration validation failed after an upgraded version was deployed.

• There were migrations applied that were missing from the reference service.

• The Consolidated DR ETL used Flyway to load seed data, those scripts were not available to the reference service.

• The combination of behaviour change in latest Flyway versions and the way the migrations were ran caused the validation failure. The validation behaviour is correct, but was not foreseen in this case.

Action Items

Lessons Learned