Non-technical Description

The reference service for NIMDTA was stuck in a cycle attempting to step in our database migration tool. A database configuration value was set incorrectly for the migration to be completed as expected.

Trigger

• OrganizationType reference type deployed to production with Flyway script to create required database table and trigger.

Detection

• Notification sent to #monitoring-prod.

Resolution

• Set the global variable log_bin_trust_function_creators to ON in the NIMDTA production database.

• Delete the OrganizationType table to avoid issues from the partial update.

• Delete the failed migration from the schema history table.

• Let the reference service restart and run the migration again with the correct access.

• 

  
  

Timeline

• 27 May 2021: 12:20 BST - New flyway migration deployed to production

• 27 May 2021: 12:23 BST - Notification in #monitoring-prod.

• 27 May 2021: 12:42 BST - Call started in #tis-dev-team

• 27 May 2021: 12:43 BST - Fix deployed.

Root Cause(s)

• Flyway migration validation failed due to a lack of permissions to create a trigger.

• Global variable log_bin_trust_function_creators on NIMDTA production database was set to OFF instead of ON

• A MySQL restart on 2021-05-13 reset this to the defaulted value OFF and it is not clear whether this was manually changed

Action Items

Lessons Learned