2021-05-27 NIMDTA TIS not displaying data
Date | May 27, 2021 |
Authors | @Andy Dingley |
Status | Done |
Summary | NIMDTA reference service could not start |
Impact | TIS NIMDTA unavailable |
Non-technical Description
The reference service for NIMDTA was stuck in a cycle attempting to step in our database migration tool. A database configuration value was set incorrectly for the migration to be completed as expected.
Trigger
OrganizationType reference type deployed to production with Flyway script to create required database table and trigger.
Detection
Notification sent to #monitoring-prod.
Resolution
Set the global variable
log_bin_trust_function_creators
toON
in the NIMDTA production database.Delete the
OrganizationType
table to avoid issues from the partial update.Delete the failed migration from the schema history table.
Let the reference service restart and run the migration again with the correct access.
Timeline
May 27, 2021: 12:20 BST - New flyway migration deployed to production
May 27, 2021: 12:23 BST - Notification in #monitoring-prod.
May 27, 2021: 12:42 BST - Call started in #tis-dev-team
May 27, 2021: 12:43 BST - Fix deployed.
Root Cause(s)
Flyway migration validation failed due to a lack of permissions to create a trigger.
Global variable
log_bin_trust_function_creators
on NIMDTA production database was set toOFF
instead ofON
A MySQL restart on 2021-05-13 reset this to the defaulted value
OFF
and it is not clear whether this was manually changed
Action Items
Action Items | Owner |
|
---|---|---|
Ensure any restart/automated process (ansible) sets | @Joseph (Pepe) Kelly |
|
Document how to investigate Flyway migration issues. | @Andy Dingley |
|
|
|
|
|
|
|
Lessons Learned
Slack: https://hee-nhs-tis.slack.com/
Jira issues: https://hee-tis.atlassian.net/issues/?filter=14213