2020-11-13 NDW and Reval ETL failures

Non-technical summary

While working on a ticket that was related the Azure to Amazon migration, some of the servers needed an update so that they could actually perform a software upgrade when needed as there were some out of date references to locations on the internet that no longer worked/existed, and this was stopping the updating from working.

In the course of this, we fixed these errors, but performed an upgrade by mistake to the whole virtual server, instead of just updating its list of software to include security and bug fixes. The unplanned upgrade had some knock-on impact on our ability to run components of TIS. This was only noticed the next day when the ETL’s tried to run.

Correcting this took a fair bit of time and initially the failure spread to more components as we started to rectify the situation, essentially rendering TIS inaccessible on one of the two load-balanced servers. So whilst we were affecting that change, we temporarily routed everyone to the functioning server. That server had some limitations compared to normal access on TIS - Bulk Uploads wasn’t available and Reval was unavailable due to the ETL’s not running overnight. After the first machine was fixed, both the NDW ETL and the Reval ETL were run successfully.

A later, much shorter incidence occurred when we tried to release new functionality of the TIS application. Later that day we scheduled in the fix to the other virtual server, so as to get back to a fully working system as soon as possible but not interfere with the users of TIS going about there normal daily business.

Timeline

Root Causes

• Accidental major version update to one of our core infrastructure tools caused a failure in a dependent tool. This was ok in itself with the containers that were running, but no new containers could launch, ie ETL’s or newly deployed software versions.

Trigger

• Accidentally upgrading rather than updating the servers that the TIS infrastructure run on.

Resolution

• Remove the server to be worked on from the load balancer so that all inbound TIS traffic is diverted to the working server.

• Stop all docker containers from running

• Restart the server so all of the upgrades/updates could apply correctly

• Remove the old containers

• Remove the docker networks associated with each of those containers

• Applied network fixes to move the network range to a non-overlapping range that wasn't in use with the new AWS infrastructure.

• Restarted all of the containers in sequence. This had to be done in series as, while this process was happening (approximately 1 hour per server), the other server in our infrastructure was in charge of keeping TIS up for everyone else.

• Check all services are up and working

• Allow load balancer to send traffic to the fixed server

• Rinse and repeat with the second server.

Detection

• NDW ETL (Prod) failure alert on Slack

• Reval / GMC sync ETL’s failure alert on Slack

Actions

• keep everything more up to date to avoid major impacts of upgrades in future

• ensure one person is not a single point of failure - required code reviews for infrastructure changes

• specific changes to the architecture to improve resilience:

  • Use of ‘serverless’ technology: ECS, RDS, DocumentDB

  • leverage the AWS fault-tolerant infrastructure

  • decommission old reval as soon as it's possible to

• check STAGE matches PROD upgrade (completed)

Lessons Learned (Good and Bad)

• Good. The load-balanced infrastructure works and we were able to keep TIS mostly working while we performed the fixes.

• Bad. More care to be taken with the commands being issued to the production server.

  • Repeatable playbooks, applied to non-production servers first.

• Bad. Highlighted where we could do with more redundancy in the architecture.

  • Loadbalancer healthchecks aren’t exhaustive

  • more granular health checks are required (this will come with ECS)

• Good. This has been applied to the TIS “2.0” Infrastructure roadmap.