Date |
| |
Authors | Andy Nash (Unlicensed) Joseph (Pepe) Kelly John Simmons (Deactivated) | |
Status | LiveDefect | In progressComplete. |
Summary | On Friday morning we received a monitoring alert that the NDW ETL (Prod) had failed overnight. Following an initial RCA, we discovered issues with the Blue server (one half of the load-balanced application, to which roughly 50% of traffic is directed) | |
Impact | NDW not updated in Prod, roughly half our users could not access TIS, bulk upload not functioning, (current) Reval not functioning until ETL’s could be run. |
...
| Table of Contents |
|---|
Non-technical summary
On While working on a ticket that was related the Azure to Amazon migration, some of the servers needed an update so that they could actually perform a software upgrade when needed as there were some out of date references to locations on the internet that no longer worked/existed, and this was stopping the upgrading from working.
In the course of this, we fixed these errors, but performed an upgrade upgrade by mistake to the whole virtual server, instead of just updating updating its list of software to be updated, and the unplanned upgrade had some knock-on impact - essentially rendering TIS inaccessible on one of the two load-balanced servers, this . This was only noticed the next day when the ETL’s tried to run and we tried to release new functionality of the TIS application.
Correcting this took a fair bit of time, so . So whilst we were affecting that change, we temporarily routed all traffic to the functioning server. That server had some limitations compared to normal access on TIS - Bulk Uploads wasn’t available and Reval was unavailable due to the ETL’s not running overnight after . After the first machine was fixed, both the NDW ETL and the Reval ETL were run successfully.
...
Thursday 12 November 2020 | John Simmons (Deactivated) Fixed Blue/Green aptitude repository settings on the blue and green servers |
Thursday 12 November 2020 | John Simmons (Deactivated) Accidentally ran the wrong command to update the servers list (apt-get upgrade instead of apt-get update) |
Friday 13 November 2020 | When new docker containers tried to start, they failed (ndw etl and reval etlsNDW ETL and Reval ETLs) |
Friday 13 November 2020 | docker Docker was restarted on green to try and get the services working, but this caused all services to not start back up again. (during this time blue server was working fine) |
Friday 13 November 2020 | found Found that docker had upgraded to the latest version (1 full version) but docker compose by the way it works hadnt hadn't been updated to match (this was then updated the the latest version as well) |
Friday 13 November 2020 | Joseph (Pepe) Kelly removed green server from the loadbalancer to stop people accessing it and get traffic flowing normally to blue |
Friday 13 November 2020 | due Due to some of the changes in the new version of docker, we had to delete all of the containers and the associated docker networks. |
Friday 13 November 2020 | we We applied a fix to change the default ip address ranges docker can use for cross container traffic |
Friday 13 November 2020 | then Then had to restart all containers to get the services back then let green back into the loadbalancer to enable full traffic flow. |
Friday 13 November 2020 | planned Planned to fix blue during out of hours |
Friday 13 November 2020 @10:54am | TCS was deployed into prod, and this killed the blue server |
Friday 13 November 2020 | removed Removed blue server from loadbalancer to allow user to continue to use TIS |
Friday 13 November 2020 @ 3:00pm | performed Performed the same operation that was applied to green, to the blue server and made sure all services were working well |
Friday 13 November 2020 @4:00pm | TIS returned to fully working status |
...
Accidental major version update to one of our core infrastructure tools caused a failure in a dependent tool, this . This was ok in itself with the containers that were running, but no new containers could launch, ie ETL’s or newly deployed software versions.
...
Accidently upgrading rather than updating the servers that the TIS infrastructure run on.
Resolution
Restarting the servers so all of the updates could apply correctly, then having to delete all docker containers/docker networks and restart all the individual microservices on each server, in the correct sequence. This had to be done in series as while this process was happening (approximately 1 hour per server) the other server in our infrastructure was in charge of keeping TIS up for everyone else.
Detection
NDW ETL (Prod) failure alert on Slack
Reval / GMC sync ETL’s failure alert on Slack
Actions
Remove the server to be worked on from the load balancer so that all inbound TIS traffic is diverted to the working server.
Stop all docker containers from running
Restart the server so all of the updates could apply correctly
Remove the old containers
Remove the docker networks associated with each of those containers
Applied network fixes to move the network range to a non-overlapping range that wasn't in use with the new AWS infrastructure.
Restarted all of the containers in sequence. This had to be done in series as, while this process was happening (approximately 1 hour per server), the other server in our infrastructure was in charge of keeping TIS up for everyone else.
Check all services are up and working
Allow load balancer to send traffic to the fixed server
Rinse and repeat with the second server.
Detection
NDW ETL (Prod) failure alert on Slack
Reval / GMC sync ETL’s failure alert on Slack
Actions
[insert actions to take to mitigate this happening in future]
e.g.
keep everything more up to date to avoid major impacts of upgrades in future
ensure one person is not a single point of failure - required code reviews for infrastructure changes
specific changes to the architecture (list them) to improve resilience
Lessons Learned (Good and Bad)
...