Date |
|
Authors | Andy Nash (Unlicensed) Joseph (Pepe) Kelly John Simmons (Deactivated) |
Status | LiveDefect In progress. |
Summary | On Friday morning we received a monitoring alert that the NDW ETL (Prod) had failed overnight. Following an initial RCA, we discovered issues with the Blue server (one half of the load-balanced application, to which roughly 50% of traffic is directed) |
Impact | NDW not updated in Prod, roughly half our users could not access TIS, bulk upload not functioning, (current) Reval not functioning until ETL’s could be run. |
Non-technical summary
On working on a ticket that was related the Azure to Amazon migration, some of the servers needed an update so that they could actually perform a software upgrade when needed as there were some out of date references to locations on the internet that no longer worked/existed, and this was stopping the upgrading from working.
In the course of this, we fixed these errors, but performed an upgrade by mistake to the whole virtual server, instead of just updating its list of software to be updated, and the unplanned upgrade had some knock-on impact - essentially rendering TIS inaccessible on one of the two load-balanced servers, this was only noticed the next day when the ETL’s tried to run and we tried to release new functionality of the TIS application.
Correcting this took a fair bit of time, so whilst we were affecting that change, we temporarily routed all traffic to the functioning server. That server had some limitations compared to normal access on TIS - Bulk Uploads wasn’t available and Reval was unavailable due to the ETL’s not running overnight after the first machine was fixed both the NDW ETL and the Reval ETL were run successfully.
Later that day we scheduled in the fix to the other virtual server, so as to get back to a fully working system as soon as possible but not interfere with the users of TIS going about there normal daily business.
Timeline
Thursday 12 November 2020 | John Simmons (Deactivated) Fixed Blue/Green aptitude repository settings on the blue and green servers |
Thursday 12 November 2020 | John Simmons (Deactivated) Accidentally ran the wrong command to update the servers list (apt-get upgrade instead of apt-get update) |
Friday 13 November 2020 | When new docker containers tried to start, they failed (ndw etl and reval etls) |
Friday 13 November 2020 | docker was restarted on green to try and get the services working, but this caused all services to not start back up again. (during this time blue server was working fine) |
Friday 13 November 2020 | found that docker had upgraded to the latest version (1 full version) but docker compose by the way it works hadnt been updated to match (this was then updated the the latest version as well) |
Friday 13 November 2020 | Joseph (Pepe) Kelly removed green server from the loadbalancer to stop people accessing it and get traffic flowing normally to blue |
Friday 13 November 2020 | due to some of the changes in the new version of docker, we had to delete all of the containers and the associated docker networks. |
Friday 13 November 2020 | we applied a fix to change the default ip address ranges docker can use for cross container traffic |
Friday 13 November 2020 | then had to restart all containers to get the services back then let green back into the loadbalancer to enable full traffic flow. |
Friday 13 November 2020 | planned to fix blue during out of hours |
Friday 13 November 2020 @10:54am | TCS was deployed into prod, and this killed the blue server |
Friday 13 November 2020 | removed blue server from loadbalancer to allow user to continue to use TIS |
Friday 13 November 2020 @ 3:00pm | performed the same operation that was applied to green, to the blue server and made sure all services were working well |
Friday 13 November 2020 @4:00pm | TIS returned to fully working status |
Root Causes
Accidental major version update to one of our core infrastructure tools caused a failure in a dependent tool, this was ok in itself with the containers that were running, but no new containers could launch, ie ETL’s or newly deployed software versions.
Trigger
Accidently upgrading rather than updating the servers that the TIS infrastructure run on.
Resolution
Restarting the servers so all of the updates could apply correctly, then having to delete all docker containers/docker networks and restart all the individual microservices on each server, in the correct sequence. This had to be done in series as while this process was happening (approximately 1 hour per server) the other server in our infrastructure was in charge of keeping TIS up for everyone else.
Detection
NDW ETL (Prod) failure alert on Slack
Reval / GMC sync ETL’s failure alert on Slack
Actions
Remove the server to be worked on from the load balancer so that all inbound TIS traffic is diverted to the working server.
Stop all docker containers from running
Restart the server
Remove the old containers
Remove the docker networks associated with each of those containers
Applied network fixes to move the network range to a non-overlapping range that wasn't in use with the new AWS infrastructure.
Restarted all of the containers in sequence
Check all services are up and working
Allow load balancer to send traffic to the fixed server
Rinse and repeat with the second server.
Lessons Learned (Good and Bad)
Good. The load-balanced infrastructure works and we were able to keep TIS mostly working while we performed the fixes.
Bad. More care to be taken with the commands being issued to the production server.
Bad. Highlighted where we could do with more redundancy in the architecture.
Good. This has been applied to the TIS “2.0” Infrastructure roadmap.
Add Comment