Non-technical Description

• One of the servers that the TIS application is load-balanced between became unavailable.

• After contacting AWS Support it turns out the server was shut down due to a scheduled retirement of the instance caused by underlying hardware issues.

• We did not receive any emails from AWS informing us of this scheduled retirement as the address they send emails to (caaa@hee.nhs.uk) is not managed by our team

Trigger

• Server instance state was “stopped” when looking at it in AWS

Detection

• Slack Alert at 4:02 AM on Jan 3, 2022

Resolution

• The server was restarted and started functioning accordingly

Timeline

• Jan 3, 2022 ~04:00 - Components started shutting down

• Jan 3, 2022 04:02 - Alerts triggered on slack

• Jan 4, 2022 08:50 - VM restarted in cloud console. Generic Upload available again.

• Jan 6, 2022 10:35 - Ticket opened with AWS Support

• Jan 6, 2022 10:55 - Response received from AWS Support

Root Cause(s)

• The AMI used by the instance was deleted

• Could this be due to the recent log4j vulnerability?

• AWS notifies us that the instance was stopped due to scheduled retirement caused by an “unrecoverable issue with the underlying hardware”.

Action Items

Lessons Learned

• Make sure we receive emails from AWS instead of it going to an account not managed by our team