Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 3 Next »

Date

Authors

Andy Dingley Liban Hirey (Unlicensed)

Status

In Progress

Summary

The SSL certificates used by TIS were accidentally overwritten with outdated certificates

Impact

Users unable to access TIS

Non-technical Description

SSL certificates are used by websites to ensure that communication to/from the website is secure, each certificate is valid for a certain length of time.

The certificates used for TIS were due to expire on the 15th April and were replaced on the 6th April.

While making some unrelated changes the outdated certificates were accidentally deployed back to TIS.

Trigger

  • api-gateway playbook ran, removing manually updated certificates

Detection

  • Detected by dev team and then reported by users shortly afterwards


Resolution

Timeline

  • : 14:55 BST - Ansible playbook ran to apply changes to api-gateway

  • : 14:59 BST - Detected by dev team and users

  • : 14:30 BST - Fix deployed

  • : Discovered failures in reading files from ESR. There were 3 attempts to trigger processing the file DE_NWN_APC_20210506_00002693.DAT which all failed with an SSL Exception.

Root Cause(s)

  • Ansible playbook replaced latest certificates with outdated ones

  • The updated certificates were added manually, so the playbook didn’t know about them

Action Items

Action Items

Owner

Ensure certificates can be applied automatically by playbook

Improve alerting from Lambdas (ESR) https://hee-tis.atlassian.net/browse/TIS21-1564

Joseph (Pepe) Kelly

Lessons Learned

  • Make sure anything applied manually will be handled automatically or do it automatically from the start

  • No labels

Slack: https://hee-nhs-tis.slack.com/
Jira issues: https://hee-tis.atlassian.net/issues/?filter=14213