...
Detected by dev team and then reported by users shortly afterwards
...
Resolution
Liban Hirey (Unlicensed) managed to recover new certificates from the stage environment and copied them to the prod blue/green servers
forgotten branch containing the new certificates was pushed and merged to github
Prod tested and found to be working well.
...
Timeline
: 14:55 BST - Ansible playbook ran to apply changes to
api-gateway: 14:59 BST - Detected by dev team and users
: 14:30 BST - Fix deployed
: Discovered failures in reading files from ESR. There were 3 attempts to trigger processing the file
DE_NWN_APC_20210506_00002693.DATwhich all failed with an SSL Exception.
...
Ansible playbook replaced latest certificates with outdated ones
The updated certificates were added manually, so the playbook didn’t know about them
branch containing the new certificates and not been pushed to GitHub
...
Action Items
Action Items | Owner | |
|---|---|---|
Ensure certificates can be applied automatically by playbook | Done | |
Improve alerting from Lambdas (ESR) https://hee-tis.atlassian.net/browse/TIS21-1564 | ||
...