...
: 14:55 BST - Ansible playbook ran to apply changes to
api-gateway: 14:59 BST - Detected by dev team and users
: 14:30 BST - Fix deployed
: Discovered failures in reading files from ESR. There were 3 attempts to trigger processing the file
DE_NWN_APC_20210506_00002693.DATwhich all failed with an SSL Exception.
Root Cause(s)
Ansible playbook replaced latest certificates with outdated ones
The updated certificates were added manually, so the playbook didn’t know about them
...
Action Items | Owner | |
|---|---|---|
Ensure certificates can be applied automatically by playbook | ||
Improve alerting from Lambdas (ESR) | ||
...
Lessons Learned
Make sure anything applied manually will be handled automatically or do it automatically from the start