Date |
|
Authors | |
Status | In progress |
Summary | The deployment of some components to our preprod environment was failing due to apparent memory constraints |
Impact | No new versions of these components could be deployed for a short period of time (~1 day) |
Non-technical Description
Trigger
Deployment of run-of-the-mill component updates (and subsequent reversion to known-good builds) failed
Detection
Failure notifications in the #notifications-deployments Slack channel
Resolution
Temporary fix: increase container memory for the affected components from 0.5GB to 1GB
Timeline
All times in BST unless indicated
: 14:39 - Failed deployment for tis-trainee-ndw-exporter reported in #notifications-deployments Slack channel, closely followed by failed deployments for tis-trainee-credentials and tis-trainee-user-management.
: 16:13 - Reverts to the updates to these components also fail to deploy.
: 11:25 - Redeploy of tis-trainee-ndw-exporter attempted with ECS task configured with 1GB memory instead of 512MB
Root Cause(s)
TODO
Action Items
Action Items | Owner | |
---|---|---|
Write-up investigation ticket for managing buildpack versioning | TO DO | |
PRs for increased memory allocation for affected components (at least 3, but possibly all with small containers) | TO DO |
Lessons Learned
TODO
Add Comment