Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

Version 1 Next »

Date

Authors

Andy Dingley Reuben Roberts

Status

In progress

Summary

The deployment of some components to our preprod environment was failing due to apparent memory constraints

Impact

No new versions of these components could be deployed for a short period of time (~1 day)

Non-technical Description


Trigger

  • Deployment of run-of-the-mill component updates (and subsequent reversion to known-good builds) failed


Detection

  • Failure notifications in the #notifications-deployments Slack channel


Resolution

  • Temporary fix: increase container memory for the affected components from 0.5GB to 1GB


Timeline

All times in BST unless indicated

  • : 14:39 - Failed deployment for tis-trainee-ndw-exporter reported in #notifications-deployments Slack channel, closely followed by failed deployments for tis-trainee-credentials and tis-trainee-user-management.

  • : 16:13 - Reverts to the updates to these components also fail to deploy.

  • : 11:25 - Redeploy of tis-trainee-ndw-exporter attempted with ECS task configured with 1GB memory instead of 512MB

Root Cause(s)

  • TODO


Action Items

Action Items

Owner

Write-up investigation ticket for managing buildpack versioning

Reuben Roberts

TO DO

PRs for increased memory allocation for affected components (at least 3, but possibly all with small containers)

TIS21-4560 - Getting issue details... STATUS

TO DO


Lessons Learned

  • TODO

  • No labels