Versions Compared

Old Version 4

changes.mady.by.user Andy Dingley

Saved on

compared with

New Version Current

changes.mady.by.user Andy Nash

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
In progress

Date

Authors

John Simmons (Deactivated) Doris.Wong Andy Dingley

Status

Done

Summary

Logging has been mis-configured in some services to generate far too many logs

Impact

This uses too much space, and therefore has an impact on cost. Specifically of CloudWatch

...

Action Items

Owner

  • Change from using gmcReferenceNumber as ID as it can be ‘null’

Doris.Wong

  • Monitoring on CloudWatch to send alerts based on cost threshold (monthly average + 10%)?

John Simmons (Deactivated) https://hee-tis.atlassian.net/browse/TIS21-1384

  • Re-enable tis-revalidation-connections cloudwatch logging (recreate log groups)

Andy Dingley https://hee-tis.atlassian.net/browse/TIS21-1382

  • Dashboard to display storage quotas (S3, Cloudwatch logs, EFS, EBS)

Andy Dingley https://hee-tis.atlassian.net/browse/TIS21-1383

...

Lessons Learned

  • Exercise caution when configuring logging, and test that the logs generated are logging only what is intended

  • Potential team sharing on the correlation between logging configuration and log sizes (over time) and costs

  • Monitoring on CloudWatch would be useful to highlight these kinds of issues before they result in high costs