Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 4 Next »

Date

Authors

Phil James (Unlicensed)

Status

Resolved

Summary

Database ran out of space and resulted in system failure.

Impact

Users were unable to log into TIS for approx 20 mins

https://hee-tis.atlassian.net/browse/TISNEW-5417

Root Cause(s)

  • Database ran out space

    • Slow logs seemed to take a disproportionate amount of space

Trigger

  • BAU? Not clear anything inparticular caused a jump in usage?

Resolution

  • Deleted some log files to clear space

Detection

  • User reported at 12

  • Uptime robot, once we took key cloak down

Timeline

  • 12:00 Users reported being unable to access TIS

  • 12:07 fire fire call started

  • 12:10ish restarted keycloak

  • 12:15ish Sachin spots SQL DB is full

  • 12:20 ish stuff is removed from database and it starts working again

Action Items

Action Items

Owner

Fix monitoring:

Alertmanager should send to #monitoring-prod rather than #monitoring?

Uptime robot didn’t report outage until keycloak was unavailable

Error messages need to be clear

Look at disk management

Decide on bigger disk?

Lessons Learned

  • No labels

0 Comments

You are not logged in. Any changes you make will be marked as anonymous. You may want to Log In if you already have an account.
 

Slack: https://hee-nhs-tis.slack.com/
Jira issues: https://hee-tis.atlassian.net/issues/?filter=14213