Versions Compared

Old Version 4

changes.mady.by.user Phil James

Saved on

compared with

New Version 5

changes.mady.by.user Philip Wilsdon

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Date

Authors

Phil James (Unlicensed)

Status

Resolved

Summary

Database ran out of space and resulted in system failure.

Impact

Users were unable to log into TIS for approx 20 mins

...

Action Items

Action Items

Owner

Fix monitoring:

Alertmanager should send to #monitoring-prod rather than #monitoring?

Uptime robot didn’t report outage until keycloak was unavailable

Error messages need to be clear

Look at disk management

Decide on bigger disk?

Lessons Learned

  • If we want more sophisticated monitoring on services then we have to either see if the API for uptime robot will be able to support this or look at another product. API info here: https://uptimerobot.com/api/