Non-technical Description

TIS is split across two different servers, blue and green, requests are balanced across these two servers for performance and resiliance reasons.

The “blue” server ran out of disk space, causing several of our services to stop functioning.

Trigger

• TIS blue server ran out of disk space.

Detection

• Slack monitoring alert.

Resolution

• Removed unused docker images to reduce disk usage.

Timeline

• Nov 18, 2021 00:12 UTC - Notification in #monitoring-prod that TCS and Reference services were down on the blue server

• Nov 18, 2021 08:34 UTC - Issue identified as low disk space

• Nov 18, 2021 08:47 UTC - Issue resolved by deleting old unused docker images

• Nov 18, 2021 10:11 UTC - Preventative action taken on green server to reduce similar disk usage

Root Cause(s)

• Blue server ran out of disk space

• Large number of outdated docker images

• We have no process to clean old docker images

• We have inadequate monitoring on disk/storage usage

Action Items

Lessons Learned

• We need better monitoring to pre-emptively warn us before disk space limitations cause downtime.