Date |
|
Authors | |
Status | Documenting |
Summary | Prod blue fell over (Prod green was up) for 12 mins |
Impact | Users could not access TIS for a short period of time |
Prod blue ran out of storage space during the GMC ETL, then fell over when space was freed up.
Prod Blue ran out of storage space and therefore couldn't perform any ETL’s as there was no data space available to store anything locally.
Adewale Adekoya mentioned that some of the Reval users had noticed there was not any data in the Reval part of TIS in the Teams Channel.
Clean out some of the “Large” logs on the server, then reboot.
- 10.17 - Server Logs trimmed and instance Rebooted (server failed to restart correctly)
- 10:33 - Forced restart on instance through AWS console
- 10:34 - First reported in Teams
- 10:38 - Server became responsive again, TIS started working again but Reval overnight workflow had to be rerun
- 10:40 - Restarted GMC-Sync-Prod
- 10:47 - Restarted Intrepid-reval-etl-all-prod
- 11:01 - All ETL services completed and confirmed that data was available.
Storage space was consumed by a huge apache modsecurity log
Action Items | Owner |
|---|---|
Add more monitoring to instance storage.