...
11:28 Numerous requests to TCS start responding with a “Bad Gateway” error. Logs show that all 3 tasks were receiving and processing some activity.
1229 user reported problem on Teams “TIS being slow / crashing when doing updates - A few of us in Wessex are experiencing issues with TIS when trying to do updates - it keeps crashing and is being really slow loading over the past couple of days.”
1300 Message acknowledged
1328 Last minute when multiple requests to TCS failed. Some errors occurred later, close to 3 pm & 5-6pm
1342 Errors spotted on esr-sentry channel so ESR services stopped to prevent potential errors
1353 Service restarted and reported to users that looks resolved
1410 Users confirmed things back to normal
16:00 Logs relating to ESR checked and ESR services restarted
11:00 Applicant records from 8th Oct sent out
12:00 Notification records from 8th Oct sent out excluding stale
Triggered a 502 error by overloading the system with a high volume of requests to increase CPU utilization
...
5 Whys (or other analysis of Root Cause)
...