Non-technical Description

Access to TIS fails for some users, some of the time, with the browser message ‘Service unavailable’. TIS recovers without intervention a few minutes later; clearing the browser cache may also restore access.

Trigger

Detection

• 

  Teams notifications from users from 7 May 2021 16:17 onwards.

  
  

Resolution

Timeline

• 7 May 2021 16:17 User reports on Teams that TIS is giving a ‘Service unavailable’ error

• 8 May 2021 01:20 Marcello reports noticing the issue while checking that the nightly sync job has completed successfully

• 10 May 2021 08:48 Various user reports of the same issue on Teams

• 10 May 2021 10:19 HEE-TIS-VM-PROD-APPS-GREEN removed from EC2 load balancing cluster

• 10 May 2021 10:24 HEE-TIS-VM-PROD-APPS-GREEN rebooted

• 10 May 2021 10:33 HEE-TIS-VM-PROD-APPS-GREEN added back to EC2 load balancing cluster

• 10 May 2021 10:46 HEE-TIS-VM-PROD-APPS-GREEN docker logging observed

Root Cause(s)

• HEE-TIS-VM-PROD-APPS-GREEN docker service was unexpectedly not recording logs
  

Action Items

Lessons Learned