Non-technical Description

A number of nightly jobs didn’t run because the application that runs them died. We restarted the application and manually triggered the jobs that are usually run on a schedule.

Trigger

• Jenkins died: Service Unavailable / not functioning properly

Detection

Build Server: Alerting via slack and team messages (Scrum Master)

Reval: User message on MS Teams

Resolution

• Restarted Service

• Reran NDW and GMC jobs

Timeline

• ???

• 12 May 2021: 15:16 BST - Jenkins raising exceptions

• 13 May 2021: 00:17 BST - Jenkins stopped logging

• 13 May 2021: 06:55 BST - Super-Scrum master flagged not only downtime but also additional consequences

  • There was an unrelated failure mentioned (STAGE PersonSync job)

• 13 May 2021: 07:25 BST - Jenkins restarted

• 13 May 2021: 07:30 BST - NDW jobs restarted

• 13 May 2021: 08:24 BST - Question about Revalidation jobs raised on MS Teams

• 13 May 2021: 08:25 BST - NDW jobs finished

• 13 May 2021: 08:50- BST - gmc-sync jobs rerun

• 13 May 2021: 09:27 BST - confirmed with reval users data had been refreshed

• 13 May 2021: 10:27 BST - Downstream NDW ETLs finished

Root Cause(s)

• NDW & Reval jobs didn’t run

  • Jenkins was unresponsive; service was up but not doing anything

    • Possibly not having write permissions?

    • Server Memory usage

      • Possibly still more digging to do Reuben Roberts

    • Lots of builds were running (Reuben Roberts going to look at dependabot builds)

      • Builds taking longer than they should have done not finishing

Action Items

Lessons Learned

• We’ve done a more complete RCA compared to when a similar outage happened: https://hee-tis.atlassian.net/wiki/spaces/NTCS/pages/1936687204/2020-08-03+TIS+NDW+ETLs+didn+t+run