...

• Restarted Service

• Reran NDW and GMC jobs

...

Timeline

• ???

• 12 May 2021: 15:16 BST - Jenkins raising exceptions

• 13 May 2021: 00:17 BST - Jenkins stopped logging

• 13 May 2021: 06:55 BST - Super-Scrum master flagged not only downtime but also additional consequences

  • There was an unrelated failure mentioned (STAGE PersonSync job)

• 13 May 2021: 07:25 BST - Jenkins restarted

• 13 May 2021: 07:30 BST - NDW jobs restarted

• 13 May 2021: 08:24 BST - Question about Revalidation jobs raised on MS Teams

• 13 May 2021: 08:25 BST - NDW jobs finished

• 13 May 2021: 08:50- BST - gmc-sync jobs rerun

• 13 May 2021: 09:27 BST - confirmed with reval users data had been refreshed

• 13 May 2021: 10:27 BST - Downstream NDW ETLs finished

...

• NDW & Reval jobs didn’t run

• Jenkins was unresponsive; service was up but not doing anything

• Possibly not having write permissions?

• Server Memory usage

  • Possibly still more digging to do Reuben Roberts

Lots of builds were running (Reuben Roberts going to look at dependabot builds)

Builds taking longer than they should have done not finishing

• Jenkins had crashed due to an out-of-memory error. From the syslog:

  • May 12 23:18:25 HEE-TIS-VM-JENKINS kernel: [1052434.449590] Out of memory: Kill process 1641 (java) score 49 or sacrifice child
May 12 23:18:25 HEE-TIS-VM-JENKINS kernel: [1052434.454439] Killed process 1641 (java) total-vm:10346296kB, anon-rss:2037444kB, file-rss:0kB
...
May 12 23:19:20 HEE-TIS-VM-JENKINS jenkins: jenkins: fatal: client (pid 1641) killed by signal 9, exiting

• When it restarted, it had lost write-permission on some of the data folders. From the jenkins log:

  • 2021-05-13 06:26:29.918+0000 [id=22] INFO jenkins.model.Jenkins#<init>: deleting obsolete workspace /home/jenkins/data/jenkins/workspace/E_TIS-EsrDataExportService_PR-64
2021-05-13 06:26:30.562+0000 [id=22] WARNING jenkins.model.Jenkins#<init>: Exception in onOnline() for the computer listener class jenkins.branch.WorkspaceLocatorImpl$Collector on the Jenkins master node
Also: java.nio.file.FileSystemException: /home/jenkins/data/jenkins/workspace/E_TIS-EsrDataExportService_PR-64/build/reports/tests/test/packages/com.hee.tis.esr.esrdataexport.integration.notification.html: Operation not permitted

...

Action Items

...