...
03:00 - 03:40 BST #monitoring-ndw Slack channel alert that the first of three overnight jobs failed.
07:53 BST Those jobs were restarted.
09:11 BST All jobs had completed successfully.
03:00 - 03:40 BST #monitoring-ndw Slack channel alert that the first of three overnight jobs failed (unfortunately the broken task images were still ‘live’ for the services)
06:30 BST These jobs were restarted.
07:30 BST All jobs had completed successfully.
09:06 BST The revised component deployed successfully, and tested on staging
Root Cause(s)
A new release of the TIS-NDW_ETL component was deployed the previous afternoon
Two assumptions triggered the subsequent failure of the ETL jobs:
That merging the pull request would only deploy the revised component to the staging environment
That no breaking-changes in 3rd party component would cause the Docker images to be invalid
In fact, the new version of the component is deployed to all three environments (staging, production and NIMDTA). In general this should be acceptable, since the job is only run at 2am, so there should be scope to test the new deployment on staging by force-running the task, with time to roll back if necessary before the nightly scheduled run.
A breaking change to the Paketo image builder rendered the new component images invalid, throwing the following error on startup:
2021-09-29T02:40:40.620Z # JRE version: (8.0_302-b08) (build )2021-09-29T02:40:40.621Z # Java VM: OpenJDK 64-Bit Server VM (25.302-b08 mixed mode linux-amd64 compressed oops)2021-09-29T02:40:40.621Z # Problematic frame:2021-09-29T02:40:40.623Z # V [libjvm.so+0xb2a2f7] VirtualMemoryTracker::add_reserved_region(unsigned char*, unsigned long, NativeCallStack const&, MemoryType, bool)+0x772021-09-29T02:40:40.623Z #2021-09-29T02:40:40.623Z # Core dump written. Default location: /workspace/core or core.1
The fix is documented here: https://github.com/paketo-buildpacks/libjvm/pull/105Workarounds to (a) temporarily force the specific (older) version of the builder or (b) to migrate the component to Java 11 were explored.
(a) Amending the pom.xml file as per:<plugin> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-maven-plugin</artifactId> <configuration> <image> <builder>gcr.io/paketo-buildpacks/builder:0.1.174-base</builder> </image> </configuration>was successful, but would need to be reverted once a new working version of the 3rd party component was released.
(b) With concomitant version changes for groovy etc., this was successful, and was the preferred method. This was implemented and a revised functional image of the component deployed 2021-09-30 8:30 AM BST
...
Action Items | Owner | |
|---|---|---|
Rebuild tis-ndw-etl using Java 11 | Done | |
Possibly update component documentation to flag-up deployment process (to mitigate against future assumptions causing this sort of issue) | Done | |
...
Lessons Learned
Assumptions as to the CI/CD pathway for a specific component can be problematic.
It would be useful to flag-up non-standard deployment paths in component documentation