Non-technical Description

• An automatically generated Pull Request was approved to update a component used in the reference service. Our build server noticed the change as it should do and the pipeline built and deployed the new application. When the application started in the stage environment it had an error in it and failed and kept restarting until we could revert it back to a known working state.

...

Trigger

• Started due to a PR needing to be appliedUpdate to “Springfox” dependency merged and deployed to production

Detection

• Slack Alert at 13:14 on 18 Jan 2022

...

Resolution

• The offending component in the reference service was reverted back to the previously known working version and redeployed

...

Timeline

• 18 Jan 2022 13:07 - Upgraded dependency deployed

• 18 Jan 2022 13:14 - Alert on Slack: AWS Service 10.160.0.137:8088 is down

• 18 Jan 2022 13:21 - Docker reports reference container is unhealthy and boot looping (syslog: 2022-01-18 13:21:44.810 WARN 1 --- [ main] ConfigServletWebServerApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'inMemorySwaggerResourcesProvider' defined in URL [jar:file:/app.jar!/WEB-INF/lib/springfox-swagger-common-3.0.0.jar!/springfox/documentation/swagger/web/InMemorySwaggerResourcesProvider.class]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [springfox.documentation.swagger.web.InMemorySwaggerResourcesProvider]: Constructor threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'swaggerSpringfoxApiDocket' defined in class path resource [io/github/jhipster/config/apidoc/SwaggerConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [springfox.documentation.spring.web.plugins.Docket]: Factory method 'swaggerSpringfoxApiDocket' threw exception; nested exception is java.lang.NoSuchMethodError: springfox.documentation.builders.PathSelectors.regex(Ljava/lang/String;)Lcom/google/common/base/Predicate;")

• 18 Jan 2022 13:24 - Andy Dingley creates new PR to revert to known working state

• 18 Jan 2022 ~13:24 - John Simmons (Deactivated) approved PR

• 18 Jan 2022 13:24 - Jenkins started building repaired version

• 18 Jan 2022 13:26 - Issue reported by users.

• 18 Jan 2022 13:27 - Fixed version starts on stage environment and is checked and approved

• 18 Jan 2022 13:28 - New version deployed to production

• 18 Jan 2022 13:29 - Fault is fixed in prod and everything is now working as it should be.

• 18 Jan 2022 13:31 - Users informed of resolution.

...

Root Cause(s)

• An Updated update to the Springfox dependency component killed the reference service and wasn't checked fully before being approved into the production environment

• No health check to indicate that stage was affected

• No manual checks were performed on stage to verify before approval to deploy to production

...

Action Items

...

Lessons Learned

•  Add some health check monitoring to the pipeline to stop rebooting containers from reaching the production environment.