Date | |
Authors | |
Status | DocumentingDone |
Summary | Reference service failed to start after upgrade: https://hee-tis.atlassian.net/browse/TIS21-2573 |
Impact | Some users were having problems accessing TIS for ~ 10 minutes |
...
An automatically generated Pull Request was approved to update a component used in the reference service. Our build server noticed the change as it should do and the pipeline built and deployed the new application. When the application started in the stage environment it had an error in it and failed and kept restarting until we could revert it back to a known working state.
...
Trigger
Started due to a PR needing to be appliedUpdate to “Springfox” dependency merged and deployed to production
Detection
Slack Alert at 13:14 on
...
The offending component in the reference service was reverted back to the previously known working version and redeployed
...
Timeline
13:07 - Upgraded dependency deployed
13:14 - Alert on Slack: AWS Service 10.160.0.137:8088 is down
13:21 - Docker reports reference container is unhealthy and boot looping (syslog:
2022-01-18 13:21:44.810 WARN 1 --- [ main] ConfigServletWebServerApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'inMemorySwaggerResourcesProvider' defined in URL [jar:file:/app.jar!/WEB-INF/lib/springfox-swagger-common-3.0.0.jar!/springfox/documentation/swagger/web/InMemorySwaggerResourcesProvider.class]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [springfox.documentation.swagger.web.InMemorySwaggerResourcesProvider]: Constructor threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'swaggerSpringfoxApiDocket' defined in class path resource [io/github/jhipster/config/apidoc/SwaggerConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [springfox.documentation.spring.web.plugins.Docket]: Factory method 'swaggerSpringfoxApiDocket' threw exception; nested exception is java.lang.NoSuchMethodError: springfox.documentation.builders.PathSelectors.regex(Ljava/lang/String;)Lcom/google/common/base/Predicate;")13:24 - Andy Dingley creates new PR to revert to known working state
~13:24 - John Simmons (Deactivated) approved PR
13:24 - Jenkins started building repaired version
13:26 - Issue reported by users.
13:27 - Fixed version starts on stage environment and is checked and approved
13:28 - New version deployed to production
13:29 - Fault is fixed in prod and everything is now working as it should be.
13:31 - Users informed of resolution.
...
Root Cause(s)
An Updated update to the Springfox dependency component killed the reference service and wasn't checked fully before being approved into the production environment
No health check to indicate that stage was affected
No manual checks were performed on stage to verify before approval to deploy to production
...
Action Items
...
Lessons Learned
Add some health check monitoring to the pipeline to stop rebooting containers from reaching the production environment.