Non-technical Description

User attempting to do a bulk upload received a timeout error (401). On investigation, it was found that an out-of-date piece of configuration information that was released prior to the issue being experienced. This impacted the bulk upload process. Once the up-to-date configuration was loaded, the problem was addressed.

Trigger

Deploying / Approving a deployment

Detection

User alerted via Teams

Resolution

• Synchronised infrastructure definition from IaC repository used by the build process and reran the CICD pipeline.

Timeline

All times BST unless otherwise indicated.

• “Infrastructure definitions” left out of date.

• 15 Oct 2024 ~12:02-13:30 The configuration used for deploying was manually edited and the pipeline executed. It was then released to production.

• 15 Oct 2024 14:44 User reported problem.  

• 16 Oct 2024 09:10 - 09:26 The Infrastructure Code definitions were updated where they are used by the build process, the pipeline was run and users notified.

5 Whys (or other analysis of Root Cause)

The page was refreshing because API calls returned 401 errors.

401 errors were being returned, probably, because bulk upload could not communicate with other services. Lack of logs from the service defined prevent us saying so with certainty

The bulk upload service was using out-of-date information.

Actions for an earlier Live Defect had not been completed and this meant that builds were using an earlier copy of our infrastructure definition.

Action Items

See also:

Lessons Learned