2020-06-05 Build server delivery pipelines blocked
Date | Jun 5, 2020 |
Authors | @Andy Dingley |
Summary | An authentication issue caused the TIS build server to become blocked from releasing new changes |
Impact | Delayed some new feature work by a couple of days |
Root Cause(s)
Build server unable to authenticate with our source code host (GitHub).
Authentication failed due to a change in GitHub’s API, where it now requires an access token to be provided instead of a password for a specific action - authentication for other actions continued working as normal.
Notifications were sent about usage of a password when calling the API, but went to an inbox that technical members didn’t have access to.
Previous notifications were noted to the technical members but at the time was thought to be due to work being done on a side project that wouldn’t affect TIS.
Trigger
Developer noticed their changes were not being built by the build server.
Resolution
Authentication credentials updated to use an access token instead of password.
Timeline
2020-06-05 09:09 - Issue with single project after a developer saw a failure, Ops notified.
2020-06-05 09:17 - Ops began investigating alongside existing workload.
2020-06-05 13:55 - First indication of a wider problem after a second project failed.
2020-06-05 14:55 - Ops focus on the issue exclusively.
2020-06-05 16:28 - Ops reach out to wider team for assistance.
2020-06-05 ??:??
Updated build server plugins related to source code retrieval.
Verified failing action from a development machine as working.
Updated credentials for source code hosting - authentication appeared to be working as it was able to access private projects but the problem action still failed.
Lots of other configuration tweaks in the build server.
2020-06-05 18:47 - Called it a day for the weekend due to lack of progress/ideas and unavailable of the wider team.
2020-06-08 10:30 - Investigation resumed, expanded to the wider team.
2020-06-08 12:24 - Issue resolved after switching our the password credentials for an access token.
Action Items
No | Action Item | Owner |
---|---|---|
1 | Improve access to GitHub email notifications by forwarding the relevant emails to multiple team members. | @Andy Nash (Unlicensed) |
2 | More intuitive build server configuration - migrate from Jenkins in VM to hosted build tool with better integrations | Vulgabee |
Slack: https://hee-nhs-tis.slack.com/
Jira issues: https://hee-tis.atlassian.net/issues/?filter=14213