2020-06-05 Build server delivery pipelines blocked

Date

Jun 5, 2020

Authors

@Andy Dingley

Summary

An authentication issue caused the TIS build server to become blocked from releasing new changes

Impact

Delayed some new feature work by a couple of days

 

Root Cause(s)

  • Build server unable to authenticate with our source code host (GitHub).

  • Authentication failed due to a change in GitHub’s API, where it now requires an access token to be provided instead of a password for a specific action - authentication for other actions continued working as normal.

  • Notifications were sent about usage of a password when calling the API, but went to an inbox that technical members didn’t have access to.

  • Previous notifications were noted to the technical members but at the time was thought to be due to work being done on a side project that wouldn’t affect TIS.

Trigger

  • Developer noticed their changes were not being built by the build server.

Resolution

  • Authentication credentials updated to use an access token instead of password.

Timeline

  • 2020-06-05 09:09 - Issue with single project after a developer saw a failure, Ops notified.

  • 2020-06-05 09:17 - Ops began investigating alongside existing workload.

  • 2020-06-05 13:55 - First indication of a wider problem after a second project failed.

  • 2020-06-05 14:55 - Ops focus on the issue exclusively.

  • 2020-06-05 16:28 - Ops reach out to wider team for assistance.

  • 2020-06-05 ??:??

    • Updated build server plugins related to source code retrieval.

    • Verified failing action from a development machine as working.

    • Updated credentials for source code hosting - authentication appeared to be working as it was able to access private projects but the problem action still failed.

    • Lots of other configuration tweaks in the build server.

  • 2020-06-05 18:47 - Called it a day for the weekend due to lack of progress/ideas and unavailable of the wider team.

  • 2020-06-08 10:30 - Investigation resumed, expanded to the wider team.

  • 2020-06-08 12:24 - Issue resolved after switching our the password credentials for an access token.

Action Items

No

Action Item

Owner

No

Action Item

Owner

1

Improve access to GitHub email notifications by forwarding the relevant emails to multiple team members.

@Andy Nash (Unlicensed)

2

More intuitive build server configuration - migrate from Jenkins in VM to hosted build tool with better integrations

Vulgabee