Date |
|
Authors | |
Status | Documenting |
Summary | The TIS profile service went down |
Impact | TIS could not be used at all |
Non-technical Description
Our “Profile” service, which is used to check user permissions, went down due to a breaking change being deployed. As user permissions could not be checked the TIS application blocked all users actions, to a user this would have appeared like a log in failure.
Trigger
A change was deployed to
tis-profilewhich caused the service to fail to start
Detection
Slack notification
Resolution
Reverted the breaking change
Timeline
: 14:03 - Breaking change deployed to production.
: 14:10 - Notification sent to slack channel
#monitoring-prod: 14:11 - Issue picked up by dev team.
: 14:17 - Fix deployed to production.
Root Cause(s)
Profile service failed to start
Change to the Sentry configuration caused a breaking change
…
…
Action Items
Action Items | Owner |
|---|---|
Lessons Learned
Test changes properly locally and on stage before pushing to production.
Add Comment