Date |
|
Authors | |
Status | In progressDone |
Summary | Bulk upload has stopped working resulting in error messages |
Impact | No users can use Bulk Upload |
...
Bulk upload of files on HEE Prod failed to upload, displaying an Unknown Server Error, due to the account used by bulk upload being disabled. Once reenabled, bulk upload went through well.
...
Trigger
KeyCloak issue - once fixed, Bulk upload appeared to workThe account used by bulk upload being disabled.
...
Detection
User notification on Teams.
Errors on the Bulk Upload page.
...
Resolution
...
Disabled user got reenabled in Keycloak.
bulk_uploaduser set to active in Profile service DB.
...
Timeline
: Users got disabled in usermanagement as part of essential maintenance
: 12:52 Mike Richardson reported an issue with Bulk upload on Teams.
: 13:47 Got LOG ID of the file from S3 and tried to Tried re-upload the same file, resulting in the same error.
: 13:52 User reenabled in keycloadkeycloak, bulk upload went through.
: 13:54 Asked users to try and re-upload.
: 14:18 Mike Richardson reported an Internal Server Error as
Reason for import failurewhen uploading a file.: 16:22 Another admin tried to upload a file and the upload fails to complete in a similar way as what Mike initially reported.
: 17:07
bulk_uploaduser enabled in profile service, previously updated in keycloak which does not sync the change the same way user management does.
...
Root Cause(s)
Bulk upload failed to connect to TCS.
Bulk upload user was disabled.
The bulk upload user was listed in a report of potentially outdated accounts.
The bulk upload has a transformuk email address.
Email address was never updated.
...
Action Items
Action Items | Owner |
|---|---|
Update the email address of all sensible users (in prod and stage) | |
Check keycloak to find other users with outdated email addresses and update them (in prod and stage) | |
Create a slack channel with email accounts associated to it to receive notifications for when passwords need resetting or logins are denied etc. |
...
Lessons Learned
(De)activating users should be done using user management and not directly in Keycloak.