Versions Compared

Old Version 1

changes.mady.by.user Andy Nash

Saved on

compared with

New Version Current

changes.mady.by.user Joseph (Pepe) Kelly

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
In progress

Date

Authors

Marcello Fabbri (Unlicensed)

Status

Done

Summary

Bulk upload has stopped working resulting in error messages

Impact

No users can use Bulk Upload

...

Non-technical Description

Bulk upload of files on HEE Prod failed to upload, displaying an Unknown Server Error, due to the account used by bulk upload being disabled. Once reenabled, bulk upload went through well.

...

Trigger

  • The account used by bulk upload being disabled.

...

Detection

  • User notification on Teams.

  • Image Added

  • Errors on the Bulk Upload page.

    Image Added

...

Resolution

  • Disabled user got reenabled in Keycloak.

  • bulk_upload user set to active in Profile service DB.

...

Timeline

  • : Users got disabled in usermanagement as part of essential maintenance

  • : 12:52 Mike Richardson reported an issue with Bulk upload on Teams.

  • : 13:47 Tried re-upload the same file, resulting in the same error.

  • : 13:52 User reenabled in keycloak, bulk upload went through.

  • : 13:54 Asked users to try and re-upload.

  • : 14:18 Mike Richardson reported an Internal Server Error as Reason for import failure when uploading a file.

  • : 16:22 Another admin tried to upload a file and the upload fails to complete in a similar way as what Mike initially reported.

  • : 17:07 bulk_upload user enabled in profile service, previously updated in keycloak which does not sync the change the same way user management does.

...

Root Cause(s)

  • Bulk upload failed to connect to TCS.

  • Bulk upload user was disabled.

  • The bulk upload user was listed in a report of potentially outdated accounts.

  • The bulk upload has a transformuk email address.

  • Email address was never updated.

...

Action Items

Action Items

Owner

Update the email address of all sensible users (in prod and stage)

John Simmons (Deactivated)

Check keycloak to find other users with outdated email addresses and update them (in prod and stage)

Edward Barclay

Create a slack channel with email accounts associated to it to receive notifications for when passwords need resetting or logins are denied etc.

John Simmons (Deactivated)

...

Lessons Learned

  • (De)activating users should be done using user management and not directly in Keycloak.