Non-technical Description

Bulk upload of files on HEE Prod failed to upload, displaying an Unknown Server Error, due to the account used by bulk upload being disabled. Once reenabled, bulk upload went through well.

Trigger

• The account used by bulk upload being disabled.

Detection

• User notification on Teams.

• Open

  

• Errors on the Bulk Upload page.

  Open

  

Resolution

• Disabled user got reenabled in Keycloak.

• bulk_upload user set to active in Profile service DB.

Timeline

• Mar 17, 2021: Users got disabled in usermanagement as part of essential maintenance

• Mar 18, 2021: 12:52 Mike Richardson reported an issue with Bulk upload on Teams.

• Mar 18, 2021: 13:47 Tried re-upload the same file, resulting in the same error.

• Mar 18, 2021: 13:52 User reenabled in keycloak, bulk upload went through.

• Mar 18, 2021: 13:54 Asked users to try and re-upload.

• Mar 18, 2021: 14:18 Mike Richardson reported an Internal Server Error as Reason for import failure when uploading a file.

• Mar 18, 2021: 16:22 Another admin tried to upload a file and the upload fails to complete in a similar way as what Mike initially reported.

• Mar 18, 2021: 17:07 bulk_upload user enabled in profile service, previously updated in keycloak which does not sync the change the same way user management does.

Root Cause(s)

• Bulk upload failed to connect to TCS.

• Bulk upload user was disabled.

• The bulk upload user was listed in a report of potentially outdated accounts.

• The bulk upload has a transformuk email address.

• Email address was never updated.

Action Items

Lessons Learned

• (De)activating users should be done using user management and not directly in Keycloak.