Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.


Date
 
AuthorsPaul Hoang (Unlicensed)
StatusDocument manager / TCS stabilised
SummaryImmediate problem fixed, new ticket in backlog
Impact

Took TCS down, caused resource issues.

Some users may not have their requests complete, so will need to attempt to download again




Impact

  • TCS would have GC issues and fall over
  • Users would have requests fail, making it look like TIS is failing (red toasts)
  • Users would have not downloaded all documents.

Root Causes

  • Document manager making too many requests.
  • Files missing from Azure
  • Double downloads
  • Resources in TCS are locked/blocked

Trigger

  • Trigger multiple bulk downloads

Resolution

  • Check is implemented to see if the file exists
  • Flow has changed to allow browser to download directly from Azure

Detection / Timeline

  • The issues was detected during working hours

Action Items

  • Fix was pushed.
  • New ticket has been created and needs to be worked on

Lessons Learned

What went well

What went wrong

  • Frontend was developed by a person that is no longer here

Where we got lucky

  • .

Supporting information

  • Old flow where the file does not exist in Azure

Image Modified


  • New flow where file does not exist in Azure

Image Modified


  • Old flow where resources a being used up

Image Modified


  • New flow where less resources are being used

Image Modified