Date | |
Authors | Paul Hoang (Unlicensed) |
Status | Document manager / TCS stabilised |
Summary | Immediate problem fixed, new ticket in backlog |
Impact | Took TCS down, caused resource issues. Some users may not have their requests complete, so will need to attempt to download again |
Impact
- TCS would have GC issues and fall over
- Users would have requests fail, making it look like TIS is failing (red toasts)
- Users would have not downloaded all documents.
Root Causes
- Document manager making too many requests.
- Files missing from Azure
- Double downloads
- Resources in TCS are locked/blocked
Trigger
- Trigger multiple bulk downloads
Resolution
- Check is implemented to see if the file exists
- Flow has changed to allow browser to download directly from Azure
Detection / Timeline
- The issues was detected during working hours
Action Items
- Fix was pushed.
- New ticket has been created and needs to be worked on
Lessons Learned
What went well
What went wrong
- Frontend was developed by a person that is no longer here
Where we got lucky
- .
Supporting information
- Old flow where the file does not exist in Azure
- New flow where file does not exist in Azure
- Old flow where resources a being used up
- New flow where less resources are being used