...
BST unless otherwise stated
14:03-14:36 - Several uploads with significant numbers of errors led to problems with the service.
09:28 - User report that page keeps refreshing
09:28 - User report that page keeps refreshing30 - Found service was running
OutOfMemory. Logs didn’t give indication why. Dashboard indicated that there *may* have been an issue since the previous day.12:20 - 09:30-13:00 - Made hotfix changes to deployment configuration to capture additional information. Found a valid cause for the additional memory use Modified the upload ser. We modified the configuration to give it more resource but missed additional resource constraints.
13:49 User report that a uploaded file was stalled. We assumed this was because the service restarted when the file was being processed.
14:40-15:40 : - Modified and monitored the service further to check that it was stable, notifying the Teams channel at 15:55.
16:15 - Got the same issue replicated on Stage after manually uploading those 3 large files (logId: 1682001412239, 1681999967423, 1681999429355 on Prod).
16:30 - Modified
ApplicationType.errorJsoncolumn for uploaded 3 large files on Stage to a single error. And checked Stage was resolved.
...
Root Cause(s)
...
Action Items
Action Items | Comments | Owner |
|---|---|---|
Fix up service deployment configuration Preferred: Move the | ||