...
14:03-14:36 - Several uploads with significant numbers of errors.
09:28 - User report that page keeps refreshing
09:30 - Found service was running
OutOfMemory. Logs didn’t give indication why. Dashboard indicated that there *may* have been an issue since the previous day.09:30-13:00 - Made hotfix changes to deployment configuration to capture additional information. Found a valid cause for the additional memory use. We modified the configuration to give it more resource but missed additional resource constraints.
13:49 User report that a an uploaded file was stalled. We assumed this was because the service restarted when the file was being processed.
14:40-15:40 - Modified and monitored the service further to check that it was stable, notifying the Teams channel at 15:55.
16:15 - Got the same issue replicated on Stage after manually uploading those 3 large files (logId: 1682001412239, 1681999967423, 1681999429355 on Prod).
16:30 - Modified
ApplicationType.errorJsoncolumn for uploaded 3 large files on Stage to a single error. And checked Stage was resolved.
...