Non-technical Description

Bulk upload service is continually restarted and bulk upload webpage is continually refreshing.

Spreadsheet had lots of rows that were blank other than a hyphen in the address field

Trigger

• There were 3 super large file uploaded and completed with thousands of errors.

Detection

• User queries:

Liam Lofthouse: Morning General - is there an issue with the bulk upload page? I seem… 

posted in TIS Support Channel / General at 21 April 2023 09:28:59

Resolution

• Increated the jvm memory from 0.5G to 4G, and reserved memory (container memory) for bulk upload from 1G to 5G, until the files with lots of errors were no longer loaded on the first bulk upload page

• Backed up the ApplicationType records for those 3 large files.

Timeline

BST unless otherwise stated

• 20 Apr 2023 14:03-14:36 - Several uploads with significant numbers of errors.

• 21 Apr 2023 09:28 - User report that page keeps refreshing

• 21 Apr 2023 09:30 - Found service was running OutOfMemory. Logs didn’t give indication why. Dashboard indicated that there *may* have been an issue since the previous day.

• 21 Apr 2023 09:30-13:00 - Made hotfix changes to deployment configuration to capture additional information. Found a valid cause for the additional memory use. We modified the configuration to give it more resource but missed additional resource constraints.

• 21 Apr 2023 13:49 User report that an uploaded file was stalled. We assumed this was because the service restarted when the file was being processed.

• 21 Apr 2023 14:40-15:40 - Modified and monitored the service further to check that it was stable, notifying the Teams channel at 15:55.

• 21 Apr 2023 16:15 - Got the same issue replicated on Stage after manually uploading those 3 large files (logId: 1682001412239, 1681999967423, 1681999429355 on Prod).

• 21 Apr 2023 16:30 - Modified ApplicationType.errorJson column for uploaded 3 large files on Stage to a single error. And checked Stage was resolved.

• 21 Apr 2023 17:10 - Exported the records of the 3 large filds from table ApplicationType locally for future use

Root Cause(s)

• Intermittently the bulk upload screen on TIS was constantly refreshing

• Admins-UI refreshes screen on errors received from backend services (503 http error from /status API)

• 503 http error indicated that generic upload service was not available.

• In bulk upload service logs: we found java.lang.OutOfMemoryError: Java heap space. When the API /status was called, bulk upload service ran out of memory.

• When someone looks at the status of bulk upload jobs, the errors for the visible jobs are loaded from the database and at this time it included an exceptional amount of data (~180K rows of errors)

• The file uploaded had lots of rows with just a hyphen / didn’t follow the template format

Action Items

Lessons Learned