Date	2017-04-18
Authors	Alex Dobre
Status	In progress
Summary	The profile service ran out of memory causing live to be inaccessible and the ETL's to fail
Impact	Andy Petherbridge was delayed in his revalidation work.

Root Cause

2017-04-15T12:19:11.860270431Z Exception in thread "Tomcat JDBC Pool Cleaner[868693306:1489998303988]" Exception in thread "http-nio-8082-Acceptor-0" java.lang.OutOfMemoryError: GC overhead limit exceeded
2017-04-15T12:19:11.860316431Z Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Tomcat JDBC Pool Cleaner[868693306:1489998303988]"
2017-04-15T12:19:11.860320831Z
2017-04-18T08:27:26.158664090Z Exception in thread "Thread-5" java.lang.OutOfMemoryError: GC overhead limit exceeded

Trigger

Resolution

For now we have increased the memory available to the profile service, we also created this bug to look into the code see if we have some sort of "memory leaks":

TISDEV-1855 Profile service in live was down because of an out of memory error

Running the GMC-sync ETL's again fixed most of the issues.

Detection

Tried to log into live and saw it was down.

Action Items

Action Item	Type	Owner	Issue
	mitigate/prevent

Timeline

Supporting Information

Mail from Andy:

· The trainee information from Intrepid isn’t showing up on the dashboard. If I go to each trainee individually, it’s all there, but it’s not pulling through on the dashboard.

· The list of doctors Under Notice doesn’t appear to have updated for about a week; I’ve looked on GMC Connect, and we have doctors Under Notice up to 10th August, but on TIS it only goes up to 5th August. I know we had this problem last month, but I thought the issue was resolved (I vaguely remember it was something to do with not being able to update because the cache was full or something)?

The first issue is annoying but doesn’t actually cause an issue because of how I currently work (jointly with spreadsheets) but there’s some trainees I need to recommend urgently that I’d rather do through TIS if possible.

I’ve also just noticed, my Submitted to GMC view is completely blank too – again, annoying but not blocking work.

Browser not supported