Fully featured low overhead profiler for Java EE and Java SE platforms.
Performance monitoring and profiling of Jenkins, TeamCity, Gradle, Maven, Ant, JUnit and TestNG.
Easy to use performance and memory profiler for .NET framework.

OutOfMemory thrown but heap dump seems less than limit

Moderators: Vladimir Kondratyev, Anton Katilin

OutOfMemory thrown but heap dump seems less than limit

Postby PeterJGeraghty » Tue Apr 24, 2018 5:07 pm

I have a consistently behaving test which throws OutOfMemoryError: Java heap space. The Yourkit profiler monitoring the application never shows it actually reaching the limit though, it never gets closer than 400MB to the limit (1.8GB). In the hprof created by setting -XX:+HeapDumpOnOutOfMemoryError the heap seems well within the limit (870MB).

I know what the general pattern of the problem is in this case, there is a manipulation of a very large String involving the need to have two Strings of that size at one time.

The reason for my post is wanting to understand better what exactly is being shown in the profiler. Based on what I can see in the heap and reconciling to the code, I don't believe the actual request which failed would have been for as much as 800MB, it should have been more like 300MB. So is the dump showing the heap after some memory has already been freed? Or is it genuinely everything that was on the heap at the time of the failure? In which case is there some other reason for an OutOfMemory even the heap limit had not been reached?

Or can you be sure that the behaviour of the profiler and the dump is fully transparent, and I just need to look harder at my reconciliation with the code?

I am using the Oracle JRE 1.8.0_66 allowing it to take default VM options.

Thanks in advance for your input.
PeterJGeraghty
 
Posts: 2
Joined: Tue Apr 24, 2018 3:14 pm

Re: OutOfMemory thrown but heap dump seems less than limit

Postby Anton Katilin » Wed Apr 25, 2018 7:02 am

Hello,

Perhaps there was a single object (obviously, array) allocation which failed. It does not mean that the heap usage at the exception time point was close to the limit.

For example, imagine the heap limit is 100 MB (-Xmx100m), current usage is 30 MB. An attempt to allocate 500 MB array is performed. It will fail with an OOME although 70% of the heap is free at the moment.

To investigate the problem, open the hprof dump created on an OOME in the profiler and switch to the Threads tab. It will show the exact exception stack trace for the OOME as well as stack traces of other threads at the moment.

Best regards,
Anton
Anton Katilin
 
Posts: 5816
Joined: Wed Aug 11, 2004 8:37 am

Re: OutOfMemory thrown but heap dump seems less than limit

Postby PeterJGeraghty » Wed Apr 25, 2018 9:05 am

Thanks Anton - as I would expect, the threads tab shows the same stack trace I had already looked at.

I think you have answered my question - I need to look harder at my analysis of what was happening in the code. Perhaps the exact test data unluckily falls into a bad example of the "doubling of capacity" in the StringBuilder - in requiring just over 300MB it has unluckily attempted to allocate 600MB of which half would turn out to be unnecessary.

Anyway, I have plenty of options for solutions of various kinds, I just wanted to be clear whether there was something about the mechanism of taking the dump or performing the analysis that meant the info should be treated as a heuristic rather than a completely accurate picture - you have reassured me that it is completely accurate. Thanks.
PeterJGeraghty
 
Posts: 2
Joined: Tue Apr 24, 2018 3:14 pm

Re: OutOfMemory thrown but heap dump seems less than limit

Postby Anton Katilin » Wed Apr 25, 2018 9:43 am

Thank you for the feedback.
Anton Katilin
 
Posts: 5816
Joined: Wed Aug 11, 2004 8:37 am


Return to Java Profiler

Who is online

Users browsing this forum: Google [Bot] and 15 guests