OutOfMemory thrown but heap dump seems less than limit

Questions about YourKit Java Profiler
Post Reply
PeterJGeraghty
Posts: 2
Joined: Tue Apr 24, 2018 3:14 pm

OutOfMemory thrown but heap dump seems less than limit

Post by PeterJGeraghty »

I have a consistently behaving test which throws OutOfMemoryError: Java heap space. The Yourkit profiler monitoring the application never shows it actually reaching the limit though, it never gets closer than 400MB to the limit (1.8GB). In the hprof created by setting -XX:+HeapDumpOnOutOfMemoryError the heap seems well within the limit (870MB).

I know what the general pattern of the problem is in this case, there is a manipulation of a very large String involving the need to have two Strings of that size at one time.

The reason for my post is wanting to understand better what exactly is being shown in the profiler. Based on what I can see in the heap and reconciling to the code, I don't believe the actual request which failed would have been for as much as 800MB, it should have been more like 300MB. So is the dump showing the heap after some memory has already been freed? Or is it genuinely everything that was on the heap at the time of the failure? In which case is there some other reason for an OutOfMemory even the heap limit had not been reached?

Or can you be sure that the behaviour of the profiler and the dump is fully transparent, and I just need to look harder at my reconciliation with the code?

I am using the Oracle JRE 1.8.0_66 allowing it to take default VM options.

Thanks in advance for your input.
Anton Katilin
Posts: 6172
Joined: Wed Aug 11, 2004 8:37 am

Re: OutOfMemory thrown but heap dump seems less than limit

Post by Anton Katilin »

Hello,

Perhaps there was a single object (obviously, array) allocation which failed. It does not mean that the heap usage at the exception time point was close to the limit.

For example, imagine the heap limit is 100 MB (-Xmx100m), current usage is 30 MB. An attempt to allocate 500 MB array is performed. It will fail with an OOME although 70% of the heap is free at the moment.

To investigate the problem, open the hprof dump created on an OOME in the profiler and switch to the Threads tab. It will show the exact exception stack trace for the OOME as well as stack traces of other threads at the moment.

Best regards,
Anton
PeterJGeraghty
Posts: 2
Joined: Tue Apr 24, 2018 3:14 pm

Re: OutOfMemory thrown but heap dump seems less than limit

Post by PeterJGeraghty »

Thanks Anton - as I would expect, the threads tab shows the same stack trace I had already looked at.

I think you have answered my question - I need to look harder at my analysis of what was happening in the code. Perhaps the exact test data unluckily falls into a bad example of the "doubling of capacity" in the StringBuilder - in requiring just over 300MB it has unluckily attempted to allocate 600MB of which half would turn out to be unnecessary.

Anyway, I have plenty of options for solutions of various kinds, I just wanted to be clear whether there was something about the mechanism of taking the dump or performing the analysis that meant the info should be treated as a heuristic rather than a completely accurate picture - you have reassured me that it is completely accurate. Thanks.
Anton Katilin
Posts: 6172
Joined: Wed Aug 11, 2004 8:37 am

Re: OutOfMemory thrown but heap dump seems less than limit

Post by Anton Katilin »

Thank you for the feedback.
Post Reply