Automatic memory snapshot on OOME

[vgritsenko]

I see that #638 got fancy new feature [1], which reminded me about another, related feature, I meant to ask before:

Will it be possible to freeze the JVM and capture memory snapshot automatically when JVM attempts to throw OutOfMemoryError?

Will it be possible to implement this feature for JDK1.4?

Thanks,
Vadim

[1] New feature: capture memory snapshots automatically, triggered by user preset memory threshold.

[Anton Katilin]

Will it be possible to freeze the JVM and capture memory snapshot automatically when JVM attempts to throw OutOfMemoryError?

Unfortunately currently existing JVMs do not allow to correctly capture snapshots in such state.

Good news are that there will be a built-in support in Java 6.0 (Mustang) for heap dumping on OutOfMemoryError in some binary format. It possibly will be ported to Java 5.0 and even 1.4.2; if it happens, we'll try to support that format.

Will it be possible to implement this feature for JDK1.4?

Please see my answer in http://forums.yourkit.com/viewtopic.php?p=1453#1453

[vgritsenko]

Good news are that there will be a built-in support in Java 6.0 (Mustang) for heap dumping on OutOfMemoryError in some binary format. It possibly will be ported to Java 5.0 and even 1.4.2;

We can only hope...

if it happens, we'll try to support that format.

Thanks for the info.

Vadim

[attila]

I've implemented something similar in a system I develop: all worker threads catch VirtualMachineError (OOME is a VME), and when one is caught, they do

Code: Select all

public static void terminateVM(VirtualMachineError e)
{
    ...
    new Controller().captureMemorySnapshot(null, false);
    ...
}

The problem is, often there's not enough memory to complete the capture. I tried preallocating a 1MB byte array that I free immediately before executing above statement:

Code: Select all

private static volatile safetyBuffer = new byte[1024*1024];
...
public static void terminateVM(VirtualMachineError e)
{
    ...
    safetyBuffer = null;
    new Controller().captureMemorySnapshot(null, false);
    ...
}

I also provided a facility in other threads to check if snapshot is being taken, and wait() until it completes so that they don't snatch this freed up memory from the snapshot capturing thread. I also bump the priority of the capturing thread to MAX_PRIORITY.

Yet, I still often encounter a problem where the snapshot capture dies with OOME. Could you maybe provide some sort of "preallocation" that we could do on as early as system startup I.e. if I write new Controller().preallocate(), it'd make a best effort to preallocate all internal objects it needs (that'd BTW also load all the classes too)

BTW, present JREs can also dump a binary heap snapshot on exit using the hprof module shipped with JDK:

Code: Select all

java -Xrunhprof:heap=dump,format=b,file=heapdump.bin,doe=y

This is however tricky, as it is possible that your software has shutdown hooks that'll execute on JVM exit and "accidentally" cleanup the leak. Mine does, that's why I wrote code to take a snapshot between the time when OOME is detected and before System.exit() kicks in.

[Anton Katilin]

Hi,

Unfortunately pre-allocation doesn't work, because OOME situation usually means that some actions cannot be successfully performed inside JVM as well as in our agent's code. We can reserve objects, but JVM, that does not do such reservation, will fail, and we cannot affect it of course.

HPROF approach has the same problems as well. It uses JVMTI API, as we do, and JVMTI calls may fail in low resource condition.

As I've said earlier in this thread, there will be a feature in Java 6.0 and possibly in 1.4/1.5, if ported, that dumps heap in HRPOF binary format when OOME happens. The key is that that feature doesn't use common API, instead it performs the dump internally, thus has very low overhead and manages to work in such tricky situations when there are no resources for general JVMTI approach.

Best regards,
Anton

[Anton Katilin]

P.S. This feature already works in latest Java 6.0 Mustang build.
It's turned on via -XX:+HeapDumpOnOutOfMemoryError .

But we do not yet support the format of produced files.

[vgritsenko]

P.S. This feature already works in latest Java 6.0 Mustang build.
It's turned on via -XX:+HeapDumpOnOutOfMemoryError .

Excellent news!

But we do not yet support the format of produced files.

Any plans to support that soon?

[Vladimir Kondratyev]

Any plans to support that soon?

Definitely not in the version 4.5 (currently we are very close to feature freeze). But this feature is a most probable candidate for version after the 4.5.