Allocated All Pools and HotSpot Anonymous Classes

[projectvalhalla]

Hi,

I am generating classes dynamically on the fly with Oracle's JDK 8 Update 11 (on Windows 7 64-bit) and loading them as "HotSpot Anonymous Classes" via sun.misc.Unsafe.defineAnonymousClass. (btw. this is also the mechanics used by Java 8 to load dynamically generated classes for lambda expressions)

Under JDK8, the class is correctly freed when no instance of it is hanging around and no reference to the Class object exists anymore. I can see that in the Memory/Classes view in YourKit profier. The number of currently loaded classes grows and shrinks accordingly.

But, the "Allocated All Pools" in the Memory/Heap Memory view increases steadily very fast until it plateaus at about 3.2GB and after then increases only moderately.

Windows reports a peak virtual memory usage of about 400MB at max at anytime, and not the whopping 3.2GB.

Is this an issue with the JVM reporting a wrong memory usage to YourKit via JVMTI or something or is it an issue with YourKit?

Cheers.

[projectvalhalla]

I have prepared a test rig to reproduce the error. Since I did not find a way to upload a .zip file I paste the test Java code below.

Note that you will need the ASM library, which can be downloaded from here http://download.forge.ow2.org/asm/asm-5.0.3-bin.zip
There you need /asm-5.0.3/lib/asm-5.0.3.jar .

Running the code under JDK8 will define a hell of a lot of dynamically generated classes (many hundreds of thousands to illustrate the issue) using the HotSpot Anonymous Classes feature described here: https://blogs.oracle.com/jrose/entry/an ... _in_the_vm

When profiling the test, you will see (I tested with the latest Early Access Build of YourKit Java Profiler 14082) that the "Allocated All Pools" metric goes through the roof when actually only about 100MB of virtual memory is allocated by the Java process.

Code: Select all

import java.lang.reflect.*;
import org.objectweb.asm.*;

public class Test implements Opcodes {
    static Object theUnsafe;
    static Method defineAnonymousClass;
    static byte[] bytecode;
    static {
        Object unsafe;
        try {
            Class<?> unsafeClass = ClassLoader.getSystemClassLoader().loadClass("sun.misc.Unsafe");
            Field theUnsafeField = unsafeClass.getDeclaredField("theUnsafe");
            theUnsafeField.setAccessible(true);
            unsafe = theUnsafeField.get(null);
            try {
                defineAnonymousClass = unsafeClass.getDeclaredMethod("defineAnonymousClass", Class.class, byte[].class, Object[].class);
            } catch (NoSuchMethodException e) {
                throw new AssertionError("Must use at least JDK7");
            }
        } catch (Exception e) {
            e.printStackTrace();
            throw new AssertionError("Something went wrong!");
        }
        theUnsafe = unsafe;
        bytecode = generateBytecode();
    }

    static byte[] generateBytecode() {
        ClassWriter cw = new ClassWriter(ClassWriter.COMPUTE_MAXS | ClassWriter.COMPUTE_FRAMES);
        cw.visit(V1_1, ACC_PUBLIC | ACC_SUPER, "GenerateClass", null, "java/lang/Object", null);
        MethodVisitor ctor = cw.visitMethod(ACC_PUBLIC, "<init>", "()V", null, null);
        ctor.visitCode();
        ctor.visitVarInsn(ALOAD, 0);
        ctor.visitMethodInsn(INVOKESPECIAL, "java/lang/Object", "<init>", "()V", false);
        ctor.visitInsn(RETURN);
        ctor.visitMaxs(-1, -1);
        ctor.visitEnd();
        cw.visitEnd();
        return cw.toByteArray();
    }

    static <T> Class<T> defineAnonymousClass(Class<?> hostClass, byte[] bytecode) throws Exception {
        return (Class<T>) defineAnonymousClass.invoke(theUnsafe, hostClass, bytecode, null);
    }

    public static void main(String[] args) throws Exception {
        int count = 99999999;
        for (int i = 0; i < count; i++) {
            defineAnonymousClass(Test.class, bytecode);
        }
    }
}

[Anton Katilin]

Hello,

Thank you for preparing the example. We see the numbers, and they are correct.

These numbers are reported by JVM and shown as is. You can run jconsole to compare.

Please note that you can turn off the allocated memory curve, to see used only. Hover the graph and use the gear icon in the right upper corner.

Windows reports a peak virtual memory usage of about 400MB at max at anytime

What you see in Task manager is likely the Memory column which shows the physical memory usage. Please customize shown columns in Task Manager to include Commit size column too, which represents allocated virtual memory. You'll see these gigabytes in it.

Best regards,
Anton

[projectvalhalla]

Thanks for your reply and the hint with the column in Task Manager!

In this case, it is definitely a bug with the JVM, I would say.

I just tried running that test under Windows XP, SP3, 32-bit and Oracle JDK7 update 60 and things are strage there, too.

Even though no more memory is committed by the Java process as millions of classes are being defined, the JVM also does not seem to report to YourKit that those classes are also being unloaded.

That is to say, the value of the currently loaded classes in YourKit Profiler (Memory/Classes) keeps on increasing. (currently I have reached the 15 Million mark and the memory stays solidly at 43.096KB)

[Anton Katilin]

Can you confirm that all is OK with Java 8, but there is a problem with Java 7 that it does not count unloaded classes?

We also see for your example that on Java 7 the number of "loaded" classes constantly grows, and there are no unloaded classes reported. The same numbers in jconsole - this must be a bug in Java 7 java.lang.management.ClassLoadingMXBean.getUnloadedClassCount()

Interesting, the classes are actually unloaded on Java 7 too. You can check this in the profiler: open Memory tab, Class list, press Refresh, select line java.lang.Class and start pressing Force GC. You'll see that the number of classes grows then eventually drops, again and again.

If you are interested in fixing this in Java 7 you may try reporting a bug, but I'm not sure that Oracle will show much passion in fixing something in Java 7 given that there is no problem with Java 8. I guess they'll simply suggest to use Java 8 instead.

[projectvalhalla]

Yes, I confirm that things work with Java 8, except for the dramatical over-commitment of virtual memory, which I figured can effectively be controlled by the "-Xmx" VM argument. So there is not an issue with YourKit.

One thing though: Having run the test app for about ten minutes on Windows 7, 64-bit with Oracle JDK8_60, YourKit suddenly reported an OutOfMemoryError and dumped an .hprof file in the execution path of the Java process.
The OutOfMemoryError was not raised by an operation the profiled app did, because it did not crash, but by something the YourKit profiler or agent did.
That file says there are about 6MB of int[] instances not reachable but still not reclaimed by GC.

Strangely though, the JVM continued to work as expected: It loaded and unloaded classes.
So, I don't exactly know what to think of it.
But one can say that creating millions of classes is not really a likely use-case.

As with Java 7, like you said, there seems to be an issue with the reporting of number of unloaded classes. Unfortunately, running Java 8 is not an option for some users that still rely on Windows XP and Java 8 is not available for Windows XP.
But that is another story not related to YourKit.

So the bottom line: There is no issue with YourKit apart from the strange OutOfMemoryError being reported by it.

[projectvalhalla]

Having run the test app for about ten minutes on Windows 7, 64-bit with Oracle JDK8_60,...

Sorry, I meant JDK8_11 here.

[Anton Katilin]

Could you please zip that hprof file, upload to DropBox or a similar service, and post the download link.
Additionally, please include into that zip the profiler log files from <user home>\.yjp\log.
We'll try to find out the reason of that OOME.

Just curious, are you really dependent on using Windows XP? Or is it Windows Server 2003 which you still keep supported?

[projectvalhalla]

I prepared a .zip file with the .hprof file and YourKit log. You can directly download it under:

https://www.dropbox.com/s/yz3czyutjj22m ... g.zip?dl=1

That OOME also does not seem to happen everytime. I just tried twice to reproduce it, but couldn't.

As with Windows XP, I happen to have a very outdated Windows XP, SP3, 32-bit installed on a desktop computer at the current customer site I work at and wanted to test and profile my app there. It is not Windows (Server) 2003. That was the reason why.

Thanks for investigating!

[Anton Katilin]

Thank you for providing the files.

The OOME is not visible in the Threads view which shows the stacks at the moment of snapshot capture. There is no sign of it in the logs either.

We'll repeat your test to reproduce the exception.

[projectvalhalla]

The OOME is not visible in the Threads view...

Yeah, that puzzled me, too. But I could see a small tooltip popping up at the right lower corner of the YourKit client application which said, that a snapshot was being created due to an OOME.
I also never created a snapshot on my own, manually.

Still, thanks for investigating.

Maybe this could help: When I let the test program run under JDK7 (update 65 or 60) without a profiler attached, the committed virtual memory does not grow.

But when I then attach YourKit profiler to it, the amount of used memory as well as committed memory increases steadily.

The increase in memory is not reported by any view in YourKit. I can only see it reported by the operating system in both "actual RAM usage" as well as "committed virtual memory".

YourKit profiler says "Allocated All Pools = 15MB" when in fact the whole JVM process occupies 870MB when about 8 million classes are loaded, and still counting.

I think I eventually will reach an OOME there, too.

[projectvalhalla]

I think it must have something to do with memory allocated by the native agent library for each new class being defined, and not being freed.

[Anton Katilin]

Indeed, the class loading probe stores for each loaded class its name in native memory, and releases it when the class is unloaded.

As we've earlier found, Java 7 does not issue events when such generated classes are unloaded, although actually unloads them. This means that the agent stores their names forever.

As a workaround (and a proof of the theory), you can disable class loading probe by specifying the agent startup option "probe_disable=.ClassLoading". If our assumptions are correct, memory usage shouldn't grow.

[Anton Katilin]

An update: disabling the class loading probe does not help.

Java 7 does not issue class loading events for the generated classes too.
We keep on investigating.

[Anton Katilin]

The newest theory is:

The problem is in the Java 7 VM, but is triggered only when agent performs bytecode instrumentation of loaded classes. The problem goes away (i.e. the memory does not grow) if the instrumentation is turned off by specifying agent startup options "disablealloc,disabletracing". Turning off both allocation recording and CPU tracing ensures that the agent returns unmodified bytecode for the generated classes.