Yourkit causes jvm to crash on Ubuntu 12.10 64 bit

Questions about YourKit Java Profiler
Post Reply
cdmihai
Posts: 3
Joined: Thu Oct 25, 2012 2:05 am

Yourkit causes jvm to crash on Ubuntu 12.10 64 bit

Post by cdmihai »

Hello,

I've downloaded yjp 11.0.9 for Ubuntu (architecture not specified on download page, presumably its both 32 and 64) and it crashes the JVM whenever I want to attach to my application.

- The JVM crash output:

Code: Select all

2012-10-24 21:07:33 Full thread dump Java HotSpot(TM) 64-Bit Server VM (23.5-b02 mixed mode): "Service Thread" daemon prio=10 tid=0x00007f23240f8800 nid=0x6901 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE "C2 CompilerThread1" daemon prio=10 tid=0x00007f23240f6000 nid=0x6900 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "C2 CompilerThread0" daemon prio=10 tid=0x00007f23240f3000 nid=0x68ff waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "Signal Dispatcher" daemon prio=10 tid=0x00007f23240f1000 nid=0x68fe waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "Finalizer" daemon prio=10 tid=0x00007f23240a4000 nid=0x68fd in Object.wait() [0x00007f231a103000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000007c1d65798> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135) - locked <0x00000007c1d65798> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177) "Reference Handler" daemon prio=10 tid=0x00007f23240a1800 nid=0x68fc in Object.wait() [0x00007f231a204000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000007c1d65320> (a java.lang.ref.Reference$Lock) at java.lang.Object.wait(Object.java:503) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133) - locked <0x00000007c1d65320> (a java.lang.ref.Reference$Lock) "main" prio=10 tid=0x00007f2324008800 nid=0x68f2 runnable [0x00007f232c9c1000] java.lang.Thread.State: RUNNABLE at codese.CoDeSeDriver.delay(Unknown Source) at codese.CoDeSeDriver.main(Unknown Source) "VM Thread" prio=10 tid=0x00007f232409a000 nid=0x68fb runnable "GC task thread#0 (ParallelGC)" prio=10 tid=0x00007f2324016800 nid=0x68f3 runnable "GC task thread#1 (ParallelGC)" prio=10 tid=0x00007f2324018800 nid=0x68f4 runnable "GC task thread#2 (ParallelGC)" prio=10 tid=0x00007f232401a000 nid=0x68f5 runnable "GC task thread#3 (ParallelGC)" prio=10 tid=0x00007f232401c000 nid=0x68f6 runnable "GC task thread#4 (ParallelGC)" prio=10 tid=0x00007f232401e000 nid=0x68f7 runnable "GC task thread#5 (ParallelGC)" prio=10 tid=0x00007f232401f800 nid=0x68f8 runnable "GC task thread#6 (ParallelGC)" prio=10 tid=0x00007f2324021800 nid=0x68f9 runnable "GC task thread#7 (ParallelGC)" prio=10 tid=0x00007f2324023800 nid=0x68fa runnable "VM Periodic Task Thread" prio=10 tid=0x00007f2324103000 nid=0x6902 waiting on condition JNI global references: 111 Heap PSYoungGen total 55680K, used 954K [0x00000007c1d60000, 0x00000007c5b80000, 0x0000000800000000) eden space 47744K, 2% used [0x00000007c1d60000,0x00000007c1e4ebf0,0x00000007c4c00000) from space 7936K, 0% used [0x00000007c53c0000,0x00000007c53c0000,0x00000007c5b80000) to space 7936K, 0% used [0x00000007c4c00000,0x00000007c4c00000,0x00000007c53c0000) ParOldGen total 127296K, used 0K [0x0000000745800000, 0x000000074d450000, 0x00000007c1d60000) object space 127296K, 0% used [0x0000000745800000,0x0000000745800000,0x000000074d450000) PSPermGen total 21248K, used 2422K [0x0000000740600000, 0x0000000741ac0000, 0x0000000745800000) object space 21248K, 11% used [0x0000000740600000,0x000000074085da10,0x0000000741ac0000) 2012-10-24 21:07:38 Full thread dump Java HotSpot(TM) 64-Bit Server VM (23.5-b02 mixed mode): "Service Thread" daemon prio=10 tid=0x00007f23240f8800 nid=0x6901 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE "C2 CompilerThread1" daemon prio=10 tid=0x00007f23240f6000 nid=0x6900 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "C2 CompilerThread0" daemon prio=10 tid=0x00007f23240f3000 nid=0x68ff waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "Signal Dispatcher" daemon prio=10 tid=0x00007f23240f1000 nid=0x68fe waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "Finalizer" daemon prio=10 tid=0x00007f23240a4000 nid=0x68fd in Object.wait() [0x00007f231a103000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000007c1d65798> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135) - locked <0x00000007c1d65798> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177) "Reference Handler" daemon prio=10 tid=0x00007f23240a1800 nid=0x68fc in Object.wait() [0x00007f231a204000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000007c1d65320> (a java.lang.ref.Reference$Lock) at java.lang.Object.wait(Object.java:503) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133) - locked <0x00000007c1d65320> (a java.lang.ref.Reference$Lock) "main" prio=10 tid=0x00007f2324008800 nid=0x68f2 runnable [0x00007f232c9c1000] java.lang.Thread.State: RUNNABLE at codese.CoDeSeDriver.delay(Unknown Source) at codese.CoDeSeDriver.main(Unknown Source) "VM Thread" prio=10 tid=0x00007f232409a000 nid=0x68fb runnable "GC task thread#0 (ParallelGC)" prio=10 tid=0x00007f2324016800 nid=0x68f3 runnable "GC task thread#1 (ParallelGC)" prio=10 tid=0x00007f2324018800 nid=0x68f4 runnable "GC task thread#2 (ParallelGC)" prio=10 tid=0x00007f232401a000 nid=0x68f5 runnable "GC task thread#3 (ParallelGC)" prio=10 tid=0x00007f232401c000 nid=0x68f6 runnable "GC task thread#4 (ParallelGC)" prio=10 tid=0x00007f232401e000 nid=0x68f7 runnable "GC task thread#5 (ParallelGC)" prio=10 tid=0x00007f232401f800 nid=0x68f8 runnable "GC task thread#6 (ParallelGC)" prio=10 tid=0x00007f2324021800 nid=0x68f9 runnable "GC task thread#7 (ParallelGC)" prio=10 tid=0x00007f2324023800 nid=0x68fa runnable "VM Periodic Task Thread" prio=10 tid=0x00007f2324103000 nid=0x6902 waiting on condition JNI global references: 111 Heap PSYoungGen total 55680K, used 954K [0x00000007c1d60000, 0x00000007c5b80000, 0x0000000800000000) eden space 47744K, 2% used [0x00000007c1d60000,0x00000007c1e4ebf0,0x00000007c4c00000) from space 7936K, 0% used [0x00000007c53c0000,0x00000007c53c0000,0x00000007c5b80000) to space 7936K, 0% used [0x00000007c4c00000,0x00000007c4c00000,0x00000007c53c0000) ParOldGen total 127296K, used 0K [0x0000000745800000, 0x000000074d450000, 0x00000007c1d60000) object space 127296K, 0% used [0x0000000745800000,0x0000000745800000,0x000000074d450000) PSPermGen total 21248K, used 2422K [0x0000000740600000, 0x0000000741ac0000, 0x0000000745800000) object space 21248K, 11% used [0x0000000740600000,0x000000074085da10,0x0000000741ac0000) sleep over CoDeSe_irbt_size10_threads1_iter1=20
However, if I manually enable the profiler, it seems that the JVM crashes with this message:

Code: Select all

 A fatal error has been detected by the Java Runtime Environment: # # SIGSEGV (0xb) at pc=0x00007fd18cad2b3a, pid=29227, tid=140537786201856 # # JRE version: 7.0_09-b05 # Java VM: Java HotSpot(TM) 64-Bit Server VM (23.5-b02 mixed mode linux-amd64 compressed oops) # Problematic frame: # V [libjvm.so+0x63bb3a] IterateOverHeapObjectClosure::do_object(oopDesc*)+0x6a
It appears that the crash occurs when using the sun.misc.Unsafe class (as far as I can tell) to allocate objects into memory. I am fiddling with a deserializer.
The code works fine without yourkit attached.

Has anybody run into this problem as well? Are there situations where yourkit cannot profile and just crashes the JVM? How does it fare with JNI calls (sun.misc.Unsafe is made out of JNI calls)?

Thanks
Anton Katilin
Posts: 6172
Joined: Wed Aug 11, 2004 8:37 am

Re: Yourkit causes jvm to crash on Ubuntu 12.10 64 bit

Post by Anton Katilin »

Hi,
I want to attach to my application.
Do you mean you attach the agent to a running process started without the profiler?
http://www.yourkit.com/docs/11/help/attach_agent.jsp

Could you please provide:

1) output of "uname -a"
2) the profiler agent log file <user home>/.yjp/log/<PID>.log
3) <full path to jdk used to run the crashed application>/bin/java -version
if I manually enable the profiler
What is the exact -agentpath parameter?
Are there situations where yourkit cannot profile and just crashes the JVM?
It shouldn't crash unless there is a lack of memory (mostly a 32-bit specific issue, solved by proper configuration) or a yet unknown bug. I need the detail mentioned above to tell the particular reason of your crash.
How does it fare with JNI calls (sun.misc.Unsafe is made out of JNI calls)?
Native methods are OK.

Best regards,
Anton
Anton Katilin
Posts: 6172
Joined: Wed Aug 11, 2004 8:37 am

Re: Yourkit causes jvm to crash on Ubuntu 12.10 64 bit

Post by Anton Katilin »

P.S.

Could you please also check whether the problem is reproducible with version 12 early access:
http://www.yourkit.com/eap/

If it does please send its agent log too.
Anton Katilin
Posts: 6172
Joined: Wed Aug 11, 2004 8:37 am

Re: Yourkit causes jvm to crash on Ubuntu 12.10 64 bit

Post by Anton Katilin »

P.P.S.

Please also provide the full crash log file content hs_err_<PID>.log
cdmihai
Posts: 3
Joined: Thu Oct 25, 2012 2:05 am

Re: Yourkit causes jvm to crash on Ubuntu 12.10 64 bit

Post by cdmihai »

The crash occurs when I start CPU profiling on an application with the yjp attached via command line.

uname -a
Linux ubuntu 3.5.0-17-generic #28-Ubuntu SMP Tue Oct 9 19:31:23 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

yjp and jvm logs:
https://dl.dropbox.com/u/17450714/logs.tar.gz

java version "1.7.0_09"
Java(TM) SE Runtime Environment (build 1.7.0_09-b05)
Java HotSpot(TM) 64-Bit Server VM (build 23.5-b02, mixed mode)

agentpath parameter:
-agentpath: /home/mihai/yjp-11.0.9/bin/linux-x86-64/libyjpagent.so

It crashes with yjp12 eap as well, here are the logs:
https://dl.dropbox.com/u/17450714/logs_yjp12.tar.gz
Anton Katilin
Posts: 6172
Joined: Wed Aug 11, 2004 8:37 am

Re: Yourkit causes jvm to crash on Ubuntu 12.10 64 bit

Post by Anton Katilin »

Thank you for the logs.

The crashed stack trace does not contain any code from the profiler agent. Instead it is all from libjvm.so

There are no errors in the agent logs.

So this looks like a JVM bug.

Is it possible to get your application for testing? We'd try to reproduce the crash and collect additional information.
Anton Katilin
Posts: 6172
Joined: Wed Aug 11, 2004 8:37 am

Re: Yourkit causes jvm to crash on Ubuntu 12.10 64 bit

Post by Anton Katilin »

Could you please try:

- Change JVM GC settings
- Use JDK instead of JRE
- Use another Java version
- Run the same application on another machine
- Specify the profiler agent startup option "disableall": http://www.yourkit.com/docs/11/help/startup_options.jsp
cdmihai
Posts: 3
Joined: Thu Oct 25, 2012 2:05 am

Re: Yourkit causes jvm to crash on Ubuntu 12.10 64 bit

Post by cdmihai »

Hello,

Well, I want to apologize for your spent time on this problem.

As it happens, the problem was in my code. The project I'm using serializes program state via code generation and does so by using the sun.misc.Unsafe class to allocate objects and write their fields.

The yourkit problem appeared because I was deserializing state serialized with a previous jdk. And, apparently, between jdk 1.6 and 1.7 the jvm computes different memory offsets for fields. Thus, on deserialization, on 1.7 it now seg faults with 1.6 serialized state.

As to why it was initially crashing only when I tried to profile it, my assumption is that yourkit at some point traverses the object graph. And since the offsets where wrong, it crashed.

If you're still interested in seeing the crash for testing purposes, here is the code:
https://dl.dropbox.com/u/17450714/codese.tar

You run the exe.sh script. To make it run on your machine you have to edit the yourkit_dir variable in the exe.sh script, at line 3, by making it point to the yourkit installation directory.
Anton Katilin
Posts: 6172
Joined: Wed Aug 11, 2004 8:37 am

Re: Yourkit causes jvm to crash on Ubuntu 12.10 64 bit

Post by Anton Katilin »

Well, the Unsafe class definitely deserves its name :)

Thank you for the update!
Post Reply