cannot capture snapshot when jvm exit automatically

Questions about YourKit Java Profiler
Post Reply
Jax
Posts: 5
Joined: Tue Mar 12, 2019 7:22 am

cannot capture snapshot when jvm exit automatically

Post by Jax »

hi everyone

I'm a newer of Yourkit java profiler. I try to use yourkit to profile Apach Spark jobs.
I run Spark on two nodes and deploy by standalone mode.
When I submit my spark job, I add java options like

Code: Select all

--conf spark.executor.extraJavaOptions="-agentpath:/root/Downloads/YourKit-JavaProfiler-2019.1/bin/linux-x86-64/libyjpagent.so=pro
be_disable=*,exceptions=disable,sampling,monitors,listen=localhost"
.

I can make sure those agents has been enable by remoting Java Profile GUI.
I also can remote attach them and capture snapshot after executing manually using Java Profile GUI.
But remote attach has some problem, it will lose some data sometime. (the total cpu time in the snapshot will be very low and it's immpossible. and it also lose some methods called.)
So, I want to captured snapshot on local when jvm exit automatically. I try to set startup option "onexit=snapshot" or set "triggers.txt" like

Code: Select all

JVM Exit -1
        CapturePerformanceSnapshot {sessionname}-{date}
or

Code: Select all

JVM Start maxTriggerCount=1
        StartCpuSampling
        StartMonitorProfiling
JVM Exit maxTriggerCount=1
        CapturePerformanceSnapshot {sessionname}-{date}
        StopCpuSampling
        StopMonitorProfiling
or

Code: Select all

Timer delay=5m maxTriggerCount=-1
        CapturePerformanceSnapshot {sessionname}-timer-{date}
I can't find any snapshots in {user-home}/Snapshots after running spark job. I only got the snapshot when I using "Ctrl+c" to shutdown my spark job and the snapshot will be captured.

so, my question is how can I capture snapshots automatically? or something wrong in my config?
In my experience, when I used java record flighter, I can got the jfr file after running spark job.

My OS is CentOS 7.4.
Java version is oracle-jdk 1.8.0_162-b12.
Scala version is 2.11.8.
Apache spark version is 2.3.1.
the log file is below as
https://drive.google.com/file/d/13rvxyQ ... sp=sharing

Best Regards
Jax
Anton Katilin
Posts: 6172
Joined: Wed Aug 11, 2004 8:37 am

Re: cannot capture snapshot when jvm exit automatically

Post by Anton Katilin »

Hello Jax

The simplest way to capture a snapshot on exit is indeed to use the startup options "onexit=memory" for a memory snapshot or "onexit=snapshot" for a performance snapshot:
https://www.yourkit.com/docs/java/help/ ... ptions.jsp

The triggers do the same, but it's more complicated approach for such simple case.

Your agent log file is located in /root/.yjp, so the on-exit snapshots should be expected in /root/Snapshots. You can customize the snapshot directory as well, e.g with the startup option "dir=<custom dir>": https://www.yourkit.com/docs/java/help/snapshot_dir.jsp

Best regards,
Anton
Jax
Posts: 5
Joined: Tue Mar 12, 2019 7:22 am

Re: cannot capture snapshot when jvm exit automatically

Post by Jax »

Hi Anton

Thanks for your reply.
Acutally, I had tried to set startup option "onexit=snapshot" but I can't find the snashopts in /root/Snapshots when jvm exit. My startop optinos is

Code: Select all

--conf spark.executor.extraJavaOptions="-agentpath:/root/Downloads/YourKit-JavaProfiler-2019.1/bin/linux-x86-64/libyjpagent.so=pro
be_disable=*,exceptions=disable,onexit=snapshot,sampling,monitors,listen=localhost"
.

I can find a green point in the Java Profiler GUI on my laptop using ssh to remotely detect them.
I think it means this agent is enable and sampling and monitors is also enable.

Do I need to enter license key on my server?
I only entered lincense key on my laptop when I opened the java profiler gui.
How can I do it? I saw this post https://www.yourkit.com/docs/java/help/ ... filing.jsp and didn't find the way to enter license key in console.

Thanks
Jax
Anton Katilin
Posts: 6172
Joined: Wed Aug 11, 2004 8:37 am

Re: cannot capture snapshot when jvm exit automatically

Post by Anton Katilin »

Hello Jax,
Acutally, I had tried to set startup option "onexit=snapshot" but I can't find the snashopts in /root/Snapshots when jvm exit. My startop optinos is
Could you please provide corresponding agent log file. The log file you previously provided does not contain these options.
I can find a green point in the Java Profiler GUI on my laptop using ssh to remotely detect them.
I think it means this agent is enable and sampling and monitors is also enable.
Can you connect to the remote application from the Welcome screen?
Do I need to enter license key on my server?
No. The key is needed for the UI only.

Best regards,
Anton
Jax
Posts: 5
Joined: Tue Mar 12, 2019 7:22 am

Re: cannot capture snapshot when jvm exit automatically

Post by Jax »

Hi Anton
Could you please provide corresponding agent log file. The log file you previously provided does not contain these options.
Here's the log files link:
https://drive.google.com/drive/folders/ ... sp=sharing
They are from different nodes but the same spark job.
Can you connect to the remote application from the Welcome screen?
Yes, I can.

Thanks
Jax
Anton Katilin
Posts: 6172
Joined: Wed Aug 11, 2004 8:37 am

Re: cannot capture snapshot when jvm exit automatically

Post by Anton Katilin »

Hello Jax

According to the log, the on-exit snapshot has been captured, e.g.:

Code: Select all

19.1-112 2829.395: Snapshot captured: /root/Snapshots/CoarseGrainedExecutorBackend-2019-03-14-shutdown-2.snapshot
Best regards,
Anton
Jax
Posts: 5
Joined: Tue Mar 12, 2019 7:22 am

Re: cannot capture snapshot when jvm exit automatically

Post by Jax »

Hi Anton

oh.. I see. Thanks.
I really missed this log and I has found snapshot in this node.

However, why did it only capture snapshot in one node?
I set java option in config of spark executor. Spark will deploy executors on my two nodes.

In log "CoarseGrainedExecutorBackend-25695.log", we can see
19.1-112 2826.566: vmDeath: 1
19.1-112 2826.566: vmDeath: 2
19.1-112 2826.569: vmDeath: 3
19.1-112 2829.199: [m: 78521 7043] WSN: starting /root/Snapshots/CoarseGrainedExecutorBackend-2019-03-14-shutdown-2.snapshot
19.1-112 2829.370: [m: 78521 7044] WSN: tables written
19.1-112 2829.370: [m: 78521 7044] WSN: heap segment closed
19.1-112 2829.370: [m: 78521 7044] WSN: map done
19.1-112 2829.393: SP: gc #0 5 ms=23 sz=44077 p=44 t=8
19.1-112 2829.393: [m: 78521 7044] WSN: done
19.1-112 2829.395: Snapshot captured: /root/Snapshots/CoarseGrainedExecutorBackend-2019-03-14-shutdown-2.snapshot
19.1-112 2829.396: vmDeath: 4
19.1-112 2829.396: vmDeath: OK
but in another one "CoarseGrainedExecutorBackend-36981.log"
org.apache.spark.storage.BlockManager$RemoteBlockTempFileManager$$anon$1
19.1-112 2826.530: SP: pin java.util.logging.LogManager$Cleaner
19.1-112 2826.531: SP: pin org.apache.hadoop.util.ShutdownHookManager$1
19.1-112 2826.551: vmDeath: 1
19.1-112 2826.551: vmDeath: 2
19.1-112 2826.552: vmDeath: 3
We can't find something related.

I can connect both of them on remote Welcome screen.
In essence, java profiler will capture both of their snapshot when their jvm process exit, right?

Thanks
Jax
Anton Katilin
Posts: 6172
Joined: Wed Aug 11, 2004 8:37 am

Re: cannot capture snapshot when jvm exit automatically

Post by Anton Katilin »

Hello Jax,

Code: Select all

but in another one "CoarseGrainedExecutorBackend-36981.log"
We can't find something related.
Perhaps the startup options have not been applied. Please check the beginning of the log file, or provide the log in full to let us check it.
I can connect both of them on remote Welcome screen.
You can also check what options are applied in the "Summary" tab, "Command line", argument "-agentpath:<...yjpagent...>".
In essence, java profiler will capture both of their snapshot when their jvm process exit, right?
Options are applied and profiling is performed on per Java process basis. So yes, if both processes have the option specified, the on-exit snapshot will be captured for both.

Best regards,
Anton
Jax
Posts: 5
Joined: Tue Mar 12, 2019 7:22 am

Re: cannot capture snapshot when jvm exit automatically

Post by Jax »

Hi Anton

Here's the log
https://drive.google.com/drive/folders/ ... sp=sharing
This job didn't capture any snapshot.
You can also check what options are applied in the "Summary" tab, "Command line", argument "-agentpath:<...yjpagent...>".
I also checked the "Command line" in "Summary" tab.

Code: Select all

-agentpath:/root/Downloads/YourKit-JavaProfiler-2019.1/bin/linux-x86-64/libyjpagent.so=probe_disable=*,exceptions=disable,onexit=snapshot,sampling,monitors,listen=localhost
"onexit" option was applied.

Thanks
Jax
Anton Katilin
Posts: 6172
Joined: Wed Aug 11, 2004 8:37 am

Re: cannot capture snapshot when jvm exit automatically

Post by Anton Katilin »

Hello Jax

These logs seem truncated: please note that they end with message "vmDeath: 3", while the log of the process which correctly captured the snapshot contains more records after "vmDeath: 3", including "Snapshot captured: xxx.snapshot" followed by "vmDeath: 4" and "vmDeath: OK".

Looks like the process terminated unexpectedly near the point where the on-exit snapshot should have been captured.

Could you please check the file system for the JVM crash log file hs_err<PID>.log where <PID> corresponds to the problematic process.

Please note that if a snapshot capture begins, some intermediate log messages should appear; please see the "good" log for example . Also, possible I/O errors should have normally been detected and logged. But we see nothing of that in the problematic logs.

Best regards,
Anton
Post Reply