I'm a newer of Yourkit java profiler. I try to use yourkit to profile Apach Spark jobs.
I run Spark on two nodes and deploy by standalone mode.
When I submit my spark job, I add java options like
Code: Select all
--conf spark.executor.extraJavaOptions="-agentpath:/root/Downloads/YourKit-JavaProfiler-2019.1/bin/linux-x86-64/libyjpagent.so=pro
be_disable=*,exceptions=disable,sampling,monitors,listen=localhost"
I can make sure those agents has been enable by remoting Java Profile GUI.
I also can remote attach them and capture snapshot after executing manually using Java Profile GUI.
But remote attach has some problem, it will lose some data sometime. (the total cpu time in the snapshot will be very low and it's immpossible. and it also lose some methods called.)
So, I want to captured snapshot on local when jvm exit automatically. I try to set startup option "onexit=snapshot" or set "triggers.txt" like
Code: Select all
JVM Exit -1
CapturePerformanceSnapshot {sessionname}-{date}
Code: Select all
JVM Start maxTriggerCount=1
StartCpuSampling
StartMonitorProfiling
JVM Exit maxTriggerCount=1
CapturePerformanceSnapshot {sessionname}-{date}
StopCpuSampling
StopMonitorProfiling
Code: Select all
Timer delay=5m maxTriggerCount=-1
CapturePerformanceSnapshot {sessionname}-timer-{date}
so, my question is how can I capture snapshots automatically? or something wrong in my config?
In my experience, when I used java record flighter, I can got the jfr file after running spark job.
My OS is CentOS 7.4.
Java version is oracle-jdk 1.8.0_162-b12.
Scala version is 2.11.8.
Apache spark version is 2.3.1.
the log file is below as
https://drive.google.com/file/d/13rvxyQ ... sp=sharing
Best Regards
Jax