Problems when running Yourkit on Spark AWS Emr

Questions about YourKit Java Profiler
Post Reply
cornelc
Posts: 4
Joined: Tue Apr 04, 2023 3:10 pm

Problems when running Yourkit on Spark AWS Emr

Post by cornelc »

Hi, I'm trying to profile a Spark application running in AWS Emr using yourkit. The agent seems to start on both spark driver/spark executors (its displaying the message [YourKit Java Profiler 2022.9-b182] Log file: /tmp/ApplicationMaster-4112.log) but nothing is written in that log file (it is not even created), also none of the ports 10001-10010 will be used by the profiler.

AWS EMR version - 6.10.0

Java version:

openjdk version "17.0.6" 2023-01-17 LTS
OpenJDK Runtime Environment Corretto-17.0.6.10.1 (build 17.0.6+10-LTS)
OpenJDK 64-Bit Server VM Corretto-17.0.6.10.1 (build 17.0.6+10-LTS, mixed mode, sharing)

Spark submit command (in case it's needed):
[*]
YOURKIT_PATH="${OUTPUT_BUCKET}/yourkit/libyjpagent.so"
YOURKIT_AGENT="libyjpagent.so=sampling,logdir=/tmp"

spark-submit --master yarn --deploy-mode cluster \
--files "${YOURKIT_PATH}" \
--conf "spark.driver.extraJavaOptions=-XX:+UseG1GC -agentpath:./${YOURKIT_AGENT} -Djava.rmi.server.useCodebaseOnly=true -Dcom.sun.jndi.ldap.object.trustURLCodebase=false -Dcom.sun.jndi.cosnaming.object.trustURLCodebase=false -Dcom.sun.jndi.rmi.object.trustURLCodebase=false" \
--conf "spark.executor.extraJavaOptions=-XX:+UseG1GC -agentpath:./${YOURKIT_AGENT} -Djava.rmi.server.useCodebaseOnly=true -Dcom.sun.jndi.ldap.object.trustURLCodebase=false -Dcom.sun.jndi.cosnaming.object.trustURLCodebase=false -Dcom.sun.jndi.rmi.object.trustURLCodebase=false" \
--conf...

Any help is appreciated

Thanks,
Cornel
Anton Katilin
Posts: 6172
Joined: Wed Aug 11, 2004 8:37 am

Re: Problems when running Yourkit on Spark AWS Emr

Post by Anton Katilin »

Hello Cornel,

Instead of opening agent ports on the ASW side which can be a challenge, please use YourKit connection broker approach:
https://www.yourkit.com/broker/features/

The broker is specially intended to simplify profiling setup in scenarios such as yours.

Best regards,
Anton
cornelc
Posts: 4
Joined: Tue Apr 04, 2023 3:10 pm

Re: Problems when running Yourkit on Spark AWS Emr

Post by cornelc »

Hello Anton,

My clusters can't open external connections for security reasons, the broker approach does not work for us.

I fixed the issue, it was something trivial - the cluster manager decided to launch the Spark driver on a core node instead of the master node so basically I was looking on the wrong machine (the master node) and wondering why the ports 1000x are unused.

Tkx,
Cornel.
Post Reply