Yourkit causing thread blocks

Questions about YourKit Java Profiler
Post Reply
doron.sekler
Posts: 6
Joined: Wed Feb 23, 2011 2:41 pm

Yourkit causing thread blocks

Post by doron.sekler »

Hi,

I am using yourkit 11.0.3 to profile my JVM which is running with 7.0.4 version, the JVM runs the application I am profliing which includes ActiveMQ whenever I attach to the JVM, or connect through JVMTI I see that the app threads become blocked, the CPU consumption drops as result and basically, I can't profile my application since it stops working almost completely

The following is the summary of my App's JVM:

-Djava.endorsed.dirs=C:/K2_BM/oo-win64-24.6.11-SNAPSHOT/bin/../tomcat/endorsed
-Dcatalina.base=C:/K2_BM/oo-win64-24.6.11-SNAPSHOT/bin/../servers/server-0
-Dcatalina.home=C:/K2_BM/oo-win64-24.6.11-SNAPSHOT/bin/../tomcat
-Djava.io.tmpdir=C:/K2_BM/oo-win64-24.6.11-SNAPSHOT/bin/../servers/server-0/temp
-Doo.home=C:/K2_BM/oo-win64-24.6.11-SNAPSHOT/bin/..
-Doo.node=servers/server-0
-Doo.lib=C:/K2_BM/oo-win64-24.6.11-SNAPSHOT/bin/../lib
-XX:+HeapDumpOnOutOfMemoryError
-XX:MaxPermSize=256m
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Djmx.http.port=39900
-Dcom.sun.management.jmxremote.port=39600
-Djmx.invoke.getters=true
-Dlog4j.configuration=file:C:/K2_BM/oo-win64-24.6.11-SNAPSHOT/bin/../servers/server-0/conf/log4j.properties
-Dmgmt.uuid=server-0
-Dspring.profiles.active=nologin
-Djava.security.policy=C:/K2_BM/oo-win64-24.6.11-SNAPSHOT/bin/../tomcat/conf/catalina.policy
-XX:+HeapDumpOnOutOfMemoryError
-Xms512m
-Xmx1024m
-XX:+UseParallelGC
-Djavax.net.ssl.trustStore=
-Djavax.net.ssl.trustStorePassword=
-Djavax.net.ssl.trustStoreType=jks
-Dhttp.port=8080
-Dtomcat.https.port=8443
-Daf.central.url=
-Daf.protocol=
-Xss1024k
-Dlwsso.init.string=
-Dlwsso.trusted.domain=
-Dlwsso.protected.domains=
-Dlwsso.overwrite.xml=
-Didm.request.header=
-Dworker.uuid=307cdefa-3a4b-416d-a00e-3374611e185f
-Dbroker.uuid=2f72d74b-14ef-4b6c-9851-0113b241c2de
-Dbroker.jms.port=6565
-Dbroker.jms.http.port=6566
-Dbroker.jms.https.port=6567
-Dbroker.web.http.port=8080
-Dbroker.web.https.port=8443
-Dmgmt.url=http://localhost:8080/oo
-Djms.prefetch=500
-Dfile.encoding=Cp1252
-Duser.language=en
-Djava.library.path=C:/K2_BM/oo-win64-24.6.11-SNAPSHOT/bin/../lib
-Dwrapper.key=nPQmPD6X1r6O-9lsI0rrmKKu55jSroxt
-Dwrapper.port=32000
-Dwrapper.jvm.port.min=31000
-Dwrapper.jvm.port.max=31999
-Dwrapper.pid=7288
-Dwrapper.version=3.5.14-pro
-Dwrapper.native_library=wrapper
-Dwrapper.cpu.timeout=10
-Dwrapper.jvmid=1
-Dwrapper.lang.domain=wrapper

and the system properties:

ava.specification.vendor=Oracle Corporation
java.specification.version=1.7
java.vendor=Oracle Corporation
java.vendor.url=http://java.oracle.com/
java.vendor.url.bug=http://bugreport.sun.com/bugreport/
java.version=1.7.0_04
java.vm.info=mixed mode
java.vm.name=Java HotSpot(TM) 64-Bit Server VM
java.vm.specification.name=Java Virtual Machine Specification
java.vm.specification.vendor=Oracle Corporation
java.vm.specification.version=1.7
java.vm.vendor=Oracle Corporation
java.vm.version=23.0-b21
javax.net.ssl.trustStore=
javax.net.ssl.trustStorePassword=
javax.net.ssl.trustStoreType=jks
jboss.i18n.generate-proxies=true
jms.prefetch=500
jmx.http.port=39900
jmx.invoke.getters=true
line.separator=\r\n
log4j.configuration=file:C:/K2_BM/oo-win64-24.6.11-SNAPSHOT/bin/../servers/server-0/conf/log4j.properties
lwsso.init.string=
lwsso.overwrite.xml=
lwsso.protected.domains=
lwsso.trusted.domain=
mgmt.url=http://localhost:8080/oo
mgmt.uuid=server-0
oo.home=C:/K2_BM/oo-win64-24.6.11-SNAPSHOT/bin/..
oo.lib=C:/K2_BM/oo-win64-24.6.11-SNAPSHOT/bin/../lib
oo.node=servers/server-0
os.arch=amd64
os.name=Windows Server 2008 R2
os.version=6.1
package.access=sun.,org.apache.catalina.,org.apache.coyote.,org.apache.tomcat.,org.apache.jasper.
package.definition=sun.,java.,org.apache.catalina.,org.apache.coyote.,org.apache.tomcat.,org.apache.jasper.
path.separator=;
server.loader=
shared.loader=
spring.profiles.active=nologin
sun.arch.data.model=64
sun.boot.class.path=C:\K2_BM\oo-win64-24.6.11-SNAPSHOT\java\lib\resources.jar;C:\K2_BM\oo-win64-24.6.11-SNAPSHOT\java\lib\rt.jar;C:\K2_BM\oo-win64-24.6.11-SNAPSHOT\java\lib\sunrsasign.jar;C:\K2_BM\oo-win64-24.6.11-SNAPSHOT\java\lib\jsse.jar;C:\K2_BM\oo-win64-24.6.11-SNAPSHOT\java\lib\jce.jar;C:\K2_BM\oo-win64-24.6.11-SNAPSHOT\java\lib\charsets.jar;C:\K2_BM\oo-win64-24.6.11-SNAPSHOT\java\lib\jfr.jar;C:\K2_BM\oo-win64-24.6.11-SNAPSHOT\java\classes
sun.boot.library.path=C:\K2_BM\oo-win64-24.6.11-SNAPSHOT\java\bin
sun.cpu.endian=little
sun.cpu.isalist=amd64
sun.desktop=windows
sun.io.unicode.encoding=UnicodeLittle
sun.java.command=org.tanukisoftware.wrapper.WrapperStartStopApp org.apache.catalina.startup.Bootstrap 1 start org.apache.catalina.startup.Bootstrap TRUE 1 stop
sun.java.launcher=SUN_STANDARD
sun.jnu.encoding=Cp1252
sun.management.compiler=HotSpot 64-Bit Tiered Compilers
sun.os.patch.level=Service Pack 1
tomcat.https.port=8443
tomcat.util.buf.StringCache.byte.enabled=true
tomcat.util.scan.DefaultJarScanner.jarsToSkip=bootstrap.jar,commons-daemon.jar,tomcat-juli.jar,annotations-api.jar,el-api.jar,jsp-api.jar,servlet-api.jar,catalina.jar,catalina-ant.jar,catalina-ha.jar,catalina-tribes.jar,jasper.jar,jasper-el.jar,ecj-*.jar,tomcat-api.jar,tomcat-util.jar,tomcat-coyote.jar,tomcat-dbcp.jar,tomcat-i18n-en.jar,tomcat-i18n-es.jar,tomcat-i18n-fr.jar,tomcat-i18n-ja.jar,tomcat-juli-adapters.jar,catalina-jmx-remote.jar,catalina-ws.jar,tomcat-jdbc.jar,commons-beanutils*.jar,commons-codec*.jar,commons-collections*.jar,commons-dbcp*.jar,commons-digester*.jar,commons-fileupload*.jar,commons-httpclient*.jar,commons-io*.jar,commons-lang*.jar,commons-logging*.jar,commons-math*.jar,commons-pool*.jar,jstl.jar,geronimo-spec-jaxrpc*.jar,wsdl4j*.jar,ant.jar,ant-junit*.jar,aspectj*.jar,jmx.jar,h2*.jar,hibernate*.jar,httpclient*.jar,jmx-tools.jar,jta*.jar,log4j*.jar,mail*.jar,slf4j*.jar,xercesImpl.jar,xmlParserAPIs.jar,xml-apis.jar,dnsns.jar,ldapsec.jar,localedata.jar,sunjce_provider.jar,sunmscapi.jar,sunpkcs11.jar,jhall.jar,tools.jar,sunec.jar,zipfs.jar,apple_provider.jar,AppleScriptEngine.jar,CoreAudio.jar,dns_sd.jar,j3daudio.jar,j3dcore.jar,j3dutils.jar,jai_core.jar,jai_codec.jar,mlibwrapper_jai.jar,MRJToolkit.jar,vecmath.jar,junit.jar,junit-*.jar,ant-launcher.jar
user.country=US
user.dir=C:\K2_BM\oo-win64-24.6.11-SNAPSHOT\bin
user.home=C:\Users\cnc
user.language=en
user.name=cnc
user.script=
user.timezone=Australia/Sydney
user.variant=
worker.uuid=307cdefa-3a4b-416d-a00e-3374611e185f
wrapper.cpu.timeout=10
wrapper.java.pid=12148
wrapper.jvm.port.max=31999
wrapper.jvm.port.min=31000
wrapper.jvmid=1
wrapper.key=nPQmPD6X1r6O-9lsI0rrmKKu55jSroxt
wrapper.lang.domain=wrapper
wrapper.native_library=wrapper
wrapper.pid=7288
wrapper.port=32000
wrapper.version=3.5.14-pro


Thanks in advance for the support

Doron
Anton Katilin
Posts: 6172
Joined: Wed Aug 11, 2004 8:37 am

Re: Yourkit causing thread blocks

Post by Anton Katilin »

Hello Doron
I am using yourkit 11.0.3
Please use the latest build instead (11.0.8 at the moment):
http://www.yourkit.com/download

I cannot see which profiler agent options you have specified.

Did you use attaching the agent to a running JVM?
http://www.yourkit.com/docs/11/help/attach_agent.jsp
If yes, please try starting it with the agent instead:
http://www.yourkit.com/docs/11/help/run ... ofiler.jsp

If you already start the JVM with the profiler agent, which agent startup options have you specified?
http://www.yourkit.com/docs/11/help/startup_options.jsp

Best regards,
Anton
Anton Katilin
Posts: 6172
Joined: Wed Aug 11, 2004 8:37 am

Re: Yourkit causing thread blocks

Post by Anton Katilin »

Also, which profiling modes do you use: CPU sampling/tracing, allocation recording etc.?

Is it a local profiling (the profiled application and the profiler UI run on the same machine) or a remote one (the profiled application and the profiler UI run on different machines)?
doron.sekler
Posts: 6
Joined: Wed Feb 23, 2011 2:41 pm

Re: Yourkit causing thread blocks

Post by doron.sekler »

Thanks for the quick reply, I will try with 11.0.8 as well

Regarding your questions:

I tried both with attaching and with the agent (no parameters other than port)

sampling or tracing is irrelevant since this happens as soon as I connect

When I work with disableall option the problem seems to go away, still it disables some functionality I work with (telemetry, etc')
Anton Katilin
Posts: 6172
Joined: Wed Aug 11, 2004 8:37 am

Re: Yourkit causing thread blocks

Post by Anton Katilin »

"disableall" is equal to the following set of individual options: disablealloc, disabletracing, disablej2ee, disableexceptiontelemetry, disablestacktelemetry, builtinprobes=none
Please see what which of them means here:
http://www.yourkit.com/docs/11/help/startup_options.jsp

You can try them one by one to see which one makes the biggest improvement.
My guess it can be disablestacktelemetry or builtinprobes=none
andrejackbia
Posts: 3
Joined: Tue Jan 21, 2020 10:30 am

Re: Yourkit causing thread blocks

Post by andrejackbia »

Hello,

I'm experiencing the same behavior using 2019.8 b137 and java 11:

java version "11.0.5" 2019-10-15 LTS
Java(TM) SE Runtime Environment 18.9 (build 11.0.5+10-LTS)
Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.5+10-LTS, mixed mode))

I'm trying to profile a public available web application developed with spring boot (https://github.com/spring-projects/spring-petclinic) during the execution of a load test suite written by me.

What happens is that when enable monitor profiling the application slows down and I can see it also from the dashboard of the load testing platform because the rate of requests per second drops dramatically.

I link a screenshot to show the performance bottleneck. Can anybody help me understanding the reason of such behavior?

https://drive.google.com/file/d/12llGhG ... sp=sharing

Thank you!
Anton Katilin
Posts: 6172
Joined: Wed Aug 11, 2004 8:37 am

Re: Yourkit causing thread blocks

Post by Anton Katilin »

Hi,

Monitor profiling can add overhead, it's normal.

Do you use other profiling modes in parallel e.g. sampling? If you do please try to perform monitor profiling alone.

Please try to also try to stop stack telemetry during monitor profiling . See "Start/stop stack telemetry":
https://www.yourkit.com/docs/java/help/threads.jsp

Alternatively, instead of monitor profiling, you can use stack telemetry samples to analyze thread blocking/waiting states.

Best regards,
Anton
Post Reply