Yourkit profiler add to Elastic

Questions about YourKit Java Profiler
Post Reply
Zaka
Posts: 4
Joined: Thu Jan 24, 2019 2:23 pm

Yourkit profiler add to Elastic

Post by Zaka »

Hi all!
Need your help with Yourkit Profiler.
I want to add profiler to elasticsearch service, when i put this string to elasticsearch/jvm.option "-agentpath:/opt/yourkit/bin/linux-x86-64/libyjpagent.so" ES service tarted up normally.
But when you try use Timelion in ES they won`t work, and log next WARN:

Caused by: java.lang.NoClassDefFoundError: com/yourkit/runtime/Callback
at org.apache.lucene.expressions.js.JavascriptCompiler$CompiledExpression.<init>(_value) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_191]
at org.apache.lucene.expressions.js.JavascriptCompiler.compileExpression(JavascriptCompiler.java:199) ~[?:?]
at org.apache.lucene.expressions.js.JavascriptCompiler.compile(JavascriptCompiler.java:148) ~[?:?]
at org.elasticsearch.script.expression.ExpressionScriptEngineService$1.run(ExpressionScriptEngineService.java:106) ~[?:?]
at org.elasticsearch.script.expression.ExpressionScriptEngineService$1.run(ExpressionScriptEngineService.java:85) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_191]
at org.elasticsearch.script.expression.ExpressionScriptEngineService.compile(ExpressionScriptEngineService.java:85) ~[?:?]
at org.elasticsearch.script.ScriptService.compile(ScriptService.java:307) ~[elasticsearch-5.6.14.jar:5.6.14]
at org.elasticsearch.search.aggregations.pipeline.bucketscript.BucketScriptPipelineAggregator.reduce(BucketScriptPipelineAggregator.java:94) ~[elasticsearch-5.6.14.jar:5.6.14]
at org.elasticsearch.search.aggregations.InternalAggregation.reduce(InternalAggregation.java:122) ~[elasticsearch-5.6.14.jar:5.6.14]
at org.elasticsearch.search.aggregations.InternalAggregations.reduce(InternalAggregations.java:77) ~[elasticsearch-5.6.14.jar:5.6.14]
at org.elasticsearch.search.aggregations.bucket.filters.InternalFilters$InternalBucket.reduce(InternalFilters.java:100) ~[elasticsearch-5.6.14.jar:5.6.14]
at org.elasticsearch.search.aggregations.bucket.filters.InternalFilters.doReduce(InternalFilters.java:205) ~[elasticsearch-5.6.14.jar:5.6.14]
at org.elasticsearch.search.aggregations.InternalAggregation.reduce(InternalAggregation.java:119) ~[elasticsearch-5.6.14.jar:5.6.14]
at org.elasticsearch.search.aggregations.InternalAggregations.reduce(InternalAggregations.java:77) ~[elasticsearch-5.6.14.jar:5.6.14]
at org.elasticsearch.action.search.SearchPhaseController.reduceAggs(SearchPhaseController.java:513) ~[elasticsearch-5.6.14.jar:5.6.14]
at org.elasticsearch.action.search.SearchPhaseController.reducedQueryPhase(SearchPhaseController.java:490) ~[elasticsearch-5.6.14.jar:5.6.14]
at org.elasticsearch.action.search.SearchPhaseController.reducedQueryPhase(SearchPhaseController.java:408) ~[elasticsearch-5.6.14.jar:5.6.14]
at org.elasticsearch.action.search.SearchPhaseController$1.reduce(SearchPhaseController.java:725) ~[elasticsearch-5.6.14.jar:5.6.14]
at org.elasticsearch.action.search.FetchSearchPhase.innerRun(FetchSearchPhase.java:102) ~[elasticsearch-5.6.14.jar:5.6.14]
at org.elasticsearch.action.search.FetchSearchPhase.access$000(FetchSearchPhase.java:45) ~[elasticsearch-5.6.14.jar:5.6.14]
at org.elasticsearch.action.search.FetchSearchPhase$1.doRun(FetchSearchPhase.java:87) ~[elasticsearch-5.6.14.jar:5.6.14]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:675) ~[elasticsearch-5.6.14.jar:5.6.14]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.6.14.jar:5.6.14]
... 3 more
Caused by: java.lang.ClassNotFoundException: com.yourkit.runtime.Callback
at java.lang.ClassLoader.findClass(ClassLoader.java:530) ~[?:1.8.0_191]
at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_191]
at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_191]
at org.apache.lucene.expressions.js.JavascriptCompiler$CompiledExpression.<init>(_value) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_191]
at org.apache.lucene.expressions.js.JavascriptCompiler.compileExpression(JavascriptCompiler.java:199) ~[?:?]
at org.apache.lucene.expressions.js.JavascriptCompiler.compile(JavascriptCompiler.java:148) ~[?:?]
at org.elasticsearch.script.expression.ExpressionScriptEngineService$1.run(ExpressionScriptEngineService.java:106) ~[?:?]
at org.elasticsearch.script.expression.ExpressionScriptEngineService$1.run(ExpressionScriptEngineService.java:85) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_191]
at org.elasticsearch.script.expression.ExpressionScriptEngineService.compile(ExpressionScriptEngineService.java:85) ~[?:?]
at org.elasticsearch.script.ScriptService.compile(ScriptService.java:307) ~[elasticsearch-5.6.14.jar:5.6.14]
at org.elasticsearch.search.aggregations.pipeline.bucketscript.BucketScriptPipelineAggregator.reduce(BucketScriptPipelineAggregator.java:94) ~[elasticsearch-5.6.14.jar:5.6.14]
at org.elasticsearch.search.aggregations.InternalAggregation.reduce(InternalAggregation.java:122) ~[elasticsearch-5.6.14.jar:5.6.14]
at org.elasticsearch.search.aggregations.InternalAggregations.reduce(InternalAggregations.java:77) ~[elasticsearch-5.6.14.jar:5.6.14]
at org.elasticsearch.search.aggregations.bucket.filters.InternalFilters$InternalBucket.reduce(InternalFilters.java:100) ~[elasticsearch-5.6.14.jar:5.6.14]
at org.elasticsearch.search.aggregations.bucket.filters.InternalFilters.doReduce(InternalFilters.java:205) ~[elasticsearch-5.6.14.jar:5.6.14]
at org.elasticsearch.search.aggregations.InternalAggregation.reduce(InternalAggregation.java:119) ~[elasticsearch-5.6.14.jar:5.6.14]
at org.elasticsearch.search.aggregations.InternalAggregations.reduce(InternalAggregations.java:77) ~[elasticsearch-5.6.14.jar:5.6.14]
at org.elasticsearch.action.search.SearchPhaseController.reduceAggs(SearchPhaseController.java:513) ~[elasticsearch-5.6.14.jar:5.6.14]
at org.elasticsearch.action.search.SearchPhaseController.reducedQueryPhase(SearchPhaseController.java:490) ~[elasticsearch-5.6.14.jar:5.6.14]
at org.elasticsearch.action.search.SearchPhaseController.reducedQueryPhase(SearchPhaseController.java:408) ~[elasticsearch-5.6.14.jar:5.6.14]
at org.elasticsearch.action.search.SearchPhaseController$1.reduce(SearchPhaseController.java:725) ~[elasticsearch-5.6.14.jar:5.6.14]
at org.elasticsearch.action.search.FetchSearchPhase.innerRun(FetchSearchPhase.java:102) ~[elasticsearch-5.6.14.jar:5.6.14]
at org.elasticsearch.action.search.FetchSearchPhase.access$000(FetchSearchPhase.java:45) ~[elasticsearch-5.6.14.jar:5.6.14]
at org.elasticsearch.action.search.FetchSearchPhase$1.doRun(FetchSearchPhase.java:87) ~[elasticsearch-5.6.14.jar:5.6.14]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:675) ~[elasticsearch-5.6.14.jar:5.6.14]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.6.14.jar:5.6.14]
... 3 more
Zaka
Posts: 4
Joined: Thu Jan 24, 2019 2:23 pm

Re: Yourkit profiler add to Elastic

Post by Zaka »

I found this solution (-Dorg.osgi.framework.bootdelegation=com.yourkit.*) but it did not help me.
Anton Katilin
Posts: 6172
Joined: Wed Aug 11, 2004 8:37 am

Re: Yourkit profiler add to Elastic

Post by Anton Katilin »

What profiler version and build do you use?

What is the output of corresponding "java -version"?

What is the output of "uname -a"?

What is the content of <user home>/.yjp/log/<session name>-<pid>.log?
Zaka
Posts: 4
Joined: Thu Jan 24, 2019 2:23 pm

Re: Yourkit profiler add to Elastic

Post by Zaka »

Anton Katilin wrote:What profiler version and build do you use?

What is the output of corresponding "java -version"?

What is the output of "uname -a"?

What is the content of <user home>/.yjp/log/<session name>-<pid>.log?
1. 2018.04-b88 and 2017.02-b63
2. openjdk version "1.8.0_191"
OpenJDK Runtime Environment (build 1.8.0_191-b12)
OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode)
3. Linux test1 3.10.0-862.el7.x86_64 #1 SMP Fri Apr 20 16:44:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
4. /bin/java
-Xms2g
-Xmx2g
-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly
-XX:+AlwaysPreTouch
-server
-Xss1m
-Djava.awt.headless=true
-Dfile.encoding=UTF-8
-Djna.nosys=true
-Djdk.io.permissionsUseCanonicalPath=true
-Dio.netty.noUnsafe=true
-Dio.netty.noKeySetOptimization=true
-Dio.netty.recycler.maxCapacityPerThread=0
-Dlog4j.shutdownHookEnabled=false
-Dlog4j2.disable.jmx=true
-Dlog4j.skipJansi=true
-XX:+HeapDumpOnOutOfMemoryError
-agentpath:/opt/yourkit/bin/linux-x86-64/libyjpagent.so
-Dorg.osgi.framework.bootdelegation=com.yourkit.*
-Des.path.home=/usr/share/elasticsearch
-cp
/usr/share/elasticsearch/lib/*
org.elasticsearch.bootstrap.Elasticsearch
-p
/var/run/elasticsearch/elasticsearch.pid
--quiet
-Edefault.path.logs=/var/log/elasticsearch
-Edefault.path.data=/var/lib/elasticsearch
-Edefault.path.conf=/etc/elasticsearch
18.04-88 0.000: Executable: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-1.el7_6.x86_64/jre/bin/java
18.04-88 0.000: uname:
sysname: Linux
release: 3.10.0-862.el7.x86_64
version: #1 SMP Fri Apr 20 16:44:24 UTC 2018
machine: x86_64
18.04-88 0.000: Linux; 64-bit process
18.04-88 0.000: hid: 360229919972Z6V4U86ZY9Y6Z4Y12W59UW 64-bit machine
18.04-88 0.000: Today is 20190124
18.04-88 0.000: full_build_stamp: 2018.04.88 201812311340
18.04-88 0.000: Agent library path: /opt/yourkit/bin/linux-x86-64/libyjpagent.so
18.04-88 0.000: getTimeNsInfo: "monotonic_raw"
18.04-88 0.000: Agent_OnLoad: orig_options=
full_options=
18.04-88 0.000: phase: 1
18.04-88 0.000: jvmtiVersion: 30010203
18.04-88 0.000: vendor: Oracle Corporation
18.04-88 0.000: version: 25.191-b12
18.04-88 0.000: info: mixed mode, sharing
18.04-88 0.000: isJava6: 0
18.04-88 0.001: session_id: 19994621102073 (1548339193,4655)
18.04-88 0.002: line_method: 1
18.04-88 0.013: Reusing existing jar file: /tmp/yjp201812311340.jar
18.04-88 0.013: JVMTI 30010203; 25.191-b12; Oracle Corporation; mixed mode, sharing
18.04-88 0.049: Successfully registered probes: 27
Registered: .AgentLoadingSupport
Registered: .AsyncChannels
Registered: .AwtEvents
Registered: .Cassandra
Registered: .CassandraLW
Registered: .Databases
Registered: .DatabasesLW
Registered: .Files
Registered: .FilesLW
Registered: .HBase
Registered: .HBaseLW
Registered: .JNDI
Registered: .JPA_DataNucleus
Registered: .JPA_EclipseLink
Registered: .JPA_Hibernate
Registered: .JPA_OpenJPA
Registered: .JUnitTests
Registered: .MongoDB
Registered: .MongoDBLW
Registered: .Processes
Registered: .Servlets
Registered: .ServletsLW
Registered: .Sockets
Registered: .SocketsLW
Registered: .Sync
Registered: .TestNG
Registered: .Threads
18.04-88 0.282: agentStart: 1
18.04-88 0.282: agentStart: 2
18.04-88 0.283: agentStart: OK
18.04-88 0.311: tracing=off: large method: 3945 sun.nio.cs.StandardCharsets$Aliases.init(java.lang.Object[])
18.04-88 0.417: vmInit: 1
18.04-88 0.417: vmInit: 2
18.04-88 0.417: agentInit: 1
18.04-88 0.418: agentInit: Core loaded
18.04-88 0.423: setTriggers:

18.04-88 0.480: setTriggers: to unregister: 0
18.04-88 0.480: setTriggers: method listeners to register: 0
18.04-88 0.480: setTriggers: OK
18.04-88 0.480: ourUnsafe: true
18.04-88 0.480: shmem 42341490: OK
18.04-88 0.480: shmem 4234148E: OK
18.04-88 0.501: socket timeout: 30000
18.04-88 0.519: YourKit Java Profiler agent is listening on port 10001, "listen=localhost"
18.04-88 0.540: *** HINT ***: To get profiling results, connect to the application from the profiler UI
18.04-88 0.540: agentInit: Core.init() called
18.04-88 0.560: agentInit: OK
18.04-88 0.560: vmInit: 3
18.04-88 0.560: onStartProfiling: entering...
18.04-88 0.560: onStartProfiling: done
18.04-88 0.560: vmInit: OK
Anton Katilin
Posts: 6172
Joined: Wed Aug 11, 2004 8:37 am

Re: Yourkit profiler add to Elastic

Post by Anton Katilin »

Everything seems OK in the settings and the log.

Could you please try the following agent option(s):

1) -agentpath:/opt/yourkit/bin/linux-x86-64/libyjpagent.so=disableall

2) -agentpath:/opt/yourkit/bin/linux-x86-64/libyjpagent.so=disablealloc,disabletracing

Do they help?
Anton Katilin
Posts: 6172
Joined: Wed Aug 11, 2004 8:37 am

Re: Yourkit profiler add to Elastic

Post by Anton Katilin »

Update: if #2 doesn't work, please also try

3) -agentpath:/opt/yourkit/bin/linux-x86-64/libyjpagent.so=disablealloc,disabletracing,disablenatives

and, if it works,

4) -agentpath:/opt/yourkit/bin/linux-x86-64/libyjpagent.so=disablenatives
Zaka
Posts: 4
Joined: Thu Jan 24, 2019 2:23 pm

Re: Yourkit profiler add to Elastic

Post by Zaka »

Resovle this issue with several option:
1. -agentpath:/opt/yourkit/bin/linux-x86-64/libyjpagent.so=disableall
2. -agentpath:/opt/yourkit/bin/linux-x86-64/libyjpagent.so=disabletracing

With another option have problem with java.class wich i posted above.
I`ll try you option and post info later.
Thank for reply guys :)
Post Reply