Yourkit Remote Profiling - Not Attaching

Questions about YourKit Java Profiler
Post Reply
lalit.r.nagpal
Posts: 13
Joined: Wed Jul 09, 2014 2:07 pm

Yourkit Remote Profiling - Not Attaching

Post by lalit.r.nagpal »

Hi,

We have a licensed version of Yourkit Java Profiler 2013 build 13066. We are trying to profile on a remote server behind a firewall so we adviced our network team to open the port 10005 to us. We also fixed the profiling port to 10005 in server.xml of WebSphere 8 Application Server on the remote server. The logs after the server starts are as follows

13086 - 7.568: java.net.InetSocketAddress$InetSocketAddressHolder
13086 - 7.578: Profiler agent is listening on port 10005
13086 - 7.584: com.optier.thirdparty.log4j.spi.ThrowableInformation
13086 - 7.589: com.yourkit.util.Util
13086 - 7.601: java.net.Socket 1 ms
13086 - 7.605: *** HINT ***: To get profiling results, connect to the application from the profiler UI
13086 - 7.605: agentInit: Core.init() called
13086 - 7.605: agentInit: OK
13086 - 7.605: startProfiling: 1
13086 - 7.605: startProfiling: OK


This indicates that the profiler did start profiling on port 10005

When we try to connect via the yourkit client UI from our machines here by specifying the <ip>:10005, it gives the error
Theres no application running at <ip> with profiler agent configured to listen on port 10005 or profiler agent is incompatible with current version of profiler
Detail: Connection timed out: connect

Connection timed out? is it that the port is not open? I can certainly ping the IP from my machine here, so the remote machine should be accessible and my network team did confirm that the port has been opened to us.

The other thing that i have tried is to run the yjp.sh -attach on our remote server itself to see which JVMs are running and to my surprise it says
No running JVMs to attach found.

but we do have a websphere application server running on this machine and the logs above did indicate that the profiler has started profiling on port 10005

on running the command : /opt/WebSphere80/profiles/appprofile/bin/serverStatus.sh -all
ADMU0116I: Tool information is being logged in file
/opt/WebSphere80/profiles/appprofile/logs/serverStatus.log
ADMU0128I: Starting tool with the appprofile profile
ADMU0503I: Retrieving server status for all servers
ADMU0505I: Servers found in configuration:
ADMU0506I: Server name: nodeagent
ADMU0506I: Server name: xxxxxxxxxxxxxx
ADMU0508I: The Node Agent "nodeagent" is STARTED

Please help on the following

1. What additional things can we do to investigate this further?

2. We have opted for advance paid support, please let me know how that can be utilized to get support from you?

Your help is much appreciated.

Thanks
Lalit Nagpal
Anton Katilin
Posts: 6172
Joined: Wed Aug 11, 2004 8:37 am

Re: Yourkit Remote Profiling - Not Attaching

Post by Anton Katilin »

Hello Lalit

About the attach mode: please see http://www.yourkit.com/docs/java/help/attach_agent.jsp

"Existing IBM VMs do not provide necessary capabilities to let YourKit Java Profiler function in the attach mode. To profile on IBM Java, please start application with the profiler agent. If necessary capabilities are added in future versions of IBM Java, the profiler will support attach mode for IBM Java too."

About the connection.
is it that the port is not open? I can certainly ping the IP from my machine here, so the remote machine should be accessible and my network team did confirm that the port has been opened to us.
The port may indeed be blocked. This may have happened on the server side, as well as on your local machine side.
Please check both sides.

What is the local machine platform? If not Windows, under which Java do you run the profiler UI on it?

Best regards,
Anton
Anton Katilin
Posts: 6172
Joined: Wed Aug 11, 2004 8:37 am

Re: Yourkit Remote Profiling - Not Attaching

Post by Anton Katilin »

P.S.
2. We have opted for advance paid support, please let me know how that can be utilized to get support from you?
You may also write to [email protected]
Anton Katilin
Posts: 6172
Joined: Wed Aug 11, 2004 8:37 am

Re: Yourkit Remote Profiling - Not Attaching

Post by Anton Katilin »

P.P.S.

Please also consider SSH tunneling approach instead of fighting with ports and firewalls.
Version 2014 EAP has a built-in support for SSH tunneling in the UI, so if you can SSH to the remote machine, you can try to connect using this new feature:
http://www.yourkit.com/eap
The Remote connection dialog in version 2014 will suggest appropriate syntax to establish SSH tunnel.
lalit.r.nagpal
Posts: 13
Joined: Wed Jul 09, 2014 2:07 pm

Re: Yourkit Remote Profiling - Not Attaching

Post by lalit.r.nagpal »

I am certainly going to investigate on whether the port is open or not on both sides.

My client platform is Windows 7 Enterprise Edition desktop - 64 bit. On the local desktop the java we use is Sun Java 1.6.0_35 version. On the remote server machine the java is 1.6.0_24. The only reason I tried the attach option was to simply see whether my server/application gets listed or not but it is not listing any of the JVMs. It says No running JVMs detected. The server logs certainly showed that YourKit did start profiling on 10005 port, so bit surprising on why it did not list my server/app in the list that comes out with yjp.sh -attach option.

For SSH, we have to go through a intermediate server to ssh the remote server that we want to profile. We do not have direct ssh access to the remote server.

Please suggest any other options I could try. If I write to [email protected], would you guys be able to provide support in investigating this further, maybe over a webex where we could show you this problem.
Anton Katilin
Posts: 6172
Joined: Wed Aug 11, 2004 8:37 am

Re: Yourkit Remote Profiling - Not Attaching

Post by Anton Katilin »

My client platform is Windows 7 Enterprise Edition desktop - 64 bit. On the local desktop the java we use is Sun Java 1.6.0_35 version.
If you installed the profiler with the Windows installer, which is the recommended option, the profiler runs with a bundled Java 7 JRE, no matter which other Java installations may exist on the machine.

So this may in fact be Java 7 on the Windows side and Java 6 on the server.

We have seen cross Java 6 and 7 connection issues (but this is not always the case), due to a different IPv6 stack implementation in Java 7.

I can suggest to try running the profiler UI with Java 6 instead. To do this, rename/remove the <profiler installation directory>\jre and jr64 directories on the Windows machine. If Java 6 is installed with the installer, the profiler UI will pick it up. You may check the Java version in the profiler UI in "Help | About".
It says No running JVMs detected. The server logs certainly showed that YourKit did start profiling on 10005 port, so bit surprising on why it did not list my server/app in the list that comes out with yjp.sh -attach option.
May it be that the server and "yjp.sh -attach" run under different users?
For SSH, we have to go through a intermediate server to ssh the remote server that we want to profile. We do not have direct ssh access to the remote server.
The feature supports indirect connections through intermediate server too. Please give it a try. It's very simple to use.
Please suggest any other options I could try. If I write to [email protected], would you guys be able to provide support in investigating this further, maybe over a webex where we could show you this problem.
Sorry, we provide email support only, as well as this forum.

If possible, could you please send the profiler agent logs in full by email. We'll try to check if there is anything special. The logs from the Windows machine are interesting too. In both cases, they are located in <user home>/.yjp/log
lalit.r.nagpal
Posts: 13
Joined: Wed Jul 09, 2014 2:07 pm

Re: Yourkit Remote Profiling - Not Attaching

Post by lalit.r.nagpal »

Hi Anton,

I upgraded my Yourkit client to 2014 EAP and tried the ssh option with
ssh <intermediate server ip address> <target server ip address>:10005.
It asked me for the password, i gave the password after this. There are two kinds of responses that I am getting

a) When I do not specify the port i.e. I use ssh <intermediate server ip address> <target server ip address>, it gives a prompt of "connecting to <intermediate server IP> via ssh" and then it gives a prompt of "Looking for Applications" ... so seems it does get connected and searches for JVMs ... it does not locate the application but ... and then finally prompts for "No applications being profiled can be automatically detected on the specified host. If you are sure there is a application, please explicity specify agent port"

There are no errors in the logs as well on Yourkit client side, just a dump of my local environment information - Properties and Environment.

b) When I specify a port i.e. I use ssh <intermediate server ip address> <target server ip address>:10005. It gives a prompt of "connecting to <intermediate server IP> via ssh" and then "Connecting to Remote Application" and then takes lots of time and then prompts for "Theres no application running at 127.0.0.1 with profiler agent configured to listen on port 65118 or profiler agent is incompatible with current version of profiler" Detail:null. There are errors in the Yourkit client side logs this time.

I am not sure why it is saying 127.0.0.1 ... thats not the server ip I specified, thats the local loopback address and also the port that I specified was 10005 and not 65118 ...

14074 - 620.225: com.yourkit.g.bu$1: error:
com.yourkit.util.i: com.yourkit.runtime.PresentableException: There's no application running at 127.0.0.1 with profiler agent configured to listen on port 65118
or profiler agent is incompatible with current version of profiler
at com.yourkit.g.a5.<init>(a:2)
at com.yourkit.d.ac$a.try(a:153)
at com.yourkit.g.ap$b.if(a:7)
at com.yourkit.g.h.m.actionPerformed(a:90)
at javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:2018)
at com.yourkit.g.h.j$1.fireActionPerformed(a:16)
at javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2341)
at javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:402)
at javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:259)
at javax.swing.plaf.basic.BasicButtonListener.mouseReleased(BasicButtonListener.java:252)
at java.awt.AWTEventMulticaster.mouseReleased(AWTEventMulticaster.java:289)
at java.awt.Component.processMouseEvent(Component.java:6516)
at javax.swing.JComponent.processMouseEvent(JComponent.java:3320)
at java.awt.Component.processEvent(Component.java:6281)
at java.awt.Container.processEvent(Container.java:2229)
at java.awt.Component.dispatchEventImpl(Component.java:4872)
at java.awt.Container.dispatchEventImpl(Container.java:2287)
at java.awt.Component.dispatchEvent(Component.java:4698)
at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4832)
at java.awt.LightweightDispatcher.processMouseEvent(Container.java:4492)
at java.awt.LightweightDispatcher.dispatchEvent(Container.java:4422)
at java.awt.Container.dispatchEventImpl(Container.java:2273)
at java.awt.Window.dispatchEventImpl(Window.java:2719)
at java.awt.Component.dispatchEvent(Component.java:4698)
at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:735)
at java.awt.EventQueue.access$200(EventQueue.java:103)
at java.awt.EventQueue$3.run(EventQueue.java:694)
at java.awt.EventQueue$3.run(EventQueue.java:692)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:76)
at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:87)
at java.awt.EventQueue$4.run(EventQueue.java:708)
at java.awt.EventQueue$4.run(EventQueue.java:706)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:76)
at java.awt.EventQueue.dispatchEvent(EventQueue.java:705)
at com.yourkit.Main1$b.dispatchEvent(a:21)
at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:242)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.yourkit.g.p.a(a:535)
at com.yourkit.g.p.a(a:315)
at com.yourkit.g.p.a(a:126)
at com.yourkit.d.ac.o(a:6)
at com.yourkit.g.u.q$1.actionPerformed(a:5)
at com.yourkit.g.br.do(a:124)
at com.yourkit.g.br.a(a:65)
at com.yourkit.g.br$1.mouseReleased(a:22)
at java.awt.Component.processMouseEvent(Component.java:6516)
at javax.swing.JComponent.processMouseEvent(JComponent.java:3320)
at java.awt.Component.processEvent(Component.java:6281)
at java.awt.Container.processEvent(Container.java:2229)
at java.awt.Component.dispatchEventImpl(Component.java:4872)
at java.awt.Container.dispatchEventImpl(Container.java:2287)
at java.awt.Component.dispatchEvent(Component.java:4698)
at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4832)
at java.awt.LightweightDispatcher.processMouseEvent(Container.java:4492)
at java.awt.LightweightDispatcher.dispatchEvent(Container.java:4422)
at java.awt.Container.dispatchEventImpl(Container.java:2273)
at java.awt.Window.dispatchEventImpl(Window.java:2719)
at java.awt.Component.dispatchEvent(Component.java:4698)
at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:735)
at java.awt.EventQueue.access$200(EventQueue.java:103)
at java.awt.EventQueue$3.run(EventQueue.java:694)
at java.awt.EventQueue$3.run(EventQueue.java:692)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:76)
at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:87)
at java.awt.EventQueue$4.run(EventQueue.java:708)
at java.awt.EventQueue$4.run(EventQueue.java:706)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:76)
at java.awt.EventQueue.dispatchEvent(EventQueue.java:705)
at com.yourkit.Main1$b.dispatchEvent(a:21)
at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:242)
at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:161)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:150)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:146)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:138)
at java.awt.EventDispatchThread.run(EventDispatchThread.java:91)
Caused by: com.yourkit.runtime.PresentableException: There's no application running at 127.0.0.1 with profiler agent configured to listen on port 65118
or profiler agent is incompatible with current version of profiler
at com.yourkit.api.ControllerImpl.a(a:404)
at com.yourkit.api.ControllerImpl.a(a:262)
at com.yourkit.api.ControllerImpl.<init>(a:67)
at com.yourkit.c.bk.<init>(a:11)
at com.yourkit.d.ac$a$2.a(a:3)
at com.yourkit.g.a5$1.run(a:1)
at com.yourkit.g.a5$b.run(a:2)
Caused by: java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:197)
at java.io.DataInputStream.readLong(DataInputStream.java:416)
at com.yourkit.runtime.Packet.readHeader(a:49)
at com.yourkit.runtime.Packet.readHeader(a:42)
at com.yourkit.runtime.Packet.readHeaderWithStandardResponse(a:9)
at com.yourkit.api.ControllerImpl$1.perform(a:6)
at com.yourkit.api.ControllerImpl.a(a:115)
... 5 more

I am checking with my company on whether I can exchange my logs with you but in the meantime can you advice what could be the issue here.
lalit.r.nagpal
Posts: 13
Joined: Wed Jul 09, 2014 2:07 pm

Re: Yourkit Remote Profiling - Not Attaching

Post by lalit.r.nagpal »

So since the IP in the error is the local loopback address 127.0.0.1 ( I had given proper IPs not this one ) and since the port is also coming out wrong. I am not sure whether I am reaching the target server or just the intermediate server in this case. On the target server we do have a instance of WebSphere App Server 8 running with our application deployed on it. The agent is configured and is running as per logs shared previously.
Vladimir Kondratyev
Posts: 1624
Joined: Tue Aug 10, 2004 7:52 pm

Re: Yourkit Remote Profiling - Not Attaching

Post by Vladimir Kondratyev »

Lalit, please send all logs from ~/.yjp/ directory. We need logs from server machine (where profiler is configured to listen port 10005) and we need logs from machine where you run profiler UI.

How many network interfaces do you have on server machine? May be profiler agent listen port 10005 on improper interface?
Vladimir Kondratyev
Posts: 1624
Joined: Tue Aug 10, 2004 7:52 pm

Re: Yourkit Remote Profiling - Not Attaching

Post by Vladimir Kondratyev »

Please also do a small test:

Profiler has command line tool which allows to control profiling modes, capture snapshots, etc.
Details are here http://www.yourkit.com/docs/java/help/c ... e_tool.jsp

Please install command line tool (yjp-controller-api-redist.jar) on the server machine and try to run any command.

The list of available command can be got by:
java -jar <Profiler Installation Directory>/lib/yjp-controller-api-redist.jar
lalit.r.nagpal
Posts: 13
Joined: Wed Jul 09, 2014 2:07 pm

Re: Yourkit Remote Profiling - Not Attaching

Post by lalit.r.nagpal »

I have also mailed you detailed logs at [email protected]. Here are some results via command line. Note: My port is no longer fixed to 10005 now, I am profiling on the port that was allocated by yourkit agent itself - 10001 in this case.

I have run the command utilities you mentioned and here is the output
Things look good here ( note: i have removed userids and servernames on purpose from logs )

[<user>@<servername> lib]$ java -jar yjp-controller-api-redist.jar localhost 10001 print-current-status
Allocation recording is off
CPU profiling is off
Monitor profiling is off

[<user>@<servername> lib]$ java -jar yjp-controller-api-redist.jar localhost 10001 start-cpu-sampling
CPU sampling started

[<user>@<servername> lib]$ java -jar yjp-controller-api-redist.jar localhost 10001 stop-cpu-profiling
CPU profiling stopped

server.xml in Websphere 80 contains the following ... plz note 2014 version, new one

<jvmEntries debugArgs="-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=7777" debugMode="false" disableJIT="false" executableJarFileName="" genericJvmArguments="-Xdump:system:none -Dclient.encoding.override=UTF-8 -DNFJAVA_DEBUG=163 -DNFJAVA_DEBUGFILE=/opt/nfast/log/nfjava_debug -DJCECSP_DEBUG=40 -DJCECSP_DEBUGFILE=/opt/nfast/log/jce_debug -javaagent:/opt/optier/wte/corefirst/tierext/websphere/common/lib/cf-plugins-instrument-agent.jar=optier.launch.properties=/opt/optier/wte/corefirst/tierext/websphere/conf/admin/Launch.properties -Doptier.home=/opt/optier/wte -Dcorefirst.home=/opt/optier/wte/corefirst -Dws.version=8.0 -agentpath:/opt/WebSphere80/yjp/yjp-2014-eap-build-14074/bin/linux-x86-64/libyjpagent.so=disablestacktelemetry,disableexceptiontelemetry,delay=10000,sessionname=WebSphere" hprofArguments="" initialHeapSize="1024" maximumHeapSize="1024" runHProf="false" verboseModeClass="false" verboseModeGarbageCollection="true" verboseModeJNI="false" xmi:id="JavaVirtualMachine_1183122130078">
<systemProperties name="com.ibm.security.jgss.debug" required="false" value="off" xmi:id="Property_1"/>
<systemProperties name="com.ibm.security.krb5.Krb5Debug" required="false" value="off" xmi:id="Property_2"/>
</jvmEntries>

Java Library Path in startServer.log

Java Library path = /opt/WebSphere80/AppServer/java/jre/lib/amd64/compressedrefs:/opt/WebSphere80/AppServer/java/jre/lib/amd64:/opt/WebSphere80/AppServer/lib/native/linux/x86_64/:/opt/WebSphere80/AppServer/bin::/opt/oracle/product/11.2.0/lib:/nas/scripts/gito/tools/TCL/lib:/opt/WebSphere80/yjp/yjp-2014-eap-build-14074/bin/linux-x86-64:/usr/lib

Log in the <home directory of server>/.yjp/log/WebSphere-27437.log
14074 - 8.490: setTriggers: to unregister: 0
14074 - 8.490: setTriggers: method listeners to register: 0
14074 - 8.490: setTriggers: OK
14074 - 8.490: shmem 42341490: OK
14074 - 8.491: shmem 4234148E: OK
14074 - 8.607: Profiler agent is listening on port 10001
14074 - 8.607: *** HINT ***: To get profiling results, connect to the application from the profiler UI
14074 - 8.607: agentInit: Core.init() called
14074 - 8.609: agentInit: OK
14074 - 8.609: startProfiling: 1
14074 - 8.609: startProfiling: OK
14074 - 28.393: total: 4000 classes 1984 ms
14074 - 54.871: total: 6000 classes 3116 ms
14074 - 79.942: total: 8000 classes 4193 ms
14074 - 102.163: total: 10000 classes 5290 ms
14074 - 120.342: total: 12000 classes 6569 ms
14074 - 148.752: total: 14000 classes 7722 ms
14074 - 150.231: warning: cannot instrument method because resulting bytecode is too long: sun/nio/cs/ext/IBM933$Encoder.<clinit>()V
14074 - 173.319: total: 16000 classes 8881 ms
14074 - 198.789: total: 18000 classes 10160 ms
14074 - 1093.206: Events from Auto probes cleared:

Logs on the client side
14074 - 34.490: com.yourkit.g.bu$1: error:
com.yourkit.util.i: com.yourkit.runtime.PresentableException: There's no application running at 127.0.0.1 with profiler agent configured to listen on port 50318
or profiler agent is incompatible with current version of profiler
at com.yourkit.g.a5.<init>(a:2)
at com.yourkit.d.ac$a.try(a:153)
at com.yourkit.g.ap$b.if(a:7)
at com.yourkit.g.h.m.actionPerformed(a:90)
at javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:2018)
at com.yourkit.g.h.j$1.fireActionPerformed(a:16)
at javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2341)
at javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:402)
at javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:259)
at javax.swing.plaf.basic.BasicButtonListener.mouseReleased(BasicButtonListener.java:252)
at java.awt.AWTEventMulticaster.mouseReleased(AWTEventMulticaster.java:289)
at java.awt.Component.processMouseEvent(Component.java:6516)
at javax.swing.JComponent.processMouseEvent(JComponent.java:3320)
at java.awt.Component.processEvent(Component.java:6281)
at java.awt.Container.processEvent(Container.java:2229)
at java.awt.Component.dispatchEventImpl(Component.java:4872)
at java.awt.Container.dispatchEventImpl(Container.java:2287)
at java.awt.Component.dispatchEvent(Component.java:4698)
at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4832)
at java.awt.LightweightDispatcher.processMouseEvent(Container.java:4492)
at java.awt.LightweightDispatcher.dispatchEvent(Container.java:4422)
at java.awt.Container.dispatchEventImpl(Container.java:2273)
at java.awt.Window.dispatchEventImpl(Window.java:2719)
at java.awt.Component.dispatchEvent(Component.java:4698)
at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:735)
at java.awt.EventQueue.access$200(EventQueue.java:103)
at java.awt.EventQueue$3.run(EventQueue.java:694)
at java.awt.EventQueue$3.run(EventQueue.java:692)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:76)
at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:87)
at java.awt.EventQueue$4.run(EventQueue.java:708)
at java.awt.EventQueue$4.run(EventQueue.java:706)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:76)
at java.awt.EventQueue.dispatchEvent(EventQueue.java:705)
at com.yourkit.Main1$b.dispatchEvent(a:21)
at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:242)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.yourkit.g.p.a(a:535)
at com.yourkit.g.p.a(a:315)
at com.yourkit.g.p.a(a:126)
at com.yourkit.d.ac.o(a:6)
at com.yourkit.g.u.q$1.actionPerformed(a:5)
at com.yourkit.g.br.do(a:124)
at com.yourkit.g.br.a(a:65)
at com.yourkit.g.br$1.mouseReleased(a:22)
at java.awt.Component.processMouseEvent(Component.java:6516)
at javax.swing.JComponent.processMouseEvent(JComponent.java:3320)
at java.awt.Component.processEvent(Component.java:6281)
at java.awt.Container.processEvent(Container.java:2229)
at java.awt.Component.dispatchEventImpl(Component.java:4872)
at java.awt.Container.dispatchEventImpl(Container.java:2287)
at java.awt.Component.dispatchEvent(Component.java:4698)
at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4832)
at java.awt.LightweightDispatcher.processMouseEvent(Container.java:4492)
at java.awt.LightweightDispatcher.dispatchEvent(Container.java:4422)
at java.awt.Container.dispatchEventImpl(Container.java:2273)
at java.awt.Window.dispatchEventImpl(Window.java:2719)
at java.awt.Component.dispatchEvent(Component.java:4698)
at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:735)
at java.awt.EventQueue.access$200(EventQueue.java:103)
at java.awt.EventQueue$3.run(EventQueue.java:694)
at java.awt.EventQueue$3.run(EventQueue.java:692)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:76)
at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:87)
at java.awt.EventQueue$4.run(EventQueue.java:708)
at java.awt.EventQueue$4.run(EventQueue.java:706)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:76)
at java.awt.EventQueue.dispatchEvent(EventQueue.java:705)
at com.yourkit.Main1$b.dispatchEvent(a:21)
at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:242)
at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:161)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:150)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:146)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:138)
at java.awt.EventDispatchThread.run(EventDispatchThread.java:91)
Caused by: com.yourkit.runtime.PresentableException: There's no application running at 127.0.0.1 with profiler agent configured to listen on port 50318
or profiler agent is incompatible with current version of profiler
at com.yourkit.api.ControllerImpl.a(a:404)
at com.yourkit.api.ControllerImpl.a(a:262)
at com.yourkit.api.ControllerImpl.<init>(a:67)
at com.yourkit.c.bk.<init>(a:11)
at com.yourkit.d.ac$a$2.a(a:3)
at com.yourkit.g.a5$1.run(a:1)
at com.yourkit.g.a5$b.run(a:2)
Caused by: java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:197)
at java.io.DataInputStream.readLong(DataInputStream.java:416)
at com.yourkit.runtime.Packet.readHeader(a:49)
at com.yourkit.runtime.Packet.readHeader(a:42)
at com.yourkit.runtime.Packet.readHeaderWithStandardResponse(a:9)
at com.yourkit.api.ControllerImpl$1.perform(a:6)
at com.yourkit.api.ControllerImpl.a(a:115)
... 5 more

"Just to see which Java Processes are reported by Yourkit". I ran this on server

No intention of profiling on server itself, have to profile via windows client of yourkit from local to remote machine. No processes being reported. The server has been started with the same user as the user that i am logged in with.

[<user name>@<server name> bin]$ ./yjp.sh -attach
Picked up JAVA_TOOL_OPTIONS:
[YourKit Java Profiler 2014 EAP build 14074] Log file: /home/wasadm/.yjp/log/yjp-30586.log
No running JVMs to attach found.

Enter PID of the application you want to attach (0 to exit) and press Enter:
>
Anton Katilin
Posts: 6172
Joined: Wed Aug 11, 2004 8:37 am

Re: Yourkit Remote Profiling - Not Attaching

Post by Anton Katilin »

Hi Lalit,

We've just answered by email.

We didn't know the port is no longer 10005, so please ignore corresponding suggestion to fix it to 10005.
The suggestion about possible multiple network interfaces is valid.

Best regards,
Anton
lalit.r.nagpal
Posts: 13
Joined: Wed Jul 09, 2014 2:07 pm

Re: Yourkit Remote Profiling - Not Attaching

Post by lalit.r.nagpal »

Hi Anton,

We tried to check if we can reach just the intermediate server first and it does connect and scan all the port from 10001 to 10002.
Since this is a intermediate server and yourkit agent is not publishing anything here to any port, naturally the connections failed, but it did connect and scan for all ports in the range 10001 - 10010, so that works good.

The /etc/log/messages file gave out this as below

Jul 18 10:25:00 <server name> sshd[20513]: error: connect_to 127.0.0.1 port 10001: failed.
Jul 18 10:25:00 <server name> sshd[20513]: error: connect_to 127.0.0.1 port 10008: failed.
Jul 18 10:25:00 <server name> sshd[20513]: error: connect_to 127.0.0.1 port 10010: failed.
Jul 18 10:25:00 <server name> sshd[20513]: error: connect_to 127.0.0.1 port 10004: failed.
Jul 18 10:25:00 <server name> sshd[20513]: error: connect_to 127.0.0.1 port 10002: failed.
Jul 18 10:25:00 <server name> sshd[20513]: error: connect_to 127.0.0.1 port 10007: failed.
Jul 18 10:25:00 <server name> sshd[20513]: error: connect_to 127.0.0.1 port 10009: failed.
Jul 18 10:25:00 <server name> sshd[20513]: error: connect_to 127.0.0.1 port 10005: failed.
Jul 18 10:25:00 <server name> sshd[20513]: error: connect_to 127.0.0.1 port 10006: failed.
Jul 18 10:25:00 <server name> sshd[20513]: error: connect_to 127.0.0.1 port 10003: failed.


After this we tried <user>@<intermediate server> <target server>:<port>

In this case the /var/log/messages 'of the intermediate server' revealed the following - call passed through it seems

Jul 18 10:27:21 <server name> sshd[20906]: Accepted keyboard-interactive/pam for wasadm from 136.49.103.65 port 54336 ssh2

and netstat -c | grep <target server hostname> command run on the target server gave out the following

tcp 0 1 <intermediate server>:26222 <target server>:10010 SYN_SENT
tcp 0 1 <intermediate server>:27762 <target server>:10003 SYN_SENT
tcp 0 1 <intermediate server>:14517 <target server>:10009 SYN_SENT
tcp 0 1 <intermediate server>:12332 <target server>:10005 SYN_SENT
tcp 0 1 <intermediate server>:10070 <target server>:10004 SYN_SENT
tcp 0 1 <intermediate server>:6886 <target server>:10006 SYN_SENT
tcp 0 1 <intermediate server>:61701 <target server>:10002 SYN_SENT
tcp 0 1 <intermediate server>:56813 <target server>:octopus SYN_SENT
tcp 0 0 <intermediate server>:30180 <target server>:ssh ESTABLISHED
tcp 0 0 <intermediate server>:28282 <target server>:ssh ESTABLISHED
tcp 0 1 <intermediate server>:26222 <target server>:10010 SYN_SENT
tcp 0 1 <intermediate server>:27762 <target server>:10003 SYN_SENT


So seems
a) The call from YK client went to intermediate server
b) The packets were also being sent from intermediate server to target server ( on ports 10001 - 10010 - SYN_SENT ),
c) But, It does not reach the target server, there was nothing in the /var/log/messages of the target server and nothing reported by netstat -c . Connection to the target server is not getting established.

We have the following queries please

a) What mechanism does yourkit code use for reaching target server after doing ssh with intermediate server - is it ssh as well or tcp connection?

b) How can we make the 2nd call i.e. the call from intermediate server to tcp server a ssh call too?

c) Any advice based on content above?

Important for us to understand this as call is not getting through from intermediate server to target server. It might be a issue in the network connection, but if you can point things, will appreciate that.

We are not receiving a reply to our emails, so thought will post it here as well. It might be a issue at our end but your thoughts will help.
lalit.r.nagpal
Posts: 13
Joined: Wed Jul 09, 2014 2:07 pm

Re: Yourkit Remote Profiling - Not Attaching

Post by lalit.r.nagpal »

Please read the following line above
and netstat -c | grep <target server hostname> command run on the target server gave out the following
as
and netstat -c | grep <target server hostname> command run on the intermediate server gave out the following
Anton Katilin
Posts: 6172
Joined: Wed Aug 11, 2004 8:37 am

Re: Yourkit Remote Profiling - Not Attaching

Post by Anton Katilin »

Hi Lalit,

We'll check the provided information thoroughly, and if we find anything to suggest we'll inform you ASAP.
a) What mechanism does yourkit code use for reaching target server after doing ssh with intermediate server - is it ssh as well or tcp connection?
If I'm not mistaken, it's a "plain" TCP. That is, it's assumed that the intermediate server can connect to <target server>:10xxx via TCP. I'll double check this fact as soon as I get access to the source code (I'm writing from home).
b) How can we make the 2nd call i.e. the call from intermediate server to tcp server a ssh call too?
If the above assumption is true, I can suggest to manually establish an SSH tunnel Local machine -> intermediate server -> target server. Please note that the built-in SSH tunnel support in the profiler UI is just a usability feature. If for some reason it does not work for you out of the box, you can manually establish the tunnel, then connect via the Local machine's local port.

Best regards,
Anton
Post Reply