Cannot start: predefined port <port> is already opened

Questions about YourKit Java Profiler
Post Reply
daniil
Posts: 4
Joined: Thu Aug 22, 2013 3:07 pm

Cannot start: predefined port <port> is already opened

Post by daniil »

Hello.

We are faced out with a situation when java application with profiling won't start with the error "predefined port <port> is already opened", but from OS point of view there are not any opened ports with <port> number (netstat, lsof etc). Error reproduced some time, and then suddenly leave off although we didn't change any parameters in configuration.

We were trying to see what happens in system calls. There is unsuccessful attempt:

Code: Select all

...
[pid  3461] shmget(0x42341491, 4, IPC_CREAT|0666) = 622595
[pid  3461] shmat(622595, 0, SHM_RND)   = ?
[pid  3461] open("/proc/3274", O_RDONLY) = 7
[pid  3461] close(7)                    = 0
[pid  3461] shmget(0x4234148e, 524288, IPC_CREAT|0666) = 491524
[pid  3461] shmat(491524, 0, SHM_RND)   = ?
[pid  3461] write(3, "11.0.10 - 1.375: shmem 4234148E: OK\n", 36) = 36
[pid  3461] write(3, "11.0.10 - 1.376: Cannot start: predefined port 40000 is already "..., 71) = 71
...
and what we can see when application was started successfully:

Code: Select all

...
[pid  3505] shmget(0x42341491, 4, IPC_CREAT|0666) = 622595
[pid  3505] shmat(622595, 0, SHM_RND)   = ?
[pid  3505] open("/proc/3274", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid  3505] shmget(0x4234148e, 524288, IPC_CREAT|0666) = 491524
[pid  3505] shmat(491524, 0, SHM_RND)   = ?
[pid  3505] write(3, "11.0.10 - 1.264: shmem 4234148E: OK\n", 36) = 36
[pid  3505] lseek(5, 56138597, SEEK_SET) = 56138597
...


It seems like YJP is storing pair "<port> - <pid>" in shared memory and if directory /proc/<pid> exists -- generates the error message mentioned above.

Looks like there is a bug, if application with <pid> exists profiler won't start despite the fact that port is not used.

Please find "steps to reproduce" below:
1. Start any java application with profiling (-agentlib:yjpagent=port=<port>);
2. Find the <pid> of the application ;
3. Stop application;
4. Start any application with the same pid, for example you can use simple script:

Code: Select all

#!/bin/bash
[ x$1 = x ] && exit 254 
new_p(){
    [ $1 -eq $BASHPID ] && read -p "$BASHPID"
}
while true; do (new_p $1); done
5. Start java application again and it fail with "predefined port <port> is already opened" error message in log.

Is that known issue? Do you have any plans to fix it?
Anton Katilin
Posts: 6172
Joined: Wed Aug 11, 2004 8:37 am

Re: Cannot start: predefined port <port> is already opened

Post by Anton Katilin »

Hi,

Which version of the profiler do you use?

What is the platform?

What is the output of "uname -a"?
It seems like YJP is storing pair "<port> - <pid>" in shared memory and if directory /proc/<pid> exists -- generates the error message mentioned above.
Yes, it tracks live processes by checking /proc/<pid> on the platforms with a procfs. Normally, OS shouldn't reuse PID numbers for reasonably long time. Once it is detected that some PID no longer corresponds to a live process, its associated port is considered free.

Best regards,
Anton
daniil
Posts: 4
Joined: Thu Aug 22, 2013 3:07 pm

Re: Cannot start: predefined port <port> is already opened

Post by daniil »

YJP's version 11.0.10.
Java:

Code: Select all

java version "1.7.0_17"
Java(TM) SE Runtime Environment (build 1.7.0_17-b02)
Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)
Kernel:

Code: Select all

Linux ***** 2.6.32-34-server #77-Ubuntu SMP Tue Sep 13 20:54:38 UTC 2011 x86_64 GNU/Linux
Actually don't understand for what reasons you have to check port/pid pair because if port is used by OS we'll get exception from JVM:
11.0.10 - 0.599: Cannot start: cannot listen on port 40000: java.net.BindException: Address already in use
at java.net.PlainSocketImpl.$$YJP$$socketBind(Native Method)
at java.net.PlainSocketImpl.socketBind(PlainSocketImpl.java)
at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:376)
at java.net.ServerSocket.bind(ServerSocket.java:376)
at java.net.ServerSocket.<init>(ServerSocket.java:237)
at java.net.ServerSocket.<init>(ServerSocket.java:128)
at com.yourkit.runtime.Core.createServerSocket(Core.java:737)
at com.yourkit.runtime.Core._init(Core.java:202)
at com.yourkit.runtime.Core.init(Core.java:131)
Anton Katilin
Posts: 6172
Joined: Wed Aug 11, 2004 8:37 am

Re: Cannot start: predefined port <port> is already opened

Post by Anton Katilin »

YJP's version 11.0.10.
This version is unsupported. Please instead use the latest released version:
http://www.yourkit.com/download
or the EAP:
http://www.yourkit.com/eap
Actually don't understand for what reasons you have to check port/pid pair because if port is used by OS we'll get exception from JVM:
One of the reasons was the change in Java 7 IPv4/IPv6 socket implementation which made it possible to open the same port as IPv4 and IPv4 at the same time, which resulted in connection issues. Another reason is that we already have this mapping for local application detection, thus can use it to avoid checking ports which are for sure busy.

Anyway, what is the problem with this mapping itself? For what purpose do you launch processes with the same PID?
daniil
Posts: 4
Joined: Thu Aug 22, 2013 3:07 pm

Re: Cannot start: predefined port <port> is already opened

Post by daniil »

This version is unsupported. Please instead use the latest released version:
But the same mechanism is working in versions 12.*, am I right?

As was mentioned above:
We are faced out with a situation when java application with profiling won't start with the error "predefined port <port> is already opened", but from OS point of view there are not any opened ports with <port> number (netstat, lsof etc). Error reproduced some time, and then suddenly leave off although we didn't change any parameters in configuration.
And was not so easy to understand what happened. If something like that'll happen in production environment users can lose many time to detect root cause.
Anton Katilin
Posts: 6172
Joined: Wed Aug 11, 2004 8:37 am

Re: Cannot start: predefined port <port> is already opened

Post by Anton Katilin »

But the same mechanism is working in versions 12.*, am I right?
Yes. But it's a good idea to always try the latest version. There were changes and fixes. Perhaps it will work for you. Anyway, we won't fix anything in 11.x but in the latest version only.

Please let me ask my question once again: why reuse the PIDs? You are the first one who reports this problem. I guess your case is unusual.
daniil
Posts: 4
Joined: Thu Aug 22, 2013 3:07 pm

Re: Cannot start: predefined port <port> is already opened

Post by daniil »

Anton Katilin wrote:Please let me ask my question once again: why reuse the PIDs? You are the first one who reports this problem. I guess your case is unusual.
We are faced with a situation when the application fails with an error "Cannot start: predefined port XXXX is already opened", although the port was not busy. In real life. And it is not clear what is happening until we took up strace and began to speculate about the possible causes.

As one of reasonable solutions -- change the error message to make it clear that the port is possible, but not necessarily, is used by an application with a specific PID.
Anton Katilin
Posts: 6172
Joined: Wed Aug 11, 2004 8:37 am

Re: Cannot start: predefined port <port> is already opened

Post by Anton Katilin »

As a workaround, please use startup option "listen=localhost:<port>" instead of "port=<port>" to avoid the port check.

http://www.yourkit.com/docs/java/help/s ... ptions.jsp
Post Reply