Fully featured low overhead profiler for Java EE and Java SE platforms.
Performance monitoring and profiling of Jenkins, TeamCity, Gradle, Maven, Ant, JUnit and TestNG.
Easy to use performance and memory profiler for .NET framework.

Cannot start: predefined port <port> is already opened

Moderators: Vladimir Kondratyev, Anton Katilin

Cannot start: predefined port <port> is already opened

Postby daniil » Thu Aug 22, 2013 4:03 pm

Hello.

We are faced out with a situation when java application with profiling won't start with the error "predefined port <port> is already opened", but from OS point of view there are not any opened ports with <port> number (netstat, lsof etc). Error reproduced some time, and then suddenly leave off although we didn't change any parameters in configuration.

We were trying to see what happens in system calls. There is unsuccessful attempt:
Code: Select all
...
[pid  3461] shmget(0x42341491, 4, IPC_CREAT|0666) = 622595
[pid  3461] shmat(622595, 0, SHM_RND)   = ?
[pid  3461] open("/proc/3274", O_RDONLY) = 7
[pid  3461] close(7)                    = 0
[pid  3461] shmget(0x4234148e, 524288, IPC_CREAT|0666) = 491524
[pid  3461] shmat(491524, 0, SHM_RND)   = ?
[pid  3461] write(3, "11.0.10 - 1.375: shmem 4234148E: OK\n", 36) = 36
[pid  3461] write(3, "11.0.10 - 1.376: Cannot start: predefined port 40000 is already "..., 71) = 71
...

and what we can see when application was started successfully:
Code: Select all
...
[pid  3505] shmget(0x42341491, 4, IPC_CREAT|0666) = 622595
[pid  3505] shmat(622595, 0, SHM_RND)   = ?
[pid  3505] open("/proc/3274", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid  3505] shmget(0x4234148e, 524288, IPC_CREAT|0666) = 491524
[pid  3505] shmat(491524, 0, SHM_RND)   = ?
[pid  3505] write(3, "11.0.10 - 1.264: shmem 4234148E: OK\n", 36) = 36
[pid  3505] lseek(5, 56138597, SEEK_SET) = 56138597
...


It seems like YJP is storing pair "<port> - <pid>" in shared memory and if directory /proc/<pid> exists -- generates the error message mentioned above.

Looks like there is a bug, if application with <pid> exists profiler won't start despite the fact that port is not used.

Please find "steps to reproduce" below:
1. Start any java application with profiling (-agentlib:yjpagent=port=<port>);
2. Find the <pid> of the application ;
3. Stop application;
4. Start any application with the same pid, for example you can use simple script:
Code: Select all
#!/bin/bash
[ x$1 = x ] && exit 254
new_p(){
    [ $1 -eq $BASHPID ] && read -p "$BASHPID"
}
while true; do (new_p $1); done

5. Start java application again and it fail with "predefined port <port> is already opened" error message in log.

Is that known issue? Do you have any plans to fix it?
daniil
 
Posts: 4
Joined: Thu Aug 22, 2013 3:07 pm

Re: Cannot start: predefined port <port> is already opened

Postby Anton Katilin » Mon Aug 26, 2013 7:16 am

Hi,

Which version of the profiler do you use?

What is the platform?

What is the output of "uname -a"?

It seems like YJP is storing pair "<port> - <pid>" in shared memory and if directory /proc/<pid> exists -- generates the error message mentioned above.

Yes, it tracks live processes by checking /proc/<pid> on the platforms with a procfs. Normally, OS shouldn't reuse PID numbers for reasonably long time. Once it is detected that some PID no longer corresponds to a live process, its associated port is considered free.

Best regards,
Anton
Anton Katilin
 
Posts: 5820
Joined: Wed Aug 11, 2004 8:37 am

Re: Cannot start: predefined port <port> is already opened

Postby daniil » Mon Aug 26, 2013 9:03 am

YJP's version 11.0.10.
Java:
Code: Select all
java version "1.7.0_17"
Java(TM) SE Runtime Environment (build 1.7.0_17-b02)
Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)

Kernel:
Code: Select all
Linux ***** 2.6.32-34-server #77-Ubuntu SMP Tue Sep 13 20:54:38 UTC 2011 x86_64 GNU/Linux


Actually don't understand for what reasons you have to check port/pid pair because if port is used by OS we'll get exception from JVM:
11.0.10 - 0.599: Cannot start: cannot listen on port 40000: java.net.BindException: Address already in use
at java.net.PlainSocketImpl.$$YJP$$socketBind(Native Method)
at java.net.PlainSocketImpl.socketBind(PlainSocketImpl.java)
at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:376)
at java.net.ServerSocket.bind(ServerSocket.java:376)
at java.net.ServerSocket.<init>(ServerSocket.java:237)
at java.net.ServerSocket.<init>(ServerSocket.java:128)
at com.yourkit.runtime.Core.createServerSocket(Core.java:737)
at com.yourkit.runtime.Core._init(Core.java:202)
at com.yourkit.runtime.Core.init(Core.java:131)
daniil
 
Posts: 4
Joined: Thu Aug 22, 2013 3:07 pm

Re: Cannot start: predefined port <port> is already opened

Postby Anton Katilin » Mon Aug 26, 2013 3:10 pm

YJP's version 11.0.10.

This version is unsupported. Please instead use the latest released version:
http://www.yourkit.com/download
or the EAP:
http://www.yourkit.com/eap

Actually don't understand for what reasons you have to check port/pid pair because if port is used by OS we'll get exception from JVM:
One of the reasons was the change in Java 7 IPv4/IPv6 socket implementation which made it possible to open the same port as IPv4 and IPv4 at the same time, which resulted in connection issues. Another reason is that we already have this mapping for local application detection, thus can use it to avoid checking ports which are for sure busy.

Anyway, what is the problem with this mapping itself? For what purpose do you launch processes with the same PID?
Anton Katilin
 
Posts: 5820
Joined: Wed Aug 11, 2004 8:37 am

Re: Cannot start: predefined port <port> is already opened

Postby daniil » Mon Aug 26, 2013 3:48 pm

This version is unsupported. Please instead use the latest released version:

But the same mechanism is working in versions 12.*, am I right?

As was mentioned above:
We are faced out with a situation when java application with profiling won't start with the error "predefined port <port> is already opened", but from OS point of view there are not any opened ports with <port> number (netstat, lsof etc). Error reproduced some time, and then suddenly leave off although we didn't change any parameters in configuration.

And was not so easy to understand what happened. If something like that'll happen in production environment users can lose many time to detect root cause.
daniil
 
Posts: 4
Joined: Thu Aug 22, 2013 3:07 pm

Re: Cannot start: predefined port <port> is already opened

Postby Anton Katilin » Mon Aug 26, 2013 4:21 pm

But the same mechanism is working in versions 12.*, am I right?

Yes. But it's a good idea to always try the latest version. There were changes and fixes. Perhaps it will work for you. Anyway, we won't fix anything in 11.x but in the latest version only.

Please let me ask my question once again: why reuse the PIDs? You are the first one who reports this problem. I guess your case is unusual.
Anton Katilin
 
Posts: 5820
Joined: Wed Aug 11, 2004 8:37 am

Re: Cannot start: predefined port <port> is already opened

Postby daniil » Tue Aug 27, 2013 3:01 pm

Anton Katilin wrote:Please let me ask my question once again: why reuse the PIDs? You are the first one who reports this problem. I guess your case is unusual.


We are faced with a situation when the application fails with an error "Cannot start: predefined port XXXX is already opened", although the port was not busy. In real life. And it is not clear what is happening until we took up strace and began to speculate about the possible causes.

As one of reasonable solutions -- change the error message to make it clear that the port is possible, but not necessarily, is used by an application with a specific PID.
daniil
 
Posts: 4
Joined: Thu Aug 22, 2013 3:07 pm

Re: Cannot start: predefined port <port> is already opened

Postby Anton Katilin » Fri Aug 30, 2013 8:56 am

As a workaround, please use startup option "listen=localhost:<port>" instead of "port=<port>" to avoid the port check.

http://www.yourkit.com/docs/java/help/s ... ptions.jsp
Anton Katilin
 
Posts: 5820
Joined: Wed Aug 11, 2004 8:37 am


Return to Java Profiler

Who is online

Users browsing this forum: No registered users and 15 guests