Fully featured low overhead profiler for Java EE and Java SE platforms.
Performance monitoring and profiling of Jenkins, TeamCity, Gradle, Maven, Ant, JUnit and TestNG.
Easy to use performance and memory profiler for .NET framework.

Predefined port misidentifed as in-use under Kubernetes

Moderators: Vladimir Kondratyev, Anton Katilin

Predefined port misidentifed as in-use under Kubernetes

Postby dominics » Fri Jul 26, 2019 3:58 am

I'm receiving constant spurious "Cannot start: predefined port 10001 is already in use" messages (i.e. when the port is not actually in use)

I've read and understand a previous issue at viewtopic.php?f=3&t=4965 which describes how the profiler agent uses an "enhanced" port-in-use detection mechanism involving shared memory. At the time, that issue lead to somewhat of an impasse:

> Please let me ask my question once again: why reuse the PIDs? You are the first one who reports this problem. I guess your case is unusual.

But since that issue was raised, Docker and Kubernetes have become much more popular. Docker reuses PIDs constantly - you'll almost always be running your main application process as PID 1. And to make matters worse, Kubernetes gives each container its own process namespace but shares the IPC namespace between containers in a pod. (In k8s terms, a pod is a closely related set of Docker containers)

The upshot of this is:

- The first time the container starts within a fresh pod, YourKit starts happily
- But if the pod ever restarts for any reason (OOM, SIGTERM to PID 1, etc.) then, when the container is restarted, it still has the same shared memory entry for the PID and port
- So, YourKit refuses to attempt to bind the port, because it thinks it is in use (because /proc/<1> exists)

The workarounds provided so far are inadequate:

- Using `listen=localhost` will mean I can't access the profiler port (I want it exposed as a container port)
- Using a range of ports would mean I'd need to expose every one, and which one is used would depend on how many restarts the container has done (and, besides, it'd still fail after 10 restarts)

Ultimately, I don't think you should be assuming that the process and IPC namespaces are aligned. But in the meantime, there should be a direct way to disable the shared memory "enhanced" port-in-use detection, because at the moment it causes way too many false positives in Kubernetes deployment scenarios.
dominics
 
Posts: 2
Joined: Fri Jul 26, 2019 3:42 am

Re: Predefined port misidentifed as in-use under Kubernetes

Postby Anton Katilin » Fri Jul 26, 2019 6:49 am

Will it solve the problem if we add the profiler agent option to disable the port and PID check?
Anton Katilin
 
Posts: 5826
Joined: Wed Aug 11, 2004 8:37 am

Re: Predefined port misidentifed as in-use under Kubernetes

Postby dominics » Fri Jul 26, 2019 8:25 am

Anton Katilin wrote:Will it solve the problem if we add the profiler agent option to disable the port and PID check?


It absolutely would! An explicit startup option for this would be ideal. And if that's the solution, amazing!

But the most annoying thing is that it doesn't even try to bind the port, and if it did try it would succeed. So the more satisfying way to fix it is: if you've been given a single port to run on (not a range), and you think the port might be already in use, try to bind it anyway as a last ditch effort so you can be sure it's in use (and handle the failure if one happens).
dominics
 
Posts: 2
Joined: Fri Jul 26, 2019 3:42 am

Re: Predefined port misidentifed as in-use under Kubernetes

Postby Anton Katilin » Mon Jul 29, 2019 8:03 am

Hello,

Next 2019.8 EAP build will offer new option "skip_advanced_port_check".

But the most annoying thing is that it doesn't even try to bind the port, and if it did try it would succeed.
This is made intentionally to address some IPv4/IPv6 issues with older Java versions which let the same port be opened differently, effectively making a profiled process invisible to the profiler.
Anton Katilin
 
Posts: 5826
Joined: Wed Aug 11, 2004 8:37 am


Return to Java Profiler

Who is online

Users browsing this forum: No registered users and 20 guests

cron

YourKit uses cookies and other tracking technologies to improve your browsing experience on our website, to show you personalized content and targeted ads, to analyze our website traffic, and to understand where our visitors are coming from.

By browsing our website, you consent to our use of cookies and other tracking technologies in accordance with the Privacy Policy.