Predefined port misidentifed as in-use under Kubernetes

[dominics]

I'm receiving constant spurious "Cannot start: predefined port 10001 is already in use" messages (i.e. when the port is not actually in use)

I've read and understand a previous issue at viewtopic.php?f=3&t=4965 which describes how the profiler agent uses an "enhanced" port-in-use detection mechanism involving shared memory. At the time, that issue lead to somewhat of an impasse:

> Please let me ask my question once again: why reuse the PIDs? You are the first one who reports this problem. I guess your case is unusual.

But since that issue was raised, Docker and Kubernetes have become much more popular. Docker reuses PIDs constantly - you'll almost always be running your main application process as PID 1. And to make matters worse, Kubernetes gives each container its own process namespace but shares the IPC namespace between containers in a pod. (In k8s terms, a pod is a closely related set of Docker containers)

The upshot of this is:

- The first time the container starts within a fresh pod, YourKit starts happily
- But if the pod ever restarts for any reason (OOM, SIGTERM to PID 1, etc.) then, when the container is restarted, it still has the same shared memory entry for the PID and port
- So, YourKit refuses to attempt to bind the port, because it thinks it is in use (because /proc/<1> exists)

The workarounds provided so far are inadequate:

- Using `listen=localhost` will mean I can't access the profiler port (I want it exposed as a container port)
- Using a range of ports would mean I'd need to expose every one, and which one is used would depend on how many restarts the container has done (and, besides, it'd still fail after 10 restarts)

Ultimately, I don't think you should be assuming that the process and IPC namespaces are aligned. But in the meantime, there should be a direct way to disable the shared memory "enhanced" port-in-use detection, because at the moment it causes way too many false positives in Kubernetes deployment scenarios.

[Anton Katilin]

Will it solve the problem if we add the profiler agent option to disable the port and PID check?

[dominics]

Will it solve the problem if we add the profiler agent option to disable the port and PID check?

It absolutely would! An explicit startup option for this would be ideal. And if that's the solution, amazing!

But the most annoying thing is that it doesn't even try to bind the port, and if it did try it would succeed. So the more satisfying way to fix it is: if you've been given a single port to run on (not a range), and you think the port might be already in use, try to bind it anyway as a last ditch effort so you can be sure it's in use (and handle the failure if one happens).

[Anton Katilin]

Hello,

Next 2019.8 EAP build will offer new option "skip_advanced_port_check".

But the most annoying thing is that it doesn't even try to bind the port, and if it did try it would succeed.

This is made intentionally to address some IPv4/IPv6 issues with older Java versions which let the same port be opened differently, effectively making a profiled process invisible to the profiler.

[Anton Katilin]

Update:

The agent startup option "skip_advanced_port_check" is available since version 2019.8:
https://www.yourkit.com/docs/java/help/ ... port_check