.Net core profile activation "not an attachable process"

[aravindjp]

Hi ,

We are trying to enable YourKit profiling for a .Net core application running on Linux ("Ubuntu") VM . Application has been started and running fine . But when we try to attach process to yourkit for remote profiling
using command : bin/YourKitTools.sh attach <PID > , we get this error

Attaching to PID: <PID>
Error: Attach failed:
not an attachable process

If we run command : bin/YourKitTools.sh status

YourKit .NET Profiler - Command line tool.
No running .NET applications detected

we are running .net core 2.1 . Can you please help ?

[Anton Katilin]

Please try to start the profiled application with the agent instead:
https://www.yourkit.com/docs/dotnet/hel ... manual.jsp

Does this work?

[aravindjp]

Hi ,

Thank you very much for your response . When I followed step as you suggested (tried with non root user and as root user) , I am able to get proper response for status command:

bin/YourKitTools.sh status
[YourKit .NET Profiler 2021.3-b99] Log file: /root/.ynp/log/YourKitTools.dll-65894.log
YourKit .NET Profiler - Command line tool.
Running .NET applications:
Name | PID | User | Profiler Agent | Port
-----------------+--------------+------+----------------+-------
YourKitTools.dll | 65894 64-bit | root | Loaded | 10001

But we were unable to attach remotely to this VM . Hence we tried to attach PID specifically as

bin/YourKitTools.sh attach <PID>, but got a response like as mentioned below

YourKit .NET Profiler - Command line tool.
Attaching to PID: 65150
Error: Attach failed:
not an attachable process

[Anton Katilin]

To enable remote connection please specify the agent startup option "listen=all":
https://www.yourkit.com/docs/dotnet/hel ... jsp#listen

From https://www.yourkit.com/docs/dotnet/hel ... manual.jsp :
"You can change profiler startup options by setting YNP_STARTUP_OPTIONS variable."

[aravindjp]

Hi ,

I have tried listen=all also , but still not helping

Let me summarise what I have done till now

I have a remote VM (OS Ubuntu ) which I want to profile . For this , downloaded and unziped yourkit in that VM
I already have .netcore application running in that machine

After that:

In Remote VM, executed following commands
-----------
export CORECLR_PROFILER={2647D6B1-EA0D-4ED5-A612-283F3678426F}
export CORECLR_ENABLE_PROFILING=1
export CORECLR_PROFILER_PATH_64=/home/YourKit-NetProfiler-2021.3/bin/linux-x86-64/libynpagent.so
bin/YourKitTools.sh enable listen=all

[YourKit .NET Profiler 2021.3-b99] Log file: /root/.ynp/log/YourKitTools.dll-76304.log
YourKit .NET Profiler - Command line tool.
Running .NET applications:
Name | PID | User | Profiler Agent | Port
-----------------+--------------+------+----------------+-------
YourKitTools.dll | 76304 64-bit | root | Loaded | 10001


Now :

bin/YourKitTools.sh attach <PID> [Is this step really needed for profiling ? ]
[YourKit .NET Profiler 2021.3-b99] Log file: /root/.ynp/log/YourKitTools.dll-76358.log
YourKit .NET Profiler - Command line tool.
Attaching to PID: 74439
Error: Attach failed:
not an attachable process

[Anton Katilin]

The command "YourKitTools.sh enable" works on Windows only, it is not available on Linux.

In addition to

Code: Select all

export CORECLR_PROFILER={2647D6B1-EA0D-4ED5-A612-283F3678426F}
export CORECLR_ENABLE_PROFILING=1
export CORECLR_PROFILER_PATH_64=/home/YourKit-NetProfiler-2021.3/bin/linux-x86-64/libynpagent.so

please also do

Code: Select all

export YNP_STARTUP_OPTIONS=listen=all

before launching the process you want to profile.

After you have ensured that the agent has loaded...

Code: Select all

YourKitTools.dll | 76304 64-bit | root | Loaded | 10001


...there is no need to do "attach" because it is just an alternative way to load the agent.

At that moment, please remotely connect to the profiled process from the profiler UI.

[aravindjp]

HI ,

Thank you very much for your response . We are still struggling with few issues .
We executed commands in following order:

export CORECLR_PROFILER={2647D6B1-EA0D-4ED5-A612-283F3678426F}
export CORECLR_ENABLE_PROFILING=1
export YNP_STARTUP_OPTIONS=listen=all
export CORECLR_PROFILER_PATH_64=/home/tech/unzip_dir/YourKit-NetProfiler-2021.3/bin/linux-x86-64/libynpagent.so


bin/YourKitTools.sh status shows:

YourKit .NET Profiler - Command line tool.
Running .NET applications:
Name | PID | User | Profiler Agent | Port
-----------------+---------------+------+----------------+-------
YourKitTools.dll | 124377 64-bit | root | Loaded | 10001

But if I check if any process is running in 10001 using below commonad, there is NO process at all listening to 10001

sudo lsof -i -P -n | grep LISTEN


YourKitTools.dll-124377.log says:

21.3-99 0.107: GCEvent(-): 1 -> 0 first GC
21.3-99 0.404: warning: multiple structs with name Interop/ErrorInfo have been detected. Fields of these types will not be written t
o memory snapshots.
21.3-99 0.508: warning: multiple structs with name Interop/ErrorInfo have been detected. Fields of these types will not be written t
o memory snapshots.
21.3-99 0.546: warning: multiple structs with name Interop/Sys/OpenFlags have been detected. Fields of these types will not be writt
en to memory snapshots.
21.3-99 0.548: warning: multiple structs with name Interop/Sys/FileStatus have been detected. Fields of these types will not be writ
ten to memory snapshots.
21.3-99 0.548: warning: multiple structs with name Interop/Error have been detected. Fields of these types will not be written to me
mory snapshots.
21.3-99 0.680: warning: multiple structs with name Interop/Error have been detected. Fields of these types will not be written to me
mory snapshots.
21.3-99 0.737: warning: multiple structs with name System.Text.ValueStringBuilder have been detected. Fields of these types will not
be written to memory snapshots.
21.3-99 0.836: warning: multiple structs with name System.Collections.Generic.ValueListBuilder<System.Int32> have been detected. Fie
lds of these types will not be written to memory snapshots.
21.3-99 1.005: warning: multiple structs with name Interop/Sys/SeekWhence have been detected. Fields of these types will not be writ
ten to memory snapshots.
21.3-99 1.083: Shutting down...

but if i run status commonad again: Yourkit output looks good

bin/YourKitTools.sh status
[YourKit .NET Profiler 2021.3-b99] Log file: <path>/log/YourKitTools.dll-124550.log
YourKit .NET Profiler - Command line tool.
Running .NET applications:
Name | PID | User | Profiler Agent | Port
-----------------+---------------+------+----------------+-------
YourKitTools.dll | 124550 64-bit | root | Loaded | 10001

[Vladimir Kondratyev]

aravindjp, from what I see, you set the environment variable in the console where you run YourKitTools.sh. As the result profiling is turned on for YourKitTools.dll itself, because it is a .NET Core app.

YourKitTools finds itself, and you see YourKitTools.dll in the table. As soon as YourKitTools finishes, profiling is stopped, so "sudo lsof -i -P -n | grep LISTEN" shows nothing.

You need to set environment for the process which you want to profile. The environment should be set _before_ your app is started.

Important notice: attach (hooking agent into running process) works for .NET Core 3.0+ and newer. If you are using .NET Core 2.1, you must set environment variables.

Regards,
Vladimir Kondratyev

[aravindjp]

Hi Vladimir,
Thank you so much for your response . We are using .Net core 2.1 .
When you say "You need to set environment for the process which you want to profile. The environment should be set _before_ your app is started" - which enviroment variables are you referring to ? If you are referring to 4 variables , CORECLR_PROFILER, CORECLR_ENABLE_PROFILING, YNP_STARTUP_OPTIONS ,CORECLR_PROFILER_PATH_64 ( last step) , yes we are setting all variables before we start .net application.

When set variable CORECLR_PROFILER_PATH_64 , status automatically returns the below output (means we don't start ANY other script) other than starting actual .net application application which is configured as a service.
YourKit .NET Profiler - Command line tool.
Running .NET applications:
Name | PID | User | Profiler Agent | Port
-----------------+--------------+------+----------------+-------
YourKitTools.dll | 76304 64-bit | root | Loaded | 10001

[Vladimir Kondratyev]

yes we are setting all variables before we start .net application.

If you want to profiler foobar.dll app, you need

1) Open terminal.
2) Set and export CORECLR_PROFILER, CORECLR_ENABLE_PROFILING, YNP_STARTUP_OPTIONS ,CORECLR_PROFILER_PATH_64 variables.
3) Start your foobar.dll from the terminal.

From another terminal (where the variable above are not exported) you start YourKitTools.sh and foobar.dll should be in the output. If you do not see foobar.dll in the list, then the variable were not set.

You can use

Code: Select all

sudo cat /proc/[process ID]/environ

command to check the environment of your .NET core app.

Please also try to add option "recursive" to YNP_STARTUP_OPTIONS. In your case it should look like YNP_STARTUP_OPTIONS=listen=all,recursive

[aravindjp]

Code: Select all

sudo cat /proc/[process ID]/environ

was a good clue to fix the issue . As the application was running as a systemd service , service was not picking the environment. variables which i was setting from shell .
We have to set varable properly as mentioned here: https://serverfault.com/questions/413397/how-to-set-environment-variable-in-systemd-service. But problem is still not resolved fully . Now using Yourkit .net profiler UI , I am successfully able to connect . Test connection also says "No Issues found" . But it displays No application found . I can confrim that yourkit status shows PID of .net application PID(190081) and even i can see
/.root/.ynp/log/dotnetLogicoTraffic.dll-190081.log. Only warning i am seeing in the log is (log of 190081.log)

Code: Select all

1.3-99 0.025: setTriggers: OK
21.3-99 0.056: warning: FindTypeDefByName failed: hr=0x80131130; name=System.Security.Permissions.SecurityAction; sc
opeRef=0x23000001
21.3-99 0.056: warning: FindTypeDefByName failed: hr=0x80131130; name=System.Security.Permissions.SecurityPermission
Attribute; scopeRef=0x23000001
21.3-99 0.062: Successfully registered probes: 13
 

[Anton Katilin]

Could you please provide the log file in full. If you cannot share it in the forum please send it attached to [email protected]

[aravindjp]

Yourkit log

Code: Select all

21.3-99 0.117: Log file: /root/.ynp/log/YourKitTools.dll-203057.log
21.3-99 0.117: Command line:
  /home/tech/unzip_dir/YourKit-NetProfiler-2021.3/bin/../dotnet/64/dotnet
  /home/tech/unzip_dir/YourKit-NetProfiler-2021.3/bin/../lib/YourKitTools/net5.0/YourKitTools.dll
  status
21.3-99 0.117: Executable: /home/tech/unzip_dir/YourKit-NetProfiler-2021.3/dotnet/64/dotnet
21.3-99 0.117: uname:
  sysname: Linux
  release: 5.4.0-1038-gcp
  version: #41-Ubuntu SMP Fri Feb 26 15:54:29 UTC 2021
  machine: x86_64
21.3-99 0.117: Linux; linux-x86-64; 64-bit process
21.3-99 0.117: hid: 36026157989Z159941X477478Y70513157 64-bit machine
21.3-99 0.117: RAM: 7265 MB
21.3-99 0.117: Today is 20210402
21.3-99 0.117: full_build_stamp: 2021.3.99 202103181946
21.3-99 0.117: Agent library path: /home/tech/unzip_dir/YourKit-NetProfiler-2021.3/lib/YourKitTools/net5.0/../../../bin/linux-x86-64/libynpagent.so
21.3-99 0.117: Profiler GUID: {2647D6B1-EA0D-4ED5-A612-283F3678426F}
21.3-99 0.117: getTimeNsInfo: "monotonic"
21.3-99 0.117: shmem 42341490: OK


dotnet######.dll-190081.log

Code: Select all


21.3-99 0.017: Log file: /root/.ynp/log/dotnetxxxxx.dll-190081.log
21.3-99 0.017: Command line:
  /usr/bin/dotnet
  /var/www/mywebsite/html/dotnetxxxxxx.dll
21.3-99 0.017: Executable: /usr/share/dotnet/dotnet
21.3-99 0.017: uname:
  sysname: Linux
  release: 5.4.0-1038-gcp
  version: #41-Ubuntu SMP Fri Feb 26 15:54:29 UTC 2021
  machine: x86_64
21.3-99 0.017: Linux; linux-x86-64; 64-bit process
21.3-99 0.018: hid: 36026157989Z159941X477478Y70513157 64-bit machine
21.3-99 0.019: RAM: 7265 MB
21.3-99 0.019: Today is 20210402
21.3-99 0.019: full_build_stamp: 2021.3.99 202103181946
21.3-99 0.019: Agent library path: /home/tech/unzip_dir/YourKit-NetProfiler-2021.3/bin/linux-x86-64/libynpagent.so
21.3-99 0.019: Profiler GUID: {2647D6B1-EA0D-4ED5-A612-283F3678426F}
21.3-99 0.019: getTimeNsInfo: "monotonic"
21.3-99 0.019: Reading startup options
21.3-99 0.019: Environment variable YNP_STARTUP_OPTIONS: listen=all,recursive
21.3-99 0.019: CLR_Agent_OnLoad: options=listen=all,recursive
21.3-99 0.019: y_setenv: COR_ENABLE_PROFILING=1
21.3-99 0.019: y_setenv: COR_PROFILER={2647D6B1-EA0D-4ED5-A612-283F3678426F}
21.3-99 0.019: y_setenv: CORECLR_ENABLE_PROFILING=1
21.3-99 0.019: y_setenv: CORECLR_PROFILER={2647D6B1-EA0D-4ED5-A612-283F3678426F}
21.3-99 0.019: y_setenv: YNP_STARTUP_OPTIONS=listen=all,recursive
21.3-99 0.019: skip advanced port check: 0; inside Kubernetes: 0
21.3-99 0.019: .NET Core version: v2.2
21.3-99 0.019: session_id: 816393295935929 (1617344953,190081)
21.3-99 0.022: line_method: 1
21.3-99 0.022: MetaDataGetDispenser: fn: OK
21.3-99 0.023: MetaDataGetDispenser: 0x00000000 00000000007A4270
21.3-99 0.023: OpenScopeOnMemory: 0x00000000
21.3-99 0.023: shmem 42341490: OK
21.3-99 0.023: Listening connections on all network interfaces
21.3-99 0.024: Profiler agent is listening on port 10001
21.3-99 0.024: GCEvent(+): 1 -> 2 GC telemetry
21.3-99 0.024: GC telemetry provider: profiler events
21.3-99 0.024: setTriggers:

21.3-99 0.025: setTriggers: OK
21.3-99 0.056: warning: FindTypeDefByName failed: hr=0x80131130; name=System.Security.Permissions.SecurityAction; scopeRef=0x23000001
21.3-99 0.056: warning: FindTypeDefByName failed: hr=0x80131130; name=System.Security.Permissions.SecurityPermissionAttribute; scopeRef=0x23000001
21.3-99 0.062: Successfully registered probes: 13
  Registered: .Cassandra
  Registered: .CassandraLW
  Registered: .Databases
  Registered: .DatabasesLW
  Registered: .Files
  Registered: .FilesLW
  Registered: .MongoDB
  Registered: .MongoDBLW
  Registered: .Processes
  Registered: .Sockets
  Registered: .SocketsLW
  Registered: .WebPages
  Registered: .WebPagesLW
21.3-99 0.062: Events from Auto probes cleared:

21.3-99 0.126: GCEvent(-): 2 -> 1  first GC


[Anton Katilin]

Hi,

Did you provide the log dotnet######.dll-190081.log in full? Was the process still running when you tried to connect to it?

According to the log, the agent is listening on port and remote connections are possible:

21.3-99 0.023: Listening connections on all network interfaces
21.3-99 0.024: Profiler agent is listening on port 10001

[aravindjp]

Yes , I have provided the full log . As you mentioned , application is ready to accept connection and "Test Connection" also is successful ! But from UI says "No Application found"