Cannot save snapshots under Solaris

[Morten Buhr]

Only occasionally do I succeed in saving a CPU snapshot when debugging an application running under Solaris. Normally I receive the error: "Cannot open file to write snapshot: <filename>".

My setup is: application running under Solaris 8 being remotely profiled from a PC running Windows 2000.

/Morten

[Anton Katilin]

Hello Morten,

Could you please provide more details regarding your Solaris configuration:

- under what user account does the profiled application run? It can be a problem with permissions, or maybe with disk quotes.
- do you use any non-ASCII characters in file name?
- have you used non-default directory to store captured snapshots?


Best regards,
Anton

[Morten Buhr]

Hi Anton,

I am running under a normal user account that do have all the necessary access rights for writing the snapshot. What seems strange to me is that I do sometimes succeed saving the snapshot after having tried saving the same snapshot a few times. Please note that a snapshot file is created at each trial, however it is empty when I do not succeed saving it. Trying to save to the same snapshot file name results in a new snapshot file where the filename is appended with a number. Fx saving to a file called "test" three times in a row where the two first trials does not succeed results in three snapshot files named: "test.cpu" "test(1).cpu" and "test(2).cpu" where the two first are empty and the last one contains the snapshot.

I have only used standard ASCII characters for the file name, as eg "test".

I have tried to use several locations for the directory to store the snapshots, including the standard location and /var/tmp.

I hope this describes my problem more clearly.

/Morten

[Vladimir Kondratyev]

OK. We will send you libyjpagent.so which prints verbose debug information (in 30 minutes). Please send to [email protected] your e-mail; and we'll mail agent to you.

[Vladimir Kondratyev]

I've already e-mailed you this libyjpagent.so library.

[Morten Buhr]

I have now had the opportunity to try out the libyjpagent.so that prints verbose information. However, the output have not made me come closer to an undestanding of the problem. The only additional info I get is:

fopen failed: No such file or directory

despite the fact that the file is actually created, but empty.

The verbose info does not appear when the saving of the snapshot succeeds.

/Morten

[Vladimir Kondratyev]

OK. Thank you very much for reply. We'll add some additional debug info into agent and send it back to you.

Your problem might happen if the number of open files exceeds the maximum limit. Do you know how many files can be simultaneously open in your environment?

[Morten Buhr]

According to ulimit the maximum number of open files are set to 256. I will try to raise this limit to 1024.

/Morten

[Morten Buhr]

Changing the maximum number of open files to 1024 did not solve the problem.

/Morten

[chadm]

I have seen this too with 4.0.6. I can capture one memory profile then sometimes I get this error for the subsequent ones in that run. I am doing remote profiling on Solaris

[chadm]

After more investigation:

It appears that when the load clients are still running the snapshot won't work. When we stop or reduce the load I can take a snapshot.

I think this may be related to open filehandles or such.

[jjoubert]

Did you guys resolve this issue?
I've got a similar issue using YJP 4.0.14 trying to make snapshots of memory dumps on Solaris 5.8 or 5.9. I'm always getting:

16:27:32,039 ERROR [STDERR] [YourKit Java Profiler 4.0.14]
16:27:32,040 ERROR [STDERR]
16:27:32,040 ERROR [STDERR] Request from
16:27:32,041 ERROR [STDERR] <ip address>
16:27:32,041 ERROR [STDERR] :
16:27:32,044 ERROR [STDERR] Cannot capture memory snapshot: Cannot open file to write snapshot: <filename>

The file is actually being created, but with a size of 0. Retries do not fix the problem and "lower" load on my application also makes no difference.
I'm using the libyjpagent.so library in the bin/solaris-sparc distribution folder, as we are running SPARC.

Any help would be appreciated

[Vladimir Kondratyev]

Unfortunately we cannot reproduce this problem (everything is just fine on our SPARC Solaris 9). Do you have any special (non latin) symbols in snapshot file name?

[jjoubert]

The snapshot file name I used was: "nisdev01.yjp.01.memory" which YJP by default dumps to my home dir, so the failed path in my case was: "/export/home/usa/nisdev01.yjp.01.memory"

The weird thing for me, is that the file gets created (0 bytes), but nothing written to it.

[Vladimir Kondratyev]

It's _very_ strange. The problem is that native "fopen" method returns NULL but creates empty file.

I've just googled "solaris fopen problem" and found that numerous people complain about libc Solaris limitation: there are only 256 open files per process.