Why heap dump limit is always around 840~850mb?

Questions about YourKit Java Profiler
Locked
jun
Posts: 11
Joined: Fri Feb 07, 2020 7:02 am

Why heap dump limit is always around 840~850mb?

Post by jun »

Hi,
I am new to this board & yourkit profiler too. So please let me know if my way of writing is not appropriate for this board.

1.
https://imgur.com/a/9M3K5is
On image there we can see heap limit is around 840mb.
When I give -Xmx 128M, 1G, 2G, .. it's always 840mb. Something wrong in my configuration?
My full configuration is below;

-Xmx 1G -Xms 1G -J-agentpath:/usr/share/yourkit/bin/linux-x86-64/libyjpagent.so=exceptions=disable,disable_async_sampling,listen=all

2.
When I made a trigger, heap memory > 90% then, CaptureMemorySnapshot + CaptureHprofSnapshot, however, I usually get only hprof file. Is there any specific reason? How to ensure getting memory snapshot

Thanks
Anton Katilin
Posts: 6172
Joined: Wed Aug 11, 2004 8:37 am

Re: Why heap dump limit is always around 840~850mb?

Post by Anton Katilin »

Hello,

1. It's strange. The limit should reflect the -Xmx value.
Do you really use spaces in "-Xmx 1G -Xms 1G"? Please specify the options without them: "-Xmx1G -Xms1G"

2. Could you please provide the full definition of your trigger.

It would help if you could share the profiler agent log file ~/.yjp/log/<session name>-<pid>.log

Best regards,
Anton
Anton Katilin
Posts: 6172
Joined: Wed Aug 11, 2004 8:37 am

Re: Why heap dump limit is always around 840~850mb?

Post by Anton Katilin »

P.S.
> CaptureMemorySnapshot + CaptureHprofSnapshot,
Please note that a memory snapshot supersedes a HPROF snapshot. You can capture memory snapshot only. If you need a HPROF snapshot per-se i.e. a .hprof file, you can extract it via the command

java -jar <path to>/lib/yourkit.jar -snapshot2hprof <source-file>.snapshot <target-file>.hprof
jun
Posts: 11
Joined: Fri Feb 07, 2020 7:02 am

Re: Why heap dump limit is always around 840~850mb?

Post by jun »

Hi,
I am using scala so I forgot giving -J option to use java option.
Now I can see heap limit is increased when giving -J-Xmx1G
Thanks
Anton Katilin
Posts: 6172
Joined: Wed Aug 11, 2004 8:37 am

Re: Why heap dump limit is always around 840~850mb?

Post by Anton Katilin »

Hi,

Have you also resolved the issue with triggers? If you haven't, we need to see the trigger definition.

Best regards,
Anton
jun
Posts: 11
Joined: Fri Feb 07, 2020 7:02 am

Re: Why heap dump limit is always around 840~850mb?

Post by jun »

Thanks,
As I wrote above, with -J, it's resolved.
Anton Katilin
Posts: 6172
Joined: Wed Aug 11, 2004 8:37 am

Re: Why heap dump limit is always around 840~850mb?

Post by Anton Katilin »

Thank you for confirmation.
jun
Posts: 11
Joined: Fri Feb 07, 2020 7:02 am

Re: Why heap dump limit is always around 840~850mb?

Post by jun »

Hi, this is another problem.
Last time, I tried to use yourkit profiler on my local machine, and tried to reach test server. With -J option I could've seen it worked.
Now I applied options for prod server, and I couldn't get memory snapshot. I changed a little bit like below;
https://imgur.com/a/j9t5II4
And there were OOM problem, so my process restarted around 13:56 so why yourkit log restarted logging like the 2nd image.
The 3rd image is the tail of logging when my process died.
I checked the server health status, and there were OOM but I couldn't get memory snapshot. What's wrong in my jvm option even though I used "usedmem=90"?
Thanks
Anton Katilin
Posts: 6172
Joined: Wed Aug 11, 2004 8:37 am

Re: Why heap dump limit is always around 840~850mb?

Post by Anton Katilin »

"usedmem" option instructs the profiler agent to capture a snapshot when memory usage reaches the specified threshold. Memory usage is checked and compared periodically in the agent's telemetry collecting thread, by default once a second. According to the first log file, the telemetry thread pauses were actually much longer due to other thread activities.

It seems that memory grown too high (as high as causing an OOME) between the samples, so the trigger didn't have chance to work. The trigger is _not_ guarantied to always work in fast memory growing scenarios.

Heap dump on OOME is a different thing. It is triggered by the JVM itself, internally. Oracle documentation explains where the dump file is created:
https://docs.oracle.com/javase/8/docs/t ... ts001.html
"
By default the heap dump is created in a file called java_pid<pid>.hprof in the working directory of the VM <...>. You can specify an alternative file name or directory with the -XX:HeapDumpPath= option. For example -XX:HeapDumpPath=/disk2/dumps will cause the heap dump to be generated in the /disk2/dumps directory.
"

To reduce telemetry sample overhead please add the agent option "disablestacktelemetry". By the way, it is recommended by default in the EE server integration wizard. It is recommended to use the wizard instead of manually composing -agentpath because the wizard offers useful option presets: https://www.yourkit.com/docs/java/help/ ... remote.jsp
jun
Posts: 11
Joined: Fri Feb 07, 2020 7:02 am

Re: Why heap dump limit is always around 840~850mb?

Post by jun »

Ok, I executed integrate.sh, then applied options.
I will take a look at the server, and write more questions if necessary.
Thanks.
jun
Posts: 11
Joined: Fri Feb 07, 2020 7:02 am

Re: Why heap dump limit is always around 840~850mb?

Post by jun »

Hi,
When I tested with and without yourkit profiler option like below, there are too much difference in memory usage. I made that option using yourkit/bin/integrate.sh
-J-Xmx7G -J-Xms7G -J-agentpath:/usr/share/yourkit/bin/linux-x86-64/libyjpagent.so=delay=10000,usedmem=90,listen=localhost
-J-Xmx7G -J-Xms7G
This is memory status. Both are the same spec servers, and run the same application. Blue line from 11th to 13th, I applied yourkit option on daytimes. After I removed it, then blue and red goes similar in 14th.
https://imgur.com/a/sszeQWX
Thanks,
Jun
Anton Katilin
Posts: 6172
Joined: Wed Aug 11, 2004 8:37 am

Re: Why heap dump limit is always around 840~850mb?

Post by Anton Katilin »

Hi Jun,

Please add "disablestacktelemetry" as earlier suggested.

Best regards,
Anton
jun
Posts: 11
Joined: Fri Feb 07, 2020 7:02 am

Re: Why heap dump limit is always around 840~850mb?

Post by jun »

Dear Anton,
Thanks for your help, now it goes stable. very similar to another for which I didn't apply yourkit.
Thanks.
Locked