What is the ratio of snapshot size to heap size requirement?
With the latest release.
I'm having trouble opening a snapshot that is 4.9GB
I went all the way up to 12Gb of heap but it still failed with out of memory error
In older versions I was usually ok with snapshot size + 1 or 2 Gb
Also when you define Xmx in the ini file it does not replace the one from the script as my jvm args contain two memory options.
Opening Large snapshots
-
- Posts: 6172
- Joined: Wed Aug 11, 2004 8:37 am
Re: Opening Large snapshots
What ini file do you edit?
Please note that in the latest version 2016.02 it is now <user home>/.yjp/ui.ini
What is your platform?
Exactly what error message do you get?
Please note that in the latest version 2016.02 it is now <user home>/.yjp/ui.ini
What is your platform?
Exactly what error message do you get?
-
- Posts: 6
- Joined: Mon Mar 14, 2016 5:33 pm
Re: Opening Large snapshots
That was the ini file I edited. It is entirely possible it was using the value set in my ini file but because it always crashed I don't really know all I noticed was that both values show up.
When I disable the setting in ini file and set it directly yjp.sh to 12GB when I try to open the snapshot I still get an out of memory error for a 4.9GB snapshot.
java.lang.OutOfMemoryError: Java heap space
It does capture a heap dump but I don't think I can attach them here.
Code: Select all
16.02-30 0.037: Command line:
/home/alex/servers/yjp-2016.02/bin/../jre64/bin/java
-Xmx4G
-XX:+HeapDumpOnOutOfMemoryError
-Xmx8G
-XX:OnError=/home/alex/servers/yjp-2016.02/bin/../bin/yjp.sh -on_error %p
-jar
/home/alex/servers/yjp-2016.02/bin/../lib/yjp.jar
16.02-30 0.037: Executable: /home/alex/servers/yjp-2016.02/jre64/bin/java
16.02-30 0.037: uname:
sysname: Linux
release: 3.13.0-83-generic
version: #127-Ubuntu SMP Fri Mar 11 00:25:37 UTC 2016
machine: x86_64
java.lang.OutOfMemoryError: Java heap space
It does capture a heap dump but I don't think I can attach them here.
-
- Posts: 6172
- Joined: Wed Aug 11, 2004 8:37 am
Re: Opening Large snapshots
If you still get the "java.lang.OutOfMemoryError: Java heap space" please set the heap size even bigger.
-
- Posts: 6
- Joined: Mon Mar 14, 2016 5:33 pm
Re: Opening Large snapshots
I understand that but that's not what I'm after.
I am interested to know what the relationship between heap size and snapshot size is.
Right now I have a problem with a 4.9GB snapshot and a 14GB heap (I tried a bit more) at this point I am going to have to pay for an aws slave with 32GB/64GB of ram just to try and open it because I don't have any hardware with more than 16GB of ram this is not a practical solution.
Assuming that all my snapshots were captured using the following settings =tracing how much heap would I need to open snapshots of these sizes:
I am interested to know what the relationship between heap size and snapshot size is.
Right now I have a problem with a 4.9GB snapshot and a 14GB heap (I tried a bit more) at this point I am going to have to pay for an aws slave with 32GB/64GB of ram just to try and open it because I don't have any hardware with more than 16GB of ram this is not a practical solution.
Assuming that all my snapshots were captured using the following settings =tracing how much heap would I need to open snapshots of these sizes:
- 500Mb
1GB
2GB
-
- Posts: 6172
- Joined: Wed Aug 11, 2004 8:37 am
Re: Opening Large snapshots
Sorry, there is no direct formula. Snapshot file on disk has its own format, it does not match the Java heap format and their sizes differ. How much memory it takes to load a snapshot does not directly depend on snapshot size on disk, but rather on how many objects it contains and how they refer each other. A 5 GB snapshot file may for example contain only 10 K objects size of some of those is huge (big arrays) or 100 M small objects. Opening the latter would take much more time and memory.
Since you mention "=tracing". Are you capturing a memory snapshot of an application that was running with CPU tracing?
Please note that heap size and CPU profiling data size have nothing in common. If your snapshot size is huge because it contains a lot of CPU tracing, please ensure adaptive tracing in on or consider using CPU sampling instead.
You can check the tracing data size by exporting a performance snapshot from Open snapshot dialog (Ctrl+O).
Since you mention "=tracing". Are you capturing a memory snapshot of an application that was running with CPU tracing?
Please note that heap size and CPU profiling data size have nothing in common. If your snapshot size is huge because it contains a lot of CPU tracing, please ensure adaptive tracing in on or consider using CPU sampling instead.
You can check the tracing data size by exporting a performance snapshot from Open snapshot dialog (Ctrl+O).
-
- Posts: 6172
- Joined: Wed Aug 11, 2004 8:37 am
Re: Opening Large snapshots
Anyway, as a rough estimation, you may consider that in some cases the top (sic!) memory needed to load a memory snapshot may be up to 3x of the snapshot size.
As a workaround please try setting the property -Dyjp.dominators=false as suggested and explained here: viewtopic.php?t=4420#p15658
You may also provide the profiler log file in full so we'll check what causes the out of memory in particular. I hope this may allow us to give you more explicit suggestions.
As a workaround please try setting the property -Dyjp.dominators=false as suggested and explained here: viewtopic.php?t=4420#p15658
You may also provide the profiler log file in full so we'll check what causes the out of memory in particular. I hope this may allow us to give you more explicit suggestions.
-
- Posts: 6
- Joined: Mon Mar 14, 2016 5:33 pm
Re: Opening Large snapshots
Thanks for the explanation, its helpful to know opening snapshots is more related to my objects than its size. The application i'm profiling makes a lot of objects!
What I actually wanted to do was to start up my app run it with some load for about 30 min to warm up the jit then capture about 5 min of profiling. I found I couldn't do this because after 30 min attaching the profiler was to intensive and caused things to fail.
I think I read somewhere that I could start up the application with the instrumentation already loaded but just attach the UI later in a lightweight way, I might try using the delay command line option if I have no success otherwise.
What I actually wanted to do was to start up my app run it with some load for about 30 min to warm up the jit then capture about 5 min of profiling. I found I couldn't do this because after 30 min attaching the profiler was to intensive and caused things to fail.
I think I read somewhere that I could start up the application with the instrumentation already loaded but just attach the UI later in a lightweight way, I might try using the delay command line option if I have no success otherwise.
-
- Posts: 6172
- Joined: Wed Aug 11, 2004 8:37 am
Re: Opening Large snapshots
Yes, you may start CPU profiling on demand at a later time point. It is almost never necessary to run CPU tracing or sampling right from the start.