Opening Large snapshots

Questions about YourKit Java Profiler
Post Reply
aedwards
Posts: 6
Joined: Mon Mar 14, 2016 5:33 pm

Opening Large snapshots

Post by aedwards »

What is the ratio of snapshot size to heap size requirement?

With the latest release.
I'm having trouble opening a snapshot that is 4.9GB
I went all the way up to 12Gb of heap but it still failed with out of memory error

In older versions I was usually ok with snapshot size + 1 or 2 Gb

Also when you define Xmx in the ini file it does not replace the one from the script as my jvm args contain two memory options.
Anton Katilin
Posts: 6172
Joined: Wed Aug 11, 2004 8:37 am

Re: Opening Large snapshots

Post by Anton Katilin »

What ini file do you edit?
Please note that in the latest version 2016.02 it is now <user home>/.yjp/ui.ini

What is your platform?
Exactly what error message do you get?
aedwards
Posts: 6
Joined: Mon Mar 14, 2016 5:33 pm

Re: Opening Large snapshots

Post by aedwards »

That was the ini file I edited. It is entirely possible it was using the value set in my ini file but because it always crashed I don't really know all I noticed was that both values show up.

Code: Select all

16.02-30 0.037: Command line:
  /home/alex/servers/yjp-2016.02/bin/../jre64/bin/java
  -Xmx4G
  -XX:+HeapDumpOnOutOfMemoryError
  -Xmx8G
  -XX:OnError=/home/alex/servers/yjp-2016.02/bin/../bin/yjp.sh -on_error %p
  -jar
  /home/alex/servers/yjp-2016.02/bin/../lib/yjp.jar
16.02-30 0.037: Executable: /home/alex/servers/yjp-2016.02/jre64/bin/java
16.02-30 0.037: uname:
  sysname: Linux
  release: 3.13.0-83-generic
  version: #127-Ubuntu SMP Fri Mar 11 00:25:37 UTC 2016
  machine: x86_64
When I disable the setting in ini file and set it directly yjp.sh to 12GB when I try to open the snapshot I still get an out of memory error for a 4.9GB snapshot.
java.lang.OutOfMemoryError: Java heap space

It does capture a heap dump but I don't think I can attach them here.
Anton Katilin
Posts: 6172
Joined: Wed Aug 11, 2004 8:37 am

Re: Opening Large snapshots

Post by Anton Katilin »

If you still get the "java.lang.OutOfMemoryError: Java heap space" please set the heap size even bigger.
aedwards
Posts: 6
Joined: Mon Mar 14, 2016 5:33 pm

Re: Opening Large snapshots

Post by aedwards »

I understand that but that's not what I'm after.

I am interested to know what the relationship between heap size and snapshot size is.

Right now I have a problem with a 4.9GB snapshot and a 14GB heap (I tried a bit more) at this point I am going to have to pay for an aws slave with 32GB/64GB of ram just to try and open it because I don't have any hardware with more than 16GB of ram this is not a practical solution.

Assuming that all my snapshots were captured using the following settings =tracing how much heap would I need to open snapshots of these sizes:
  • 500Mb
    1GB
    2GB
Anton Katilin
Posts: 6172
Joined: Wed Aug 11, 2004 8:37 am

Re: Opening Large snapshots

Post by Anton Katilin »

Sorry, there is no direct formula. Snapshot file on disk has its own format, it does not match the Java heap format and their sizes differ. How much memory it takes to load a snapshot does not directly depend on snapshot size on disk, but rather on how many objects it contains and how they refer each other. A 5 GB snapshot file may for example contain only 10 K objects size of some of those is huge (big arrays) or 100 M small objects. Opening the latter would take much more time and memory.

Since you mention "=tracing". Are you capturing a memory snapshot of an application that was running with CPU tracing?

Please note that heap size and CPU profiling data size have nothing in common. If your snapshot size is huge because it contains a lot of CPU tracing, please ensure adaptive tracing in on or consider using CPU sampling instead.

You can check the tracing data size by exporting a performance snapshot from Open snapshot dialog (Ctrl+O).
Anton Katilin
Posts: 6172
Joined: Wed Aug 11, 2004 8:37 am

Re: Opening Large snapshots

Post by Anton Katilin »

Anyway, as a rough estimation, you may consider that in some cases the top (sic!) memory needed to load a memory snapshot may be up to 3x of the snapshot size.

As a workaround please try setting the property -Dyjp.dominators=false as suggested and explained here: viewtopic.php?t=4420#p15658

You may also provide the profiler log file in full so we'll check what causes the out of memory in particular. I hope this may allow us to give you more explicit suggestions.
aedwards
Posts: 6
Joined: Mon Mar 14, 2016 5:33 pm

Re: Opening Large snapshots

Post by aedwards »

Thanks for the explanation, its helpful to know opening snapshots is more related to my objects than its size. The application i'm profiling makes a lot of objects!

What I actually wanted to do was to start up my app run it with some load for about 30 min to warm up the jit then capture about 5 min of profiling. I found I couldn't do this because after 30 min attaching the profiler was to intensive and caused things to fail.

I think I read somewhere that I could start up the application with the instrumentation already loaded but just attach the UI later in a lightweight way, I might try using the delay command line option if I have no success otherwise.
Anton Katilin
Posts: 6172
Joined: Wed Aug 11, 2004 8:37 am

Re: Opening Large snapshots

Post by Anton Katilin »

Yes, you may start CPU profiling on demand at a later time point. It is almost never necessary to run CPU tracing or sampling right from the start.
Post Reply