Fully featured low overhead profiler for Java EE and Java SE platforms.
Ultimate profiling and monitoring solution for Gradle, Maven, Ant, JUnit and TestNG.
Easy to use performance and memory profiler for .NET framework.

Opening Large snapshots

Moderators: Vladimir Kondratyev, Anton Katilin

Opening Large snapshots

Postby aedwards » Wed Mar 16, 2016 6:17 pm

What is the ratio of snapshot size to heap size requirement?

With the latest release.
I'm having trouble opening a snapshot that is 4.9GB
I went all the way up to 12Gb of heap but it still failed with out of memory error

In older versions I was usually ok with snapshot size + 1 or 2 Gb

Also when you define Xmx in the ini file it does not replace the one from the script as my jvm args contain two memory options.
aedwards
 
Posts: 6
Joined: Mon Mar 14, 2016 5:33 pm

Re: Opening Large snapshots

Postby Anton Katilin » Thu Mar 17, 2016 8:00 am

What ini file do you edit?
Please note that in the latest version 2016.02 it is now <user home>/.yjp/ui.ini

What is your platform?
Exactly what error message do you get?
Anton Katilin
 
Posts: 5576
Joined: Wed Aug 11, 2004 8:37 am

Re: Opening Large snapshots

Postby aedwards » Thu Mar 17, 2016 3:17 pm

That was the ini file I edited. It is entirely possible it was using the value set in my ini file but because it always crashed I don't really know all I noticed was that both values show up.

Code: Select all
16.02-30 0.037: Command line:
  /home/alex/servers/yjp-2016.02/bin/../jre64/bin/java
  -Xmx4G
  -XX:+HeapDumpOnOutOfMemoryError
  -Xmx8G
  -XX:OnError=/home/alex/servers/yjp-2016.02/bin/../bin/yjp.sh -on_error %p
  -jar
  /home/alex/servers/yjp-2016.02/bin/../lib/yjp.jar
16.02-30 0.037: Executable: /home/alex/servers/yjp-2016.02/jre64/bin/java
16.02-30 0.037: uname:
  sysname: Linux
  release: 3.13.0-83-generic
  version: #127-Ubuntu SMP Fri Mar 11 00:25:37 UTC 2016
  machine: x86_64


When I disable the setting in ini file and set it directly yjp.sh to 12GB when I try to open the snapshot I still get an out of memory error for a 4.9GB snapshot.
java.lang.OutOfMemoryError: Java heap space

It does capture a heap dump but I don't think I can attach them here.
aedwards
 
Posts: 6
Joined: Mon Mar 14, 2016 5:33 pm

Re: Opening Large snapshots

Postby Anton Katilin » Thu Mar 17, 2016 3:25 pm

If you still get the "java.lang.OutOfMemoryError: Java heap space" please set the heap size even bigger.
Anton Katilin
 
Posts: 5576
Joined: Wed Aug 11, 2004 8:37 am

Re: Opening Large snapshots

Postby aedwards » Thu Mar 17, 2016 4:30 pm

I understand that but that's not what I'm after.

I am interested to know what the relationship between heap size and snapshot size is.

Right now I have a problem with a 4.9GB snapshot and a 14GB heap (I tried a bit more) at this point I am going to have to pay for an aws slave with 32GB/64GB of ram just to try and open it because I don't have any hardware with more than 16GB of ram this is not a practical solution.

Assuming that all my snapshots were captured using the following settings =tracing how much heap would I need to open snapshots of these sizes:
    500Mb
    1GB
    2GB
aedwards
 
Posts: 6
Joined: Mon Mar 14, 2016 5:33 pm

Re: Opening Large snapshots

Postby Anton Katilin » Fri Mar 18, 2016 7:17 am

Sorry, there is no direct formula. Snapshot file on disk has its own format, it does not match the Java heap format and their sizes differ. How much memory it takes to load a snapshot does not directly depend on snapshot size on disk, but rather on how many objects it contains and how they refer each other. A 5 GB snapshot file may for example contain only 10 K objects size of some of those is huge (big arrays) or 100 M small objects. Opening the latter would take much more time and memory.

Since you mention "=tracing". Are you capturing a memory snapshot of an application that was running with CPU tracing?

Please note that heap size and CPU profiling data size have nothing in common. If your snapshot size is huge because it contains a lot of CPU tracing, please ensure adaptive tracing in on or consider using CPU sampling instead.

You can check the tracing data size by exporting a performance snapshot from Open snapshot dialog (Ctrl+O).
Anton Katilin
 
Posts: 5576
Joined: Wed Aug 11, 2004 8:37 am

Re: Opening Large snapshots

Postby Anton Katilin » Fri Mar 18, 2016 7:23 am

Anyway, as a rough estimation, you may consider that in some cases the top (sic!) memory needed to load a memory snapshot may be up to 3x of the snapshot size.

As a workaround please try setting the property -Dyjp.dominators=false as suggested and explained here: viewtopic.php?t=4420#p15658

You may also provide the profiler log file in full so we'll check what causes the out of memory in particular. I hope this may allow us to give you more explicit suggestions.
Anton Katilin
 
Posts: 5576
Joined: Wed Aug 11, 2004 8:37 am

Re: Opening Large snapshots

Postby aedwards » Fri Mar 18, 2016 3:07 pm

Thanks for the explanation, its helpful to know opening snapshots is more related to my objects than its size. The application i'm profiling makes a lot of objects!

What I actually wanted to do was to start up my app run it with some load for about 30 min to warm up the jit then capture about 5 min of profiling. I found I couldn't do this because after 30 min attaching the profiler was to intensive and caused things to fail.

I think I read somewhere that I could start up the application with the instrumentation already loaded but just attach the UI later in a lightweight way, I might try using the delay command line option if I have no success otherwise.
aedwards
 
Posts: 6
Joined: Mon Mar 14, 2016 5:33 pm

Re: Opening Large snapshots

Postby Anton Katilin » Fri Mar 18, 2016 4:00 pm

Yes, you may start CPU profiling on demand at a later time point. It is almost never necessary to run CPU tracing or sampling right from the start.
Anton Katilin
 
Posts: 5576
Joined: Wed Aug 11, 2004 8:37 am


Return to Java Profiler

Who is online

Users browsing this forum: No registered users and 6 guests

cron