Fully featured low overhead profiler for Java EE and Java SE platforms.
Ultimate profiling and monitoring solution for Gradle, Maven, Ant, JUnit and TestNG.
Easy to use performance and memory profiler for .NET framework.

700MB VM size vs. 80MB CLR heap size

Moderators: Vladimir Kondratyev, Anton Katilin

700MB VM size vs. 80MB CLR heap size

Postby jexpert » Tue Apr 24, 2012 1:26 pm

In my application I experience a problem, that the process memory hogs up huge amounts of memory. I tried to tackle down this problem with YourKit.NET profiler and finally optimized a few minor issues in our code.

Now we still face usage patterns of 700MB+ vs. small heaps like 80MB or less. I asked for potential reasons on stackoverflow and got the hin that this seems to origin from unmanaged memory allocations by graphics, which is reasonable, because we heavily create PDFs and charts.

Which is the preferred way to tackle the root cause for this issue down with YourKit .NET profiler?

Thanks
- Ben
jexpert
 
Posts: 8
Joined: Tue Apr 24, 2012 1:20 pm

Re: 700MB VM size vs. 80MB CLR heap size

Postby Anton Katilin » Tue Apr 24, 2012 9:53 pm

Hello Ben

The profiler can track only allocations in CLR heap, it does not track allocations in unmanaged code (C/C++ heap).
However, VM size minus the heap size can be used as a hint on how much unmanaged memory is used.

You can try to use the VM size telemetry graph to locate the problem. Find the time point(s) where the VM usage graph quickly grows. Use Stack traces and/or CPU usage estimation around those points. Resulting stack traces or call trees will tell what the application was doing then, and whether the suspected PDF and chart-related calls are indeed involved.

Best regards,
Anton
Anton Katilin
 
Posts: 5574
Joined: Wed Aug 11, 2004 8:37 am

Tracing large .NET VM size vs. small CLR heap size

Postby jexpert » Wed Apr 25, 2012 8:49 am

Hej Anton!

Thank you for your kind reply! I'm a loyal user and a in-house evangelist for Yourkit Java Profiler. Therefore I expected YourKit.NET Profiler would be the same all-you-need solution for performance issues in our .NET projects as in our Java projects and I was a bit disappointed to find it is not: Trying to correlate memory allocations/code by time is a bit too fuzzy and not really helpful enough. To be fair it must be said, that most competitor products in the .NET range seem to suffer the same blind spot as well. Sadly in my case this blind spot exactly is the only and really big performance issues/problem in my project.

As a pointer for others who might stumble upon this: For tracing down leaks in unmanaged code there is a free Microsoft solution called DebugDiag. This a tool specially to spot memory leaks in the unamanged parts of a .NET application. But don't expect to much : The best pointer you'll get from there is a "stack trace" like:

ntdll.dll is responsible for 270.95 KBytes worth of outstanding allocations. The following are the top 2 memory consuming functions:
ntdll!RtlpDphNormalHeapAllocate+1d: 263.78 KBytes worth of outstanding allocations.
ntdll!RtlCreateHeap+5fc: 6.00 KBytes worth of outstanding allocations.


As a introductional article I found http://www.dotnetfunda.com/articles/art ... leaks.aspx to be very helpful.

Hope this helps other, too!
- Ben
jexpert
 
Posts: 8
Joined: Tue Apr 24, 2012 1:20 pm

Re: 700MB VM size vs. 80MB CLR heap size

Postby Anton Katilin » Wed Apr 25, 2012 2:34 pm

Hello Ben

Trying to correlate memory allocations/code by time is a bit too fuzzy and not really helpful enough.

Could you please provide the snapshot. It's curious to see how quickly the usage changes.

ftp://upload.yourkit.com/pub/
user ftp
empty password

And thank you for the links.

Best regards,
Anton
Anton Katilin
 
Posts: 5574
Joined: Wed Aug 11, 2004 8:37 am

Re: 700MB VM size vs. 80MB CLR heap size

Postby jexpert » Thu Apr 26, 2012 9:20 am

Hej Anton!
I uploaded an example snapshot. At 1min and 19min the app is idling in the start screen. The problematic issues is the huge gap between CLR and VM (185MB vs. 530MB in this case) which skyrockets in production and fails due to the 32bit address space limitation.

The other issue is the leak which is visible after 19m:30s. Obviously after manually triggering GC some finalize Method seems to get release most of the unmanaged memory as well in the end now, but I'm not sure if the remaining 140MB VM vs. 18M CLR heap size indicates a still open leak.

So it seems I was able to soften the issue but still have no exact picture about where those excess 300MB+ gets eaten.
jexpert
 
Posts: 8
Joined: Tue Apr 24, 2012 1:20 pm

Re: 700MB VM size vs. 80MB CLR heap size

Postby Anton Katilin » Thu Apr 26, 2012 5:58 pm

Thank you for the snapshot.
It is indeed hard to guess the problematic code basing on the memory usage graph in this case.

Do you have experience of running this application with any C/C++ native memory profiler?
Anton Katilin
 
Posts: 5574
Joined: Wed Aug 11, 2004 8:37 am

Re: 700MB VM size vs. 80MB CLR heap size

Postby jexpert » Fri Apr 27, 2012 8:55 am

Only the mentioned DebugDiag if you can call it so. DebugDiag at least pointed me to a DLL function in a similar profiling session:
GdiPlus.dll is responsible for 400,80 MBytes worth of outstanding allocations.
The following are the top 2 memory consuming functions:
GdiPlus!GdipSaveImageToStream+56d1: 301,45 MBytes worth of outstanding allocations.
GdiPlus+6425: 99,35 MBytes worth of outstanding allocations.


Nevertheless DebugDiag did not gave me full stacktraces including CLR symbold (though it should). So i guessed I miss something in the setup of it.

For now I tried to tackle the memory consumption of our generated images and further watch how the problem persists.

Thank you for having taken a look on my snapshot.
- Ben
jexpert
 
Posts: 8
Joined: Tue Apr 24, 2012 1:20 pm


Return to .NET Profiler

Who is online

Users browsing this forum: No registered users and 2 guests