Fully featured low overhead profiler for Java EE and Java SE platforms.
Ultimate profiling and monitoring solution for Gradle, Maven, Ant, JUnit and TestNG.
Easy to use performance and memory profiler for .NET framework.

Request: regex match filter for allocation recording

Moderators: Vladimir Kondratyev, Anton Katilin

Request: regex match filter for allocation recording

Postby ecosky » Fri Aug 24, 2012 5:44 am

Hi,

I have what I hope is a trivial feature request. I would like to be able to record all allocations of objects of a specific type. To make it more useful, I'd like to specify the type using a regex expression so I can occasionally do more complex filters on the type names being evaluated.

In my current situation, I have identified a certain type as being allocated far more often than I expected, but there is a lot of other activity in the program and I don't want to record allocation information for every type, I am only interested in this one type. This type is legitimately allocated quite often, but there is apparently at least one circumstance where it is allocated where I'm not expecting it. The number of objects I am dealing with are in the tens of thousands, which is why I would like to continue to use the allocation tree view to see where groups of these are allocated. If I record everything, I have to manually find the objects I'm actually interested in by drilling down a very complete tree of my program's execution where all allocations are logged - this is unworkable given the complexity of the app. I would much prefer to only have to drill down into code paths that I know have allocated the objects I am specifically interested in.

It seems that this would be a very simple filter to add to the existing allocation filters, hopefully this is actually the case.

Thanks for reading!
ecosky
 
Posts: 10
Joined: Fri Apr 20, 2012 5:54 pm

Re: Request: regex match filter for allocation recording

Postby ecosky » Fri Aug 24, 2012 6:48 am

Woops! I overlooked the Allocation call tree in the captured snapshot. This does give me the information I need. I still think it would be useful to filter allocation recording using a regex to speed things up and eliminate things I really don't care about at the moment, but this will at least get me the information I need right now.
ecosky
 
Posts: 10
Joined: Fri Apr 20, 2012 5:54 pm

Re: Request: regex match filter for allocation recording

Postby Anton Katilin » Fri Aug 24, 2012 7:10 am

Hello,

Thank you for the suggestion.

Could you please clarify what should the regex filter be applied to:
I would like to be able to record all allocations of objects of a specific type. To make it more useful, I'd like to specify the type using a regex expression so I can occasionally do more complex filters on the type names being evaluated.

the class name of the allocated object?
I would much prefer to only have to drill down into code paths that I know have allocated the objects I am specifically interested in.

...or the stack trace where the object is allocated?

there is apparently at least one circumstance where it is allocated where I'm not expecting it

Could you please describe the circumstance in detail.
Is it what happens inside a particular method, which is called from different places?
Or is it just a single subtree of the entire call tree?

If I record everything, I have to manually find the objects I'm actually interested in by drilling down a very complete tree of my program's execution where all allocations are logged

Which views do you use?
Depending on the answers to my question above, it may be possible to effectively use existing UI when working with the opened snapshot. For example, if you are interested in objects created in particular method, select that method in the allocation method list, then see the class list in a slave view below, select your objects of interest, then open them in a new tab with F4 (or popup menu). The same for a specific subtree but using the allocation call tree.
http://www.yourkit.com/docs/net70/help/ ... _alloc.jsp

If this doesn't help, could you please provide an example snapshot and more detailed explanations of which objects you are searching for.

It seems that this would be a very simple filter to add to the existing allocation filters

Unfortunately, it is not that easy.

Allocation recording uses a CLR event issued when objects are allocated. There is no built-in filtering capability neither for object type nor for call stack: the event is issued for all objects, if enabled. If we applied our own filtering in runtime (take each allocated object's class, match the name; match the stack), it would add overhead, perhaps significant. Profiling would run slower and collect less information at the same time - the worst thing to do.

So it makes sense to filter in UI instead,i.e. to record all object and provide UI means to easily retrieve interesting information on demand.

Best regards,
Anton
Anton Katilin
 
Posts: 5575
Joined: Wed Aug 11, 2004 8:37 am

Re: Request: regex match filter for allocation recording

Postby Anton Katilin » Fri Aug 24, 2012 7:20 am

P.S. Our posts have crossed.
Anton Katilin
 
Posts: 5575
Joined: Wed Aug 11, 2004 8:37 am

Re: Request: regex match filter for allocation recording

Postby ecosky » Fri Aug 24, 2012 7:57 am

Thanks for your response. I now understand the UI a little better and can get the information I need using existing views, so I'll spare the details on your questions and discuss the filtering itself which I don't think I explained very well before.

I am only referring to filtering allocation recording based on the class name of the allocated object. Everything else would remain the way it is. In the event handler when YouKit is notified of an allocation, there would be an event handler triggered along the lines of the OnAllocationEvent shown in this off-the-cuff pseudocode class:
Code: Select all
class FilterByType
{
RegEx includePattern;

public FilterByType(RegEx includePattern)
{
this.includePattern = includePattern;
}

// No idea what kind of signature the allocation event actually requires.
// Returns true if the allocation should be tracked.
public bool OnAllocationEvent(object allocatedObject)
{
retrun includePattern.IsMatch(allocatedObject.GetType().FullName);
}

}


That's pretty much all I'm suggesting. The rest of the app would continue to show what allocations did make it past the filters (this one and the existing size/frequency ones) the same way it does now.

As you pointed out (which I also noticed right after posting the first one) I can use the existing UI to get much of the information I need, however it does slow down the game quite a lot (periodic stalls approaching 2 seconds) to have it logging allocations for everything. I'm hoping this filter will actually speed it up a bit since it seems likely there is more work going on with tracking all the events than what this filter would cost in overhead when it is in use. I have no way of knowing though, so if you say it's not practical I'm happy to take your word on it.

This is not an essential feature but it seems like it would probably be helpful if in fact it did reduce the overhead of tracking allocations due to the limited number of types being logged, much the same way that recording every 10th or less can help. I'd expect a regex test would be pretty quick, and if it is not fast enough a simple wildcard would be almost as useful.

Thanks for reading,
ecosky
 
Posts: 10
Joined: Fri Apr 20, 2012 5:54 pm

Re: Request: regex match filter for allocation recording

Postby Anton Katilin » Fri Aug 24, 2012 9:34 am

Thank you for the explanation.

Indeed, such filtering won't help improving performance while we use the CLR's allocation event issued for all allocated objects.

However, things may change. We are currently working on code instrumentation feature which means an ability to insert profiler hooks into particular method calls. The hooks are (almost) arbitrary callbacks, which can access "this", method parameters, returned value etc.

The main intention is to enable collecting higher level profiling detail for monitored methods, such as executed SQL statements or URLs passed to a web application, etc.

But we probably can use this approach as an engine for objects allocation recording and CPU tracing, which now depend on CLR events. Unlike the CLR events which are issued for all object allocations or method hits and in general result in a significant overhead when enabled, instrumentation is performed explicitly, and this is where we can apply filters to decide which classes and methods to instrument. The work is in progress and we have not yet tested this instrumentation-based allocation recording approach, but there are good chances there will be a performance improvement with properly selected filters.
Anton Katilin
 
Posts: 5575
Joined: Wed Aug 11, 2004 8:37 am

Re: Request: regex match filter for allocation recording

Postby ecosky » Fri Aug 24, 2012 4:35 pm

Thanks for the explanation, that makes sense.

I have a related question - I am trying to discover what objects are being allocated within a certain time window. When I turn on allocation recording at the beginning of this time window, then capture snapshot at the end, I get the expected snapshot view of all the objects. I am not sure how or if I can limit the view to only those objects created during the capture window; it seems to only be able to show all object in the heap. With no apparent way to distinguish between those created before I began allocation recording from those created while allocation recording, I find it difficult to find the objects created during allocation recording.

Do you have some advice on how to find these objects more easily? For instance, I have in my application's capture window, memory tab, class list:
Name Objects ShallowSize Objects(+/-) Size(+/-)
System.Int32[] 9819 29824460 5300 20284320

When I take a snapshot, it will show a list of (for instance) 9819 objects, with no way to limit it to the objects allocated during the time that allocation recording was enabled. Is there a feature I am overlooking that allows the snapshot to limit display to objects created after starting of the allocation capture itself?

Thanks again for your help,
ecosky
 
Posts: 10
Joined: Fri Apr 20, 2012 5:54 pm

Re: Request: regex match filter for allocation recording

Postby Anton Katilin » Mon Aug 27, 2012 9:42 am

Please open the snapshot, tab Memory.
Click "Call tree (all threads together)" in the Allocations section (see the left stripe).
Select the <All threads node>.
Press F4 (View selected objects action - also available via a popup menu.)
You will have a new tab opened with the objects of interest.

Please note that allocation recording records each 10th object by default. To get all objects created in the time windows, please switch to recording each object.
Anton Katilin
 
Posts: 5575
Joined: Wed Aug 11, 2004 8:37 am


Return to .NET Profiler

Who is online

Users browsing this forum: No registered users and 5 guests