High-level questions about how sampling profiler works

[yang]

Hi, I have a few high-level questions about the internals of sample based profiling. This should allow me to better make sense and use of the results I'm seeing.

Currently, when the results show that a method is taking up n% of the CPU time, does that mean it's appearing in n% of all sampled stack traces? Or in n% of the samples?

How do you differentiate among stack traces of threads that are blocking, that are executing, and that are executable but not executing? How do multiple CPUs factor in?

How does the sampling profiler work - does it use

- Thread.getAllStackTraces(),
- ThreadMXBean (getAllThreadIds(), getThreadInfo()), or
- JVMTI?

If using JVMTI, do you use GetStackTrace() or AsyncGetCallTrace()?

Thanks in advance for any answers!

[Anton Katilin]

Hello yang,

YourKit Java Profiler is a proprietary, closed-source software, so please don't expect we'll share all the secrets

How do you differentiate among stack traces of threads that are blocking, that are executing, and that are executable but not executing? How do multiple CPUs factor in?

We utilize thread status information as reported by JVMTI. We use thread CPU time, and multiple CPUs are handled properly.

How does the sampling profiler work - does it use

- Thread.getAllStackTraces(),
- ThreadMXBean (getAllThreadIds(), getThreadInfo()), or
- JVMTI?

The profiler uses JVMTI.

If using JVMTI, do you use GetStackTrace() or AsyncGetCallTrace()?

AsyncGetStackTrace does not exist in final JVMTI specification, it only existed in its draft version. Please see detail in JVMTI specification change log:
http://java.sun.com/javase/6/docs/platf ... jvmti.html

19 Nov 2002 Added AsyncGetStackTrace.
22 Jan 2003 Fixed names to standardized naming convention. Removed AsyncGetStackTrace.

Best regards,
Anton

[yang]

One of the main reasons we ask is because for some reason in our *CPU* profiler results we see things like blocking IO syscalls and lock operations taking up time. Not as much time as they probably really are taking, but they make their way to the top of our chart in terms of time dominance, which is fishy.

If our understanding is correct, these (at least the synchronization operations) should only show up in the monitor profiling results and not the CPU profiling results. Indeed, attempts to profile minimal Java apps that just Thread.sleep() result in no samples. So why is the above phenomenon occurring? Does the CPU profiler in fact actually measure what's actively running? If you had many threads that are "schedulable" (not blocked) but only four hyperthreads (four cores if you will) - will these all get counted?

[Anton Katilin]

1. Methods which do not consume CPU will be presented in CPU profiling results (CPU sampling or CPU tracing) if they are in the customizable list of "wall time" methods: http://www.yourkit.com/docs/80/help/times.jsp

2. Sampling takes CPU time actually spent in a thread between samples and adds it the method which is at the top. If methods such as Object.wait() and Thread.sleep() are at the top, the time is ignored. Otherwise results would look confusing: methods which never supposed to consume CPU would get non zero CPU time.

If you need to measure time spent in wait() or in sleep(), you can add them to the wall time method list, as described above.

[plethora]

The AsyncGetCallTrace() API is not part of JVMTI, it's still supported and maintained.
See: [http://code.metager.de/source/xref/open ... /forte.cpp]

While it could be considered an internal API, perhaps it would be interesting to hack up YK integration?
The asynchronous nature of the API does promise more accurate sampling possibilities

[DoorDan]

THank you for clarifications, Anton! I was looking into issue and found it suddenly.

[plethora]

For those interested in the topic, some relevant links:

A proof-of-concept:
https://code.google.com/p/lightweight-java-profiler/

A discussion on the "mechanical sympathy" group:
https://groups.google.com/forum/#!topic ... fStzLvunnQ

[plethora]

For those willing to experiment:
https://github.com/RichardWarburton/honest-profiler