Trying to find performance issues in an I/O intensive data store application

Questions about YourKit Java Profiler
Post Reply
johannes
Posts: 4
Joined: Thu Apr 07, 2022 8:53 pm

Trying to find performance issues in an I/O intensive data store application

Post by johannes »

Hello,

I'm currently trying to find out, why my data store needs between 10 - 14 minutes instead of around 3 minutes on my Laptop for MongoDB to import a 3,8 Gb JSON file. I'm running MongoDB within a Docker Container. Currently I'm simply running a test to import this JSON file inside IntelliJ with Gradle. But still, as it doesn't have the additional network layer something is clearly wrong in my code. Sometimes one of the CPUs is around 100% (seen via `htop`), but YourKit always displays the CPU percentage way below 50%. Also I'm getting potential deadlocks displayed:

jdk.internal.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:506)
java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3464)
java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3435)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1623)
org.gradle.internal.remote.internal.hub.queue.EndPointQueue.take(EndPointQueue.java:49)
org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:403)
org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
java.lang.Thread.run(Thread.java:833)

jdk.internal.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:506)
java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3464)
java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3435)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1623)
org.gradle.internal.remote.internal.hub.queue.EndPointQueue.take(EndPointQueue.java:49)
org.gradle.internal.remote.internal.hub.MessageHub$ConnectionDispatch.run(MessageHub.java:322)
org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
java.lang.Thread.run(Thread.java:833)

jdk.internal.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkUntil(LockSupport.java:410)
java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1726)
java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1623)
java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)

I think they might be false positives and the first two seem to be from Gradle itself, but maybe the third one is already the major issue I'm trying to track down. In the flame graph I can't see anything special, but serializing records to their byte-serialization needs some time.

https://github.com/sirixdb/sirix/blob/3 ... t.java#L72

with the following file:
https://www.filemail.com/d/gafhsvnvepqdabn

the snapshot from YourKit:
https://www.filemail.com/d/bhyykjtytcbcykt

kind regards
Johannes
Anton Katilin
Posts: 6172
Joined: Wed Aug 11, 2004 8:37 am

Re: Trying to find performance issues in an I/O intensive data store application

Post by Anton Katilin »

Hello Johannes,

With the existing snapshot please try using the CPU usage estimation feature:
https://www.yourkit.com/docs/java/help/ ... mation.jsp

1. Right click the CPU usage graph and select from the popup menu the option "Scale graphs to" -> "Fit to Width" to avoid horizontal scrolling.
2. Select the whole time range.
3. Sort the below table by "Samples" column.
This will give you wall time estimation ( https://www.yourkit.com/docs/java/help/times.jsp ).

For more accurate and per-thread results, please consider running with CPU sampling ( https://www.yourkit.com/docs/java/help/sampling.jsp ) or asynchronous periodic sampling ( https://www.yourkit.com/docs/java/help/ ... mpling.jsp ) turned on.

Please ensure that wall time measurement is opted in sampling settings ( https://www.yourkit.com/docs/java/help/ ... ttings.jsp ).

Best regards,
Anton
Anton Katilin
Posts: 6172
Joined: Wed Aug 11, 2004 8:37 am

Re: Trying to find performance issues in an I/O intensive data store application

Post by Anton Katilin »

P.S.
Sometimes one of the CPUs is around 100% (seen via `htop`), but YourKit always displays the CPU percentage way below 50%.
The profiler always normalizes CPU usage to 100%, where 100% corresponds to full usage on each and every CPU. When one CPU usage is 100% and the other CPUs are idle, YourKit shows usage below 100%.
Post Reply