I'm running 3.1 EAP build 443. The UI is on a Windows XP box, and the application being profiled is on a solaris 9/sparc machine. The app being profiled is a web app running inside Tomcat 4.1.29. The JVM on the Windows XP machine is 1.4.2_05, the JVM on the server is 1.4.2_04. The server VM isn't being run with any wierd flags (other than -Xrunyjpagent).
I've been able to get one memory snapshot off the server, but the second 'capture snapshot' has caused the UI to hang at "63%" complete, and the JVM being profiled is now unresponsive (tomcat isn't responding to HTTP requests). It's been hung for maybe 20 minutes as I type this.
The directory where the snapshots are being saved shows the first snapshot at ~36mb, and the second (incomplete) snapshot at 8.4mb. the timestamp on the second snapshot indicates that the file hasn't been changed for awhile.
Doing a 'netstat' on both machines shows that there's an active connection on port 10000, but there's no traffic on it.
The server was doing work while the second snapshot was being taken, and the server JVM reported a java.lang.OutOfMemoryError in a HTTP thread while the second snapshot was being taken - this is probably the root cause of the cause of the hang.
The server JVM shows some minor CPU usage - it's not idle, but it's using around 0.25% CPU.
If I'm able to reproduce this, anything I should look for? Would you like the two snapshot files to look at?
Client Hang - build 443
-
Vladimir Kondratyev
- Posts: 1626
- Joined: Tue Aug 10, 2004 7:52 pm
-
pclark
- Posts: 10
- Joined: Tue Oct 19, 2004 8:44 pm
OK - I got it to do bad things again, in a slightly different scenario. This is the same hardware/software setup - the JVM being profiled is a 1.4.2_04 JVM running on solaris 9 on sparc. The profiler UI is running on windows XP in a 1.4.2_05 JVM.
In this case, I was triggering a GC cycle rather than trying to get a heap dump. I think that I told the YourKit profiler to trigger a GC cycle right as the out of memory error was occurring in another thread.
Here's a stack trace from Tomcat:
2004-10-26 19:31:17 StandardWrapperValve[action]: Servlet.service() for servlet action threw exception
javax.servlet.ServletException: Servlet execution threw an exception
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:256)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at org.apache.catalina.core.StandardContext.invoke(StandardContext.java:2417)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:180)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at org.apache.catalina.valves.ErrorDispatcherValve.invoke(ErrorDispatcherValve.java:171)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:172)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:174)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at org.apache.coyote.tomcat4.CoyoteAdapter.service(CoyoteAdapter.java:193)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:781)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:549)
at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:589)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:666)
at java.lang.Thread.run(Thread.java:534)
----- Root Cause -----
java.lang.OutOfMemoryError
----------------------------------
Here's what came out on the console (with some context ahead):
[YourKit Java Profiler 3.1 EAP #443] Request from 10.0.0.4: Memory snapshot is saved to /Users/einsof/tomcat-solaris(3).memory
[DEBUG] 18:44:58,821 [http8101-Processor3] (CPNDataBase.java:579) - Found a matching field: clinicDescription
[YourKit Java Profiler 3.1 EAP #443] Request from 10.0.0.4: Garbage collected. Size of objects in heap: before=31766168 bytes, after=31766168 bytes
2004-10-26 18:44:07,215 [DEBUG] [http8101-Processor8] (TilesRequestProcessor.java:304) - processForwardConfig(/login.do, false)
[YourKit Java Profiler 3.1 EAP #443] Request from 10.0.0.4: Garbage collected. Size of objects in heap: before=32597336 bytes, after=32597336 bytes
[DEBUG] 19:29:38,923 [http8101-Processor3] (CPNDataBase.java:595) - value for this field: null
[DEBUG] 19:31:05,202 [http8101-Processor3] (CPNDataBase.java:569) - Trying to find a matching field for column[24]: primaryCare
[DEBUG] 19:31:15,575 [http8101-Processor3] (CPNDataBase.java:579) - Found a matching field: primaryCare
[DEBUG] 19:31:15,584 [http8101-Processor3] (CPNDataBase.java:595) - value for this field: 0
java.lang.OutOfMemoryError[YourKit Java Profiler 3.1 EAP #443]java.lang.OutOfMemoryErrorjava.lang.OutOfMemoryErrorjava.lang.OutOfMemoryErrorjava.lang.OutOfMemoryError
Oct 26, 2004 7:31:17 PM org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler processConnection
SEVERE: Error reading request, ignored
java.lang.OutOfMemoryError
Oct 26, 2004 7:31:17 PM org.apache.tomcat.util.threads.ThreadPool$ControlRunnable run
SEVERE: Caught exception executing org.apache.tomcat.util.net.TcpWorkerThread@543d41, terminating thread
java.lang.OutOfMemoryError
Oct 26, 2004 7:31:17 PM org.apache.coyote.http11.Http11Processor process
SEVERE: Error processing request
java.lang.OutOfMemoryError
Oct 26, 2004 7:31:17 PM org.apache.tomcat.util.net.TcpWorkerThread runIt
SEVERE: Unexpected error
java.lang.OutOfMemoryError
Oct 26, 2004 7:31:17 PM org.apache.tomcat.util.net.TcpWorkerThread runIt
SEVERE: Unexpected error
java.lang.OutOfMemoryError
Oct 26, 2004 7:31:17 PM org.apache.coyote.http11.Http11Processor process
SEVERE: Error processing request
java.lang.OutOfMemoryError
----------------------------------
The JVM is still running, although it's unresponsive. The profiler UI is running on a Windows XP box, and when I try to perform an action from the Control Panel after the OutOfMemoryError, I first got a 'broken connection' or 'broken socket' error (I forget the exact wording), and I now get a message saying "There's no application running at 10.0.0.X with profiler agent configured to listen on port 10000".
I think the test case to explore here is that the OutOfMemoryError is being caused in a different thread from the one handling the YourKit profiling, but is causing problems for the YourKit thread.
In this case, I was triggering a GC cycle rather than trying to get a heap dump. I think that I told the YourKit profiler to trigger a GC cycle right as the out of memory error was occurring in another thread.
Here's a stack trace from Tomcat:
2004-10-26 19:31:17 StandardWrapperValve[action]: Servlet.service() for servlet action threw exception
javax.servlet.ServletException: Servlet execution threw an exception
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:256)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at org.apache.catalina.core.StandardContext.invoke(StandardContext.java:2417)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:180)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at org.apache.catalina.valves.ErrorDispatcherValve.invoke(ErrorDispatcherValve.java:171)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:172)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:174)
at org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at org.apache.coyote.tomcat4.CoyoteAdapter.service(CoyoteAdapter.java:193)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:781)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:549)
at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:589)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:666)
at java.lang.Thread.run(Thread.java:534)
----- Root Cause -----
java.lang.OutOfMemoryError
----------------------------------
Here's what came out on the console (with some context ahead):
[YourKit Java Profiler 3.1 EAP #443] Request from 10.0.0.4: Memory snapshot is saved to /Users/einsof/tomcat-solaris(3).memory
[DEBUG] 18:44:58,821 [http8101-Processor3] (CPNDataBase.java:579) - Found a matching field: clinicDescription
[YourKit Java Profiler 3.1 EAP #443] Request from 10.0.0.4: Garbage collected. Size of objects in heap: before=31766168 bytes, after=31766168 bytes
2004-10-26 18:44:07,215 [DEBUG] [http8101-Processor8] (TilesRequestProcessor.java:304) - processForwardConfig(/login.do, false)
[YourKit Java Profiler 3.1 EAP #443] Request from 10.0.0.4: Garbage collected. Size of objects in heap: before=32597336 bytes, after=32597336 bytes
[DEBUG] 19:29:38,923 [http8101-Processor3] (CPNDataBase.java:595) - value for this field: null
[DEBUG] 19:31:05,202 [http8101-Processor3] (CPNDataBase.java:569) - Trying to find a matching field for column[24]: primaryCare
[DEBUG] 19:31:15,575 [http8101-Processor3] (CPNDataBase.java:579) - Found a matching field: primaryCare
[DEBUG] 19:31:15,584 [http8101-Processor3] (CPNDataBase.java:595) - value for this field: 0
java.lang.OutOfMemoryError[YourKit Java Profiler 3.1 EAP #443]java.lang.OutOfMemoryErrorjava.lang.OutOfMemoryErrorjava.lang.OutOfMemoryErrorjava.lang.OutOfMemoryError
Oct 26, 2004 7:31:17 PM org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler processConnection
SEVERE: Error reading request, ignored
java.lang.OutOfMemoryError
Oct 26, 2004 7:31:17 PM org.apache.tomcat.util.threads.ThreadPool$ControlRunnable run
SEVERE: Caught exception executing org.apache.tomcat.util.net.TcpWorkerThread@543d41, terminating thread
java.lang.OutOfMemoryError
Oct 26, 2004 7:31:17 PM org.apache.coyote.http11.Http11Processor process
SEVERE: Error processing request
java.lang.OutOfMemoryError
Oct 26, 2004 7:31:17 PM org.apache.tomcat.util.net.TcpWorkerThread runIt
SEVERE: Unexpected error
java.lang.OutOfMemoryError
Oct 26, 2004 7:31:17 PM org.apache.tomcat.util.net.TcpWorkerThread runIt
SEVERE: Unexpected error
java.lang.OutOfMemoryError
Oct 26, 2004 7:31:17 PM org.apache.coyote.http11.Http11Processor process
SEVERE: Error processing request
java.lang.OutOfMemoryError
----------------------------------
The JVM is still running, although it's unresponsive. The profiler UI is running on a Windows XP box, and when I try to perform an action from the Control Panel after the OutOfMemoryError, I first got a 'broken connection' or 'broken socket' error (I forget the exact wording), and I now get a message saying "There's no application running at 10.0.0.X with profiler agent configured to listen on port 10000".
I think the test case to explore here is that the OutOfMemoryError is being caused in a different thread from the one handling the YourKit profiling, but is causing problems for the YourKit thread.