[#634] JVM Crash - WebLogic - JDK141_05 - Win2000AS

[vgritsenko]

Each time 'record memory allocations' is turned on, JVM consistently crashes in 5 min or less. JVM parameters are:

-server -Xrunyjpagent Xms1024m -Xmx1024m -XX:PermSize=256m -XX:MaxPermSize=256m -XX:NewRatio=2 -verbose:gc

Default 'NewRatio' parameter causes crashes as well. Crash data below. Any ideas / advice?

Thanks,
Vadim


An unexpected exception has been detected in native code outside the VM.
Unexpected Signal : unknown exception code occurred at PC=0x7C59BC3F
Function=RaiseException+0x56
Library=C:\WINNT\system32\KERNEL32.dll

Current Java thread:
at java.lang.String.toUpperCase(String.java:2083)
at java.lang.String.toUpperCase(String.java:2095)
at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:3492)
at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2585)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:197)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:170)

Dynamic libraries:
0x00400000 - 0x00406000 C:\bea\jdk141_05\bin\java-dss.exe
0x77F80000 - 0x77FFC000 C:\WINNT\system32\ntdll.dll
0x7C2D0000 - 0x7C335000 C:\WINNT\system32\ADVAPI32.dll
0x7C570000 - 0x7C623000 C:\WINNT\system32\KERNEL32.dll
0x77D30000 - 0x77DA8000 C:\WINNT\system32\RPCRT4.dll
0x78000000 - 0x78045000 C:\WINNT\system32\MSVCRT.dll
0x6D470000 - 0x6D6E1000 C:\bea\jdk141_05\jre\bin\server\jvm.dll
0x77E10000 - 0x77E79000 C:\WINNT\system32\USER32.dll
0x77F40000 - 0x77F7F000 C:\WINNT\system32\GDI32.dll
0x77570000 - 0x775A0000 C:\WINNT\system32\WINMM.dll
0x6D1E0000 - 0x6D1E7000 C:\bea\jdk141_05\jre\bin\hpi.dll
0x6D310000 - 0x6D31E000 C:\bea\jdk141_05\jre\bin\verify.dll
0x6D220000 - 0x6D239000 C:\bea\jdk141_05\jre\bin\java.dll
0x6D330000 - 0x6D33D000 C:\bea\jdk141_05\jre\bin\zip.dll
0x53060000 - 0x530E5000 C:\Progra~1\YOURKI~1.5EA\bin\win32\yjpagent.dll
0x75030000 - 0x75044000 C:\WINNT\system32\WS2_32.dll
0x75020000 - 0x75028000 C:\WINNT\system32\WS2HELP.DLL
0x6D2E0000 - 0x6D2EE000 C:\bea\jdk141_05\jre\bin\net.dll
0x75050000 - 0x75058000 C:\WINNT\system32\WSOCK32.dll
0x74FD0000 - 0x74FEE000 C:\WINNT\system32\msafd.dll
0x75010000 - 0x75017000 C:\WINNT\System32\wshtcpip.dll
0x782C0000 - 0x782CC000 C:\WINNT\System32\rnr20.dll
0x77980000 - 0x779A4000 C:\WINNT\system32\DNSAPI.DLL
0x77340000 - 0x77353000 C:\WINNT\system32\iphlpapi.dll
0x77520000 - 0x77525000 C:\WINNT\system32\ICMP.DLL
0x77320000 - 0x77337000 C:\WINNT\system32\MPRAPI.DLL
0x75150000 - 0x75160000 C:\WINNT\system32\SAMLIB.DLL
0x7CDC0000 - 0x7CE13000 C:\WINNT\system32\NETAPI32.DLL
0x751C0000 - 0x751C6000 C:\WINNT\system32\NETRAP.dll
0x77BF0000 - 0x77C01000 C:\WINNT\system32\NTDSAPI.dll
0x77950000 - 0x7797B000 C:\WINNT\system32\WLDAP32.DLL
0x7C340000 - 0x7C34F000 C:\WINNT\system32\SECUR32.DLL
0x7CE20000 - 0x7CF21000 C:\WINNT\system32\OLE32.DLL
0x779B0000 - 0x77A4B000 C:\WINNT\system32\OLEAUT32.DLL
0x773B0000 - 0x773DF000 C:\WINNT\system32\ACTIVEDS.DLL
0x77380000 - 0x773A3000 C:\WINNT\system32\ADSLDPC.DLL
0x77830000 - 0x7783E000 C:\WINNT\system32\RTUTILS.DLL
0x77880000 - 0x7790E000 C:\WINNT\system32\SETUPAPI.DLL
0x7C0F0000 - 0x7C154000 C:\WINNT\system32\USERENV.DLL
0x774E0000 - 0x77514000 C:\WINNT\system32\RASAPI32.DLL
0x774C0000 - 0x774D1000 C:\WINNT\system32\rasman.dll
0x77530000 - 0x77552000 C:\WINNT\system32\TAPI32.dll
0x71710000 - 0x71794000 C:\WINNT\system32\COMCTL32.DLL
0x70A70000 - 0x70AD6000 C:\WINNT\system32\SHLWAPI.DLL
0x77360000 - 0x77379000 C:\WINNT\system32\DHCPCSVC.DLL
0x777E0000 - 0x777E8000 C:\WINNT\System32\winrnr.dll
0x777F0000 - 0x777F5000 C:\WINNT\system32\rasadhlp.dll
0x586E0000 - 0x586E5000 C:\bea\weblogic81\server\bin\stackdump.dll
0x586F0000 - 0x586F5000 C:\bea\weblogic81\server\bin\wlntio.dll
0x6D2F0000 - 0x6D2F8000 C:\bea\jdk141_05\jre\bin\nio.dll
0x6D200000 - 0x6D206000 C:\bea\jdk141_05\jre\bin\ioser12.dll
0x59C90000 - 0x59C98000 C:\Entrust\getAccess\shared\getAccLic.dll
0x59CA0000 - 0x59CA7000 C:\Entrust\getAccess\shared\Kencrypt.dll
0x59CB0000 - 0x59CB6000 C:\Entrust\getAccess\shared\JKencrypt.dll
0x6D300000 - 0x6D305000 C:\bea\jdk141_05\jre\bin\rmi.dll
0x7CA00000 - 0x7CA23000 C:\WINNT\system32\rsaenh.dll
0x7C740000 - 0x7C7CC000 C:\WINNT\system32\CRYPT32.dll
0x77430000 - 0x77441000 C:\WINNT\system32\MSASN1.dll
0x77920000 - 0x77943000 C:\WINNT\system32\imagehlp.dll
0x72A00000 - 0x72A2D000 C:\WINNT\system32\DBGHELP.dll
0x60450000 - 0x6045B000 C:\WINNT\system32\PSAPI.DLL

Local Time = Wed Aug 03 19:25:26 2005
Elapsed Time = 760
#
# The exception above was detected in native code outside the VM
#
# Java VM: Java HotSpot(TM) Server VM (1.4.1_05-b01 mixed mode)
#

[Anton Katilin]

Hi,

The profiler agent code does not explicitly present in the dump, so this crash is probably caused by a bug in the JVM. I see the Java version is 1.4.1_05-b01, which is quite old, and there were a lot of bug fixes in the 1.4 family JVM since 1.4.1. Is it possibly to try to run the server with latest Java 1.4.2_08 ?

[vgritsenko]

Just tried 1.4.2_08, crashed in around 5 minutes after started allocations recording. Anything else I could try?

Thanks,
Vadim


An unexpected exception has been detected in native code outside the VM.
Unexpected Signal : unknown exception code (0xe06d7363) occurred at PC=0x7C59BC3F
Function=RaiseException+0x56
Library=C:\WINNT\system32\KERNEL32.dll

Current Java thread:
at oracle.jdbc.ttc7.TTCItem.unmarshal(TTCItem.java:416)
at oracle.jdbc.ttc7.NonPlsqlTTCColumn.unmarshal(NonPlsqlTTCColumn.java:155)
at oracle.jdbc.ttc7.NonPlsqlTTCDataSet.unmarshalRow(NonPlsqlTTCDataSet.java:134)
at oracle.jdbc.ttc7.TTIrxd.unmarshal(TTIrxd.java:140)
at oracle.jdbc.ttc7.Oall7.receive(Oall7.java:512)
at oracle.jdbc.ttc7.TTC7Protocol.doOall7(TTC7Protocol.java:1983)
at oracle.jdbc.ttc7.TTC7Protocol.fetch(TTC7Protocol.java:1250)
- locked <0x17a72180> (a oracle.jdbc.ttc7.TTC7Protocol)
at oracle.jdbc.driver.OracleResultSetImpl.next(OracleResultSetImpl.java:295)
- locked <0x2aa41a70> (a oracle.jdbc.driver.OracleResultSetImpl)
at weblogic.jdbc.wrapper.ResultSet_oracle_jdbc_driver_OracleResultSetImpl.next(Unknown Source)
at <censored - application call stack>
at java.lang.Thread.run(Thread.java:534)

Dynamic libraries:
0x00400000 - 0x0040B000 C:\Java\j2sdk1.4.2_08\bin\java-dss.exe
0x77F80000 - 0x77FFC000 C:\WINNT\system32\ntdll.dll
0x7C2D0000 - 0x7C335000 C:\WINNT\system32\ADVAPI32.dll
0x7C570000 - 0x7C623000 C:\WINNT\system32\KERNEL32.dll
0x77D30000 - 0x77DA8000 C:\WINNT\system32\RPCRT4.dll
0x78000000 - 0x78045000 C:\WINNT\system32\MSVCRT.dll
0x08000000 - 0x082AE000 C:\Java\j2sdk1.4.2_08\jre\bin\server\jvm.dll
0x77E10000 - 0x77E79000 C:\WINNT\system32\USER32.dll
0x77F40000 - 0x77F7F000 C:\WINNT\system32\GDI32.dll
0x77570000 - 0x775A0000 C:\WINNT\system32\WINMM.dll
0x10000000 - 0x10007000 C:\Java\j2sdk1.4.2_08\jre\bin\hpi.dll
0x008F0000 - 0x008FE000 C:\Java\j2sdk1.4.2_08\jre\bin\verify.dll
0x00900000 - 0x00919000 C:\Java\j2sdk1.4.2_08\jre\bin\java.dll
0x00920000 - 0x0092E000 C:\Java\j2sdk1.4.2_08\jre\bin\zip.dll
0x605C0000 - 0x60645000 C:\Progra~1\YOURKI~1.5EA\bin\win32\yjpagent.dll
0x75030000 - 0x75044000 C:\WINNT\system32\WS2_32.dll
0x75020000 - 0x75028000 C:\WINNT\system32\WS2HELP.DLL
0x60E30000 - 0x60E3F000 C:\Java\j2sdk1.4.2_08\jre\bin\net.dll
0x74FD0000 - 0x74FEE000 C:\WINNT\system32\msafd.dll
0x75010000 - 0x75017000 C:\WINNT\System32\wshtcpip.dll
0x782C0000 - 0x782CC000 C:\WINNT\System32\rnr20.dll
0x77980000 - 0x779A4000 C:\WINNT\system32\DNSAPI.DLL
0x75050000 - 0x75058000 C:\WINNT\system32\WSOCK32.dll
0x77340000 - 0x77353000 C:\WINNT\system32\iphlpapi.dll
0x77520000 - 0x77525000 C:\WINNT\system32\ICMP.DLL
0x77320000 - 0x77337000 C:\WINNT\system32\MPRAPI.DLL
0x75150000 - 0x75160000 C:\WINNT\system32\SAMLIB.DLL
0x7CDC0000 - 0x7CE13000 C:\WINNT\system32\NETAPI32.DLL
0x751C0000 - 0x751C6000 C:\WINNT\system32\NETRAP.dll
0x77BF0000 - 0x77C01000 C:\WINNT\system32\NTDSAPI.dll
0x77950000 - 0x7797B000 C:\WINNT\system32\WLDAP32.DLL
0x7C340000 - 0x7C34F000 C:\WINNT\system32\SECUR32.DLL
0x7CE20000 - 0x7CF21000 C:\WINNT\system32\OLE32.DLL
0x779B0000 - 0x77A4B000 C:\WINNT\system32\OLEAUT32.DLL
0x773B0000 - 0x773DF000 C:\WINNT\system32\ACTIVEDS.DLL
0x77380000 - 0x773A3000 C:\WINNT\system32\ADSLDPC.DLL
0x77830000 - 0x7783E000 C:\WINNT\system32\RTUTILS.DLL
0x77880000 - 0x7790E000 C:\WINNT\system32\SETUPAPI.DLL
0x7C0F0000 - 0x7C154000 C:\WINNT\system32\USERENV.DLL
0x774E0000 - 0x77514000 C:\WINNT\system32\RASAPI32.DLL
0x774C0000 - 0x774D1000 C:\WINNT\system32\rasman.dll
0x77530000 - 0x77552000 C:\WINNT\system32\TAPI32.dll
0x71710000 - 0x71794000 C:\WINNT\system32\COMCTL32.DLL
0x70A70000 - 0x70AD6000 C:\WINNT\system32\SHLWAPI.DLL
0x77360000 - 0x77379000 C:\WINNT\system32\DHCPCSVC.DLL
0x777E0000 - 0x777E8000 C:\WINNT\System32\winrnr.dll
0x777F0000 - 0x777F5000 C:\WINNT\system32\rasadhlp.dll
0x65D20000 - 0x65D25000 C:\bea\weblogic81\server\bin\stackdump.dll
0x65D30000 - 0x65D35000 C:\bea\weblogic81\server\bin\wlntio.dll
0x66350000 - 0x66358000 C:\Java\j2sdk1.4.2_08\jre\bin\nio.dll
0x66420000 - 0x66426000 C:\Java\j2sdk1.4.2_08\jre\bin\ioser12.dll
0x69810000 - 0x69818000 C:\Entrust\getAccess\shared\getAccLic.dll
0x69820000 - 0x69827000 C:\Entrust\getAccess\shared\Kencrypt.dll
0x69830000 - 0x69836000 C:\Entrust\getAccess\shared\JKencrypt.dll
0x7CA00000 - 0x7CA23000 C:\WINNT\system32\rsaenh.dll
0x7C740000 - 0x7C7CC000 C:\WINNT\system32\CRYPT32.dll
0x77430000 - 0x77441000 C:\WINNT\system32\MSASN1.dll
0x6F160000 - 0x6F165000 C:\Java\j2sdk1.4.2_08\jre\bin\rmi.dll
0x77920000 - 0x77943000 C:\WINNT\system32\imagehlp.dll
0x72A00000 - 0x72A2D000 C:\WINNT\system32\DBGHELP.dll
0x0EFE0000 - 0x0EFEB000 C:\WINNT\system32\PSAPI.DLL

Heap at VM Abort:
Heap
def new generation total 104832K, used 93184K [0x10010000, 0x171d0000, 0x171d0000)
eden space 93184K, 100% used [0x10010000, 0x15b10000, 0x15b10000)
from space 11648K, 0% used [0x16670000, 0x16670000, 0x171d0000)
to space 11648K, 0% used [0x15b10000, 0x15b10000, 0x16670000)
tenured generation total 932096K, used 328060K [0x171d0000, 0x50010000, 0x50010000)
the space 932096K, 35% used [0x171d0000, 0x2b22f010, 0x2b22f200, 0x50010000)
compacting perm gen total 262144K, used 64550K [0x50010000, 0x60010000, 0x60010000)
the space 262144K, 24% used [0x50010000, 0x53f19bc0, 0x53f19c00, 0x60010000)

Local Time = Fri Aug 05 10:34:19 2005
Elapsed Time = 893
#
# The exception above was detected in native code outside the VM
#
# Java VM: Java HotSpot(TM) Server VM (1.4.2_08-b03 mixed mode)
#

[Anton Katilin]

Hi,

The new trace either doesn't contain any code of profiler agent.

Current Java thread:
at oracle.jdbc.ttc7.TTCItem.unmarshal(TTCItem.java:416)
at oracle.jdbc.ttc7.NonPlsqlTTCColumn.unmarshal(NonPlsqlTTCColumn.java:155)
at oracle.jdbc.ttc7.NonPlsqlTTCDataSet.unmarshalRow(NonPlsqlTTCDataSet.java:134)
at oracle.jdbc.ttc7.TTIrxd.unmarshal(TTIrxd.java:140)

Does the crash always happen in this code (Oracle JDBC stuff), or there are other cases?

[vgritsenko]

No, it seems it crashes every time in different place, probably (my guess) when trying to allocate memory. I can capture more of those crash files if it is of any use.

Have to notice that no crashes happen if I don't start allocation recording. Memory consumption is also moderate, heap stays within 150 to 200 Mb out of 1Gb allocated.

Vadim

[Anton Katilin]

When memory allocation is turned on in Java 1.4/JVMPI, an event model is used. I.e. the JVM issues an event each time an object is created; then the event is processed by the profiler agent. Since there's no code of the agent in the crash dump, I can suggest that something wrong happens inside the JVM when it tries to issue that events.

You can try Java 5.0/JVMTI version of the agent, that uses a different approach - byte code instrumentation. It is not event-based, so the JVM should behave differently.

[vgritsenko]

You can try Java 5.0/JVMTI version of the agent, that uses a different approach - byte code instrumentation. It is not event-based, so the JVM should behave differently.

Unfortunately, I can't:

java.io..: javax.management.MBeanAttributeInfo; local class incompatible: stream classdesc serialVersionUID = 7043855487133450673, local class serialVersionUID = 8644704819898
565848

Seems like WebLogic 8.1 violently objects against this one... It has bunch of serialized .mbi files in it

Vadim