java.lang.VerifyError: Inconsistent stackmap frames

Questions about YourKit Java Profiler
Post Reply
wojtekD
Posts: 5
Joined: Wed Jul 01, 2020 6:40 am

java.lang.VerifyError: Inconsistent stackmap frames

Post by wojtekD »

Hi All,

I'm getting 'java.lang.VerifyError: Inconsistent stackmap frames at branch target', when trying to run my application with yourkit agent, using intellij plugin on dev environment as well as using agent on production.
Running without yourkit agent goes smoothly.
I only found very old topics related to similar error.
Any suggestions?

Running locally with:
macOS
jdk 12.0.2-open (but tested on various 11 and 12 versions)
YourKit Java Profiler 2019.8-b141

Code: Select all

020-06-23 10:04:29,301 [ERROR] [)-127.0.0.1] [ContextLoader            ] - Context initialization failed
org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name ...
(...)
Caused by: java.lang.VerifyError: Inconsistent stackmap frames at branch target 78
Exception Details:
  Location:
    com/[Path/to/my/Class]$$EnhancerBySpringCGLIB$$57ce6b5c.run()V @70: ifeq
  Reason:
    Current frame's stack size doesn't match stackmap.
  Current Frame:
    bci: @70
    flags: { }
    locals: { 'com/[Path/to/my/Class]$$EnhancerBySpringCGLIB$$57ce6b5c', integer, top, long, long_2nd }
    stack: { 'java/lang/Object', integer }
  Stackmap Frame:
    bci: @78
    flags: { }
    locals: { 'com/[Path/to/my/Class]$$EnhancerBySpringCGLIB$$57ce6b5c', integer, top, long, long_2nd }
    stack: { }
  Bytecode:
    0000000: 1302 3214 0233 b802 20c4 3700 0303 3c2a
    0000010: c102 1a99 000b 2ac0 021a b802 143c 2ab4
    0000020: 0033 59c7 000c 572a b800 372a b400 3359
    0000030: c600 292a b200 39b2 003b b200 3db9 0043
    0000040: 0500 2ac1 021a 9900 081b 01b8 0218 1302
    0000050: 32c4 1600 03b8 0224 b12a b700 312a c102
    0000060: 1a99 0008 1b01 b802 1813 0232 c416 0003
    0000070: b802 24b1 2ac1 021a 9900 0a59 4d1b 2cb8
    0000080: 0218 bf                                
  Exception Handler Table:
    bci [30, 116] => handler: 116
  Stackmap Table:
    append_frame(@30,Integer,Top,Long)
    same_locals_1_stack_item_frame(@47,Object[#63])
    same_frame(@78)
    same_locals_1_stack_item_frame(@89,Object[#63])
    same_frame(@105)
    same_locals_1_stack_item_frame(@116,Object[#433])
    same_locals_1_stack_item_frame(@130,Object[#433])

	at java.base/java.lang.Class.forName0(Native Method)
	at java.base/java.lang.Class.forName(Class.java:415)
	at org.springframework.cglib.core.ReflectUtils.defineClass(ReflectUtils.java:571)
	at org.springframework.cglib.core.AbstractClassGenerator.generate(AbstractClassGenerator.java:363)
	at org.springframework.cglib.proxy.Enhancer.generate(Enhancer.java:582)
	at org.springframework.cglib.core.AbstractClassGenerator$ClassLoaderData$3.apply(AbstractClassGenerator.java:110)
	at org.springframework.cglib.core.AbstractClassGenerator$ClassLoaderData$3.apply(AbstractClassGenerator.java:108)
	at org.springframework.cglib.core.internal.LoadingCache$2.call(LoadingCache.java:54)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at org.springframework.cglib.core.internal.LoadingCache.createEntry(LoadingCache.java:61)
Anton Katilin
Posts: 6172
Joined: Wed Aug 11, 2004 8:37 am

Re: java.lang.VerifyError: Inconsistent stackmap frames

Post by Anton Katilin »

Hi Wojtek

Is the problem reproducible with 2020.7 EAP?
https://www.yourkit.com/eap/

If it is, could you please contact [email protected] . We'll ask you to provide additional detail.

As a possible workaround, please try these agent startup options (see https://www.yourkit.com/docs/java/help/ ... ptions.jsp ):

1) "disableall"
2) If 1 helps, try "disablealloc,disabletracing,probe_disable=*"
3) If 2 helps, please try them separately or combinations of two.

Best regards,
Anton
wojtekD
Posts: 5
Joined: Wed Jul 01, 2020 6:40 am

Re: java.lang.VerifyError: Inconsistent stackmap frames

Post by wojtekD »

Hi Anton,

Thanks for quick response. I've been struggling with this for days.

The same problem occurs on 2020.7 EAP.

I was testing your workaround and the results are:
1. It works when run with 'disableall'
2. It works when running with 'disablealloc,disabletracing,probe_disable=*'
3. It doesn't work when running with 'disablealloc,disabletracing'
4. It works when running with 'probe_disable=*'

So 'probe_disable=*' is the solution.

Q1: Could you elaborate more on what it means for me? What functionality will be limited?
Q2: Can I turn it on later after proper app initialisation? The problem was occurring during my app initialisation. On my dev environment I was able to attach agent from YourKit App UI, when my application was already up and running without specifying any agent options.
Q3: Is this something that YourKit does not yet support on newer java versions?

Best Regards,
Wojtek
Anton Katilin
Posts: 6172
Joined: Wed Aug 11, 2004 8:37 am

Re: java.lang.VerifyError: Inconsistent stackmap frames

Post by Anton Katilin »

Hi Wojtek,
Q1: Could you elaborate more on what it means for me? What functionality will be limited?
Built-in probes are be disabled:
https://www.yourkit.com/docs/java/help/ ... probes.jsp
This affects the tabs "Events" ( https://www.yourkit.com/docs/java/help/probes_ui.jsp ) and "Performance charts" ( https://www.yourkit.com/docs/java/help/perf_charts.jsp )

Alternate workaround is to specify java command line option "-noverify". At least it helped other users with similar problem.
Q2: Can I turn it on later after proper app initialisation? The problem was occurring during my app initialisation. On my dev environment I was able to attach agent from YourKit App UI, when my application was already up and running without specifying any agent options.
No. Disabling probes works during entire run time. Unfortunately, it seems that the softer option "probe_off" which allows to activate probes in runtime won't help in your case.
Q3: Is this something that YourKit does not yet support on newer java versions?
It's not related with Java version.
Instead, it's related with unusual bytecode patterns generated by CGLIB.

We have a request about this problem, but did not manage to addressed it yet.

Which CGLIB version do you use?

Best regards,
Anton
wojtekD
Posts: 5
Joined: Wed Jul 01, 2020 6:40 am

Re: java.lang.VerifyError: Inconsistent stackmap frames

Post by wojtekD »

Thanks again for your answers.

'-noverify' is not an option on production since it introduces security risk. I've tested it earlier on dev environment and it of course fixes the problem since bytcode verification is turned off.

We are using CGLIB version 3.2.10. But we haven't changed this since at least end of 2019. And we started experiencing the issue around March 2020.

I have another thought. We are using spring framework and it is also using CGLIB. I've checked and found out that we started experiencing the issue around the time when we switched major spring version. We didn't observe the issue on 4.2.4.RELEASE and it showed up on 5.2.0.RELEASE. Any thoughts about that?

Going back to CGLIB version, do you know of any versions which are proven to not cause similar issues?

Best Regards,
Wojtek
Anton Katilin
Posts: 6172
Joined: Wed Aug 11, 2004 8:37 am

Re: java.lang.VerifyError: Inconsistent stackmap frames

Post by Anton Katilin »

Hello Wojtek,

The problem is apparently related with CGLIB usage in Spring, because the problematic class has "$$EnhancerBySpringCGLIB" in its name.

According to https://en.wikipedia.org/wiki/Spring_Framework the latest stable release is
5.2.7.RELEASE. Can you check with it instead of 5.2.0?

Unfortunately, we don't know if there are CGLIB not causing the problem. Our proposed approach to solve it is to change the profiler agent's bytecode instrumenter to not generate instruction patterns conflicting with CGLIB generated code. We estimate that the required changes should be rather big, so the solution won't be ready for the upcoming version 2020.7 unfortunately.

Best regards,
Anton
wojtekD
Posts: 5
Joined: Wed Jul 01, 2020 6:40 am

Re: java.lang.VerifyError: Inconsistent stackmap frames

Post by wojtekD »

Hello Anton,

I've checked spring framework 5.2.4.RELEASE and 5.2.7.RELEASE and the issue remains.
At least we now know the root cause and have acceptable workaround.

Thank you again for your help and expertise.

Best Regards,
Wojtek
Anton Katilin
Posts: 6172
Joined: Wed Aug 11, 2004 8:37 am

Re: java.lang.VerifyError: Inconsistent stackmap frames

Post by Anton Katilin »

Hello Wojtek

Could you please check whether disabling only one probe Threads with "probe_disable=.Threads" (it's a shortcut for com.yourkit.probes.builtin.Threads) solves the problem too, alternatively to disabling all probes with "probe_disable=*".

Best regards,
Anton
Anton Katilin
Posts: 6172
Joined: Wed Aug 11, 2004 8:37 am

Re: java.lang.VerifyError: Inconsistent stackmap frames

Post by Anton Katilin »

Hello Wojtek

Did you have a chance to try "probe_disable=.Threads" ?

Best regards,
Anton
wojtekD
Posts: 5
Joined: Wed Jul 01, 2020 6:40 am

Re: java.lang.VerifyError: Inconsistent stackmap frames

Post by wojtekD »

Hi Anton,

Apologise for delayed response.

Looks like disabling only Threads probe is enough and our application starts normally.

Can you share what's your further findings and why exactly this probe is causing issues?

Regards,
Wojtek
Anton Katilin
Posts: 6172
Joined: Wed Aug 11, 2004 8:37 am

Re: java.lang.VerifyError: Inconsistent stackmap frames

Post by Anton Katilin »

Thank you for the prompt reply.

Yes, to our current understanding a part of Threads probe is the cause, and we consider removal of that part (not the whole probe!) because it proved to be practically useless.

Technically, this is a recording of a lasting event for entire thread execution time. The problem is that the method pattern "* : run()" is too wide, it matches not only run() methods overridden in Thread subclasses but also all other such methods e.g. in all Runnables. Some non-Thread cases can be filtered out in bytecode instrumentation time so we don't instrument non-Threads, others not, and a supporting code should be inserted to check class in runtime before invoking the probe callback.

The problematic methods in your case match by pattern, and the supporting class-checking code interferes with unusual instruction sequences made by CGLIB.
Post Reply