[YANGTOOLS-629] Performance optimization potential in yang loading? Profiling SingleFeatureTest shows Hot Methods in e.g. StmtContextUtils & StatementDefinitionContext Created: 05/Jul/16  Updated: 10/Apr/22  Resolved: 24/Aug/16

Status: Resolved
Project: yangtools
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug
Reporter: Michael Vorburger Assignee: Peter Kajsa
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: All
Platform: All


Attachments: Zip Archive SingleFeatureTest-Karaf-JavaFlightRecorder4817713825730748944.jfr.zip    
Issue Links:
Blocks
blocks ODLPARENT-38 SingleFeatureTest fails with (ClassNo... Resolved
External issue ID: 6158

 Description   

I was curious today why that SingleFeatureTest was taking so long (e.g. approx. 15' on netvirt/vpnservice/features currently), and did some profiling using (Oracle, not OpenJDK) Java Mission Control of a local Maven build. I'll attach the zipped *.jfr file if anyone else wants to have a look.

If I'm interpreting things correctly, which I may not, then a (to me) surprising fraction of the time spent in SingleFeatureTest, something like 20% (maybe I'm adding up things that are showing twice the same), is spent not in Karaf code loading features and all, nor in actual functional bundle initialization stuff (like netvirt or so) but in yangtools related stuff to load *.yang files. If this is just perfectly normal and to be expected, in the SingleFeatureTest use case, then please feel free to close this as WONTFIX.

Just wondering if there isn't any Performance optimization potential in yang loading ... I've played with 1-2 things myself locally, related to more efficient collection use (I'll put in a Gerrit for that). My change does get the related code out of the Hot Methods in JMC, but overall still seems to have very little impact on the ca. 15' SingleFeatureTest of netvirt/vpnservice/features, unfortunately.

After that change of mine, the Hot Methods in JMC seems to say that ca. 10% of the total time is in java.util.Collections$UnmodifiableCollection$1.hasNext() related to org.opendaylight.yangtools.yang.parser.spi.meta.StmtContextUtils.findFirstEffectiveSubstatement, and another 10% (which may be the same; I'm not 100% sure how to interpret) related to org.opendaylight.yangtools.yang.parser.stmt.reactor.StatementDefinitionContext getPublicView() and org.opendaylight.yangtools.yang.parser.stmt.reactor.StatementContextBase getPublicDefinition(), which doesn't really make any sense to me, as upon a closer look the call chain just seems to be a bunch of getters calling through, so I'm not sure what to make of that.

Without fully understanding the internals, I'm wondering it it may be feasible to apply some caching somewhere (related to what jumps out as Hot Methods in JMC) ?



 Comments   
Comment by Michael Vorburger [ 05/Jul/16 ]

> played with 1-2 things myself locally, related to more efficient collection

==> https://git.opendaylight.org/gerrit/41339

Comment by Michael Vorburger [ 05/Jul/16 ]

Attachment SingleFeatureTest-Karaf-JavaFlightRecorder4817713825730748944.jfr.zip has been added with description: Java Mission Control flight recorder performance profiling of SingleFeatureTest

Comment by Michael Vorburger [ 05/Jul/16 ]

> did some profiling using (Oracle, not OpenJDK) Java Mission Control

--> using https://git.opendaylight.org/gerrit/#/c/41349/

Comment by Robert Varga [ 06/Jul/16 ]

Performance improvements are ceratinly welcome. I think this problem comes specifically from if-features implemention and will need to be analyzed a bit.

I do not have JMC handy, will take a look at call sites for fFES() – but I doubt there is much we can do to cache that, as it really is filtering on dynamically interpreted context – e.g. cache invalidations are a problem.

Comment by Peter Kajsa [ 07/Jul/16 ]

https://git.opendaylight.org/gerrit/#/c/41471

Comment by Michael Vorburger [ 07/Jul/16 ]

> https://git.opendaylight.org/gerrit/#/c/41471

I have just verified the impact of this on the SingleFeatureTest build time on netvirt/vpnservice/features, and can confirm that this approx. HALFS the time, 7:30 instead of the ca. 15' from above! This is AMAZING! GREAT. PS: Can you work even more optimization magic like this?

I cannot verify the proposed change itself of course, as my current understanding of yangtools internals is still too limited.

I can also not confirm that this has no other side effects, as I haven't actually run any code generated by this with "real" usage - only built netvirt/vpnservice/features (SingleFeatureTest).

Comment by Michael Vorburger [ 24/Aug/16 ]

FTR: YANGTOOLS-652 has something new that's a bit similar to this.

Generated at Wed Feb 07 20:53:50 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.