https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/ODLPARENT-39 odlparent <p>I have finally found a moment to performance profile the yangtools/mdsal code generation which occurs during a Maven build by attaching Java Mission Control (JMC) from Oracle JDK (not available on OpenJDK) to a local Maven build.</p> <p>This is similar to earlier <a href="https://jira.opendaylight.org/browse/YANGTOOLS-629" title="Performance optimization potential in yang loading? Profiling SingleFeatureTest shows Hot Methods in e.g. StmtContextUtils &amp; StatementDefinitionContext" class="issue-link" data-issue-key="YANGTOOLS-629"><del>YANGTOOLS-629</del></a> but here focusing on looking at Maven itself, Maven plugins, code generation and build - so intentionally excluding the SingleFeatureTest and integration tests etc. using my new -Pq quick profile which local developers are increasingly using now.</p> <p>Here are the exact steps I've used, if anyone wants to reproduce:</p> <p>$ export JAVA_HOME=/home/vorburger/bin/jdk1.8.0_92-ORACLE<br/> $ export PATH=$JAVA_HOME/bin:$PATH<br/> $ export MAVEN_OPTS="-XX:+UnlockCommercialFeatures -XX:+FlightRecorder -XX:FlightRecorderOptions=defaultrecording=true,dumponexit=true,dumponexitpath=/home/vorburger/dev/ODL/git/netvirt/mvn.jfr"<br/> $ mvno -o -Pq clean install</p> <p>The result, attached as netvirt-mvn-o-Pq-clean-install.jfr.7z, is interesting IMHO - unless JMC is totally off, it would seem in <span class="error">&#91;2&#93;</span> that we are spending ca. 20% of total Maven build time of e.g. netvirt in Xtend's org.eclipse.xtend2.lib.StringConcatenation called by YangTemplate, just to build those "verboseClassComments" for 'generated classes will include javadoc comments which re useful for users' with the "This class represents the following YANG schema fragment defined in module ...". The 20% just for that seems a lot - that's just a bunch of String concatenations, through the Xtend helper, which is supposed to "allow for efficient <span class="error">&#91;, indentation aware&#93;</span> concatenation of character sequences" - I think either something is wrong with, or we're using it wrong. But the latter seems unlikely, as <span class="error">&#91;1&#93;</span> YangTemplate looks like a pretty standard plain Xtend template class, to me...</p> <p>As a first step, I'll try to raise this with the Xmen (Xtend community), to see what they have to say about this.</p> <p><span class="error">&#91;1&#93;</span> <a href="https://github.com/opendaylight/mdsal/blob/32d45af3bfde08f4e8741632ecab43241831d1fe/binding/mdsal-binding-generator-impl/src/main/java/org/opendaylight/yangtools/sal/binding/generator/impl/YangTemplate.xtend" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/opendaylight/mdsal/blob/32d45af3bfde08f4e8741632ecab43241831d1fe/binding/mdsal-binding-generator-impl/src/main/java/org/opendaylight/yangtools/sal/binding/generator/impl/YangTemplate.xtend</a></p> <p><span class="error">&#91;2&#93;</span> JMC Code top Hot Method TXT copy/paste:</p> <p>Stack Trace Sample Count Percentage(%)<br/> org.eclipse.xtend2.lib.StringConcatenation.appendSegments(String, int, List, String) 2,407 19.344<br/> org.eclipse.xtend2.lib.StringConcatenation.append(Object, String, int) 2,407 19.344<br/> org.eclipse.xtend2.lib.StringConcatenation.append(Object, String) 2,407 19.344<br/> org.opendaylight.yangtools.sal.binding.generator.impl.YangTemplate.writeListSchemaNode(ListSchemaNode) 674 5.417<br/> org.opendaylight.yangtools.sal.binding.generator.impl.YangTemplate.writeDataSchemaNode(DataSchemaNode) 674 5.417<br/> org.opendaylight.yangtools.sal.binding.generator.impl.YangTemplate.writeDataSchemaNodes(Collection) 609 4.894<br/> org.opendaylight.yangtools.sal.binding.generator.impl.YangTemplate.writeContSchemaNode(ContainerSchemaNode) 423 3.4<br/> org.opendaylight.yangtools.sal.binding.generator.impl.YangTemplate.writeDataSchemaNode(DataSchemaNode) 423 3.4<br/> org.opendaylight.yangtools.sal.binding.generator.impl.YangTemplate.writeDataSchemaNodes(Collection) 223 1.792<br/> org.opendaylight.yangtools.sal.binding.generator.impl.YangTemplate.writeListSchemaNode(ListSchemaNode) 128 1.029<br/> org.opendaylight.yangtools.sal.binding.generator.impl.YangTemplate.writeDataSchemaNode(DataSchemaNode) 128 1.029<br/> org.opendaylight.yangtools.sal.binding.generator.impl.YangTemplate.writeDataSchemaNodes(Collection) 105 0.844<br/> org.opendaylight.yangtools.sal.binding.generator.impl.YangTemplate.writeListSchemaNode(ListSchemaNode) 34 0.273<br/> org.opendaylight.yangtools.sal.binding.generator.impl.YangTemplate.writeDataSchemaNode(DataSchemaNode) 34 0.273<br/> org.opendaylight.yangtools.sal.binding.generator.impl.YangTemplate.writeDataSchemaNodes(Collection) 31 0.249<br/> org.opendaylight.yangtools.sal.binding.generator.impl.YangTemplate.writeContSchemaNode(ContainerSchemaNode) 11 0.088<br/> org.opendaylight.yangtools.sal.binding.generator.impl.YangTemplate.writeDataSchemaNode(DataSchemaNode) 11 0.088<br/> org.opendaylight.yangtools.sal.binding.generator.impl.YangTemplate.generateYangSnipet(SchemaNode) 8 0.064<br/> org.opendaylight.yangtools.sal.binding.generator.impl.BindingGeneratorImpl.createDescription(SchemaNode, String) 8 0.064</p> <p>Operating System: All<br/> Platform: All</p> ODLPARENT-39

Code Generation Performance optimization required in YangTemplate (Xtend bug?)

Bug Resolved Done Michael Vorburger Michael Vorburger Tue, 19 Jul 2016 12:12:50 +0000 Wed, 24 Jan 2018 14:27:03 +0000 Thu, 2 Feb 2017 18:44:13 +0000 General 0 2 <p>Attachment netvirt-mvn-o-Pq-clean-install.jfr.7z has been added with description: Oracle Java Mission Control JFR profiling of "mvno -o -Pq clean install" in netvirt on latest netvirt master today (65fb44e) with export MAVEN_OPTS="-Xmx4096m -Xverify:none"</p> <p>&gt; raise this with the Xmen (Xtend community), to see what they have to say</p> <p>==&gt; <a href="https://bugs.eclipse.org/bugs/show_bug.cgi?id=427909" class="external-link" target="_blank" rel="nofollow noopener">https://bugs.eclipse.org/bugs/show_bug.cgi?id=427909</a></p> <p>any updates on this? Thanks.</p> <p>Martin, thanks for the ping - it's sometimes useful to get reminders... <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/wink.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Dug a little more into this today. In <a href="https://git.opendaylight.org/gerrit/#/c/47036/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/47036/</a>, I'm introducing a proposed new "yang.skip.javadoc" system property, which if true skips the generation of those "verboseClassComments" for 'generated classes will include javadoc comments which re useful for users' with the "This class represents the following YANG schema fragment defined in module ...".</p> <p>The results are too small to be significant on a single project, but on a multi-module reactor as small as e.g. netvirt/vpnservice this looks very interesting, and makes me thing I'm onto something that is worth digging more into, check this out:</p> <p>netvirt/vpnservice:<br/> mvn -o -Pq clean package: Total time: 03:14-21 min<br/> mvn -o -Pq -Dyang.skip.javadoc=true clean package: Total time: 02:15-18 min</p> <p>At the scale of a full ODL (autobuild), this would obviously likely be even more significant.</p> <p>I'll use this as motivation to reach out to eclipse.org friends about <a href="https://bugs.eclipse.org/bugs/show_bug.cgi?id=427909" class="external-link" target="_blank" rel="nofollow noopener">https://bugs.eclipse.org/bugs/show_bug.cgi?id=427909</a>. In the mean time, if we would get <a href="https://git.opendaylight.org/gerrit/#/c/47036/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/47036/</a> into ODL (and I would follow with an oldparent change to leverage this IFF -Pq), that would probably be something useful in the very short term already.</p> <p>Michael, is this still being worked on?</p> <p>&gt; Michael, is this still being worked on?</p> <p>Robert, yup it is; this is proceeding on two levels, as outlined above:</p> <p>a) <a href="https://git.opendaylight.org/gerrit/#/c/47036/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/47036/</a> for short-term</p> <p>b) <a href="https://bugs.eclipse.org/bugs/show_bug.cgi?id=427909" class="external-link" target="_blank" rel="nofollow noopener">https://bugs.eclipse.org/bugs/show_bug.cgi?id=427909</a> for medium term. I've actually discussed that with Sven Efftinge (AKA Mr. Xtext &amp; Xtend..) at EclipseCon just last week! I've demonstrated the problem, and convinced him that there is one, which is always a good first step.. <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/wink.png" height="16" width="16" align="absmiddle" alt="" border="0"/> We attempted a 5' quick fix together (replacement of an ArrayList by a LinkedList), but saw that was no good, and that Xtend lib StringConcatenation would need more than 5' of work. If I have nothing else on my pile, I may spend a bit more time on it, but it's not a high priority for me, given that we have a short-term work around.</p> <p>yeah, it's maybe 30 minutes &#8211; if that protected splitter method can be made non-public <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/> If it needs to be retained, you are in a pickle...</p> <p>Attachment xtextlib.diff has been added with description: prototype dirty fix</p> <p>Robert, as per private email exchange, I've tested your "prototype dirty fix" (above), fixed what we agreed in email was left in there but you meant to remove, and done a few other things in StringConcatenation orthogonal to your change, which in my measurements have an additional measurable positive impact; see <a href="https://bugs.opendaylight.org/show_bug.cgi?id=6236" class="external-link" target="_blank" rel="nofollow noopener">https://bugs.opendaylight.org/show_bug.cgi?id=6236</a> for full details, and <a href="https://github.com/eclipse/xtext-lib/pull/16/files" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/eclipse/xtext-lib/pull/16/files</a> for the proposed combined change. It's a step, but somehow I'm not entirely satisfied - still hot spot, when the goal should be to not have any in three.. Please do shout if you see any potential for further improvement?</p> <p>I've attempted to try out the full scenario from above with this fix, by replacing the x3 xtext/xbase/xtend (except xtend-maven-plugin) 2.9.2 in odlparent with locally built 2.11.0-SNAPSHOT:</p> <p>An "mvno -o -Pq clean install" in netvirt/vpnservice seems to go from approx. 3:07 down to 2:40, so 27s gained (BTW with a master 2.11.0-SNAPSHOT without changes it's still 3:06; tested).</p> <p>That's something, and reason enough for me to keep suggesting to Xtend to do something about this, on their side. For us in ODL in the short term though, as even this gain, if they accept it, when we upgrade, is still less than the more significant 1’ gain seen earlier above when just completely skipping Xtend with <a href="https://git.opendaylight.org/gerrit/#/c/47036/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/47036/</a>. My suggestion would thus be that we integrate c/47036 anyway.</p> <p>PS: When I JMC profile mvn then the hotstop still looks like the old one, so I doubted for a moment whether the lib switch from 2.9.2 to 2.11.0-SNAPSHOT even worked.. but as there is a measurable difference, that must be something else - the most likely explanation I have is that perhaps the test YangTemplateWithoutAnyDependencies.xtend doesn't lead to the exact same result as our "real" YangTemplate.xtend?</p> <p>Another performance improvement for codegen to not emit empty indentation strings: <a href="https://bugs.eclipse.org/bugs/show_bug.cgi?id=507077" class="external-link" target="_blank" rel="nofollow noopener">https://bugs.eclipse.org/bugs/show_bug.cgi?id=507077</a></p> <p><a href="https://git.opendaylight.org/gerrit/#/c/48290/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/48290/</a></p> <p>Hi Michael</p> <p>I guess this can be closed as RESOLVED FIXED, right?</p> <p>&gt; I guess this can be closed as RESOLVED FIXED, right?</p> <p>I would suggest to keep it open until a new Xtend version 2.11 including our (Robert's) fixes is released, and then switch to it, and retest.. According to <a href="https://typefox.io/xtext-2-11-beta-1-is-here" class="external-link" target="_blank" rel="nofollow noopener">https://typefox.io/xtext-2-11-beta-1-is-here</a>, "The Xtext 2.11 release has been rescheduled for January 24th 2017."</p> <p>boron backport: <a href="https://git.opendaylight.org/gerrit/49234" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/49234</a></p> <p>The rest should be handled by odlparent upgrading to xtend to 2.11 once it is out.</p> <p>&gt; boron backport: <a href="https://git.opendaylight.org/gerrit/49234" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/49234</a></p> <p>OK but in that case, also <a href="https://git.opendaylight.org/gerrit/#/c/49235/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/49235/</a></p> <p>The workarounds have been delivered to Boron-SR3, upstream patches have been accepted and will be part of xtend-2.11, which is scheduled to release on Jan 24.</p> <p>Targeting for M4 for the corresponding version bump.</p> <p>xtext release has been delayed, we will probably not make it to M5.</p> <p>patch to bump to 2.11.0.RC2: <a href="https://git.opendaylight.org/gerrit/51185" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/51185</a></p> <p>final bump: <a href="https://git.opendaylight.org/gerrit/51357" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/51357</a></p> Blocks MDSAL-54 Development External issue ID 6236 External issue URL Issue Type ODL SR Target Milestone Rank 0|i022xj: