https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/YANGTOOLS-1093 yangtools <p>This may be user error, or maybe improperly written yang, but when I try to run the  model validator against some juniper yang models from the <a href="https://github.com/YangModels/yang" class="external-link" target="_blank" rel="nofollow noopener">yang github repo</a> the tool consumes all given memory, spikes the CPU and eventually crashes with an OOM.</p> <p>Command to recreate running from root of yang repo:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> java -Xmx12288m -jar ../yangtools/yang/yang-model-validator/target/yang-model-validator-5.0.0-SNAPSHOT-jar-with-dependencies.jar -p ./vendor/juniper/18.2/18.2R1/ -r -d vendor/juniper/18.2/18.2R1/junos/conf/junos-conf-root@2018-01-01.yang </pre> </div></div> <p>Not very helpful, but here is one OOM crash trace:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> Exception in thread <span class="code-quote">"main"</span> java.lang.OutOfMemoryError: Java heap space at org.antlr.v4.runtime.misc.DoubleKeyMap.&lt;init&gt;(DoubleKeyMap.java:19) at org.antlr.v4.runtime.atn.ParserATNSimulator.computeReachSet(ParserATNSimulator.java:776) at org.antlr.v4.runtime.atn.ParserATNSimulator.execATNWithFullContext(ParserATNSimulator.java:664) at org.antlr.v4.runtime.atn.ParserATNSimulator.execATN(ParserATNSimulator.java:505) at org.antlr.v4.runtime.atn.ParserATNSimulator.adaptivePredict(ParserATNSimulator.java:393) at org.opendaylight.yangtools.yang.parser.antlr.YangStatementParser.statement(YangStatementParser.java:276) at org.opendaylight.yangtools.yang.parser.antlr.YangStatementParser.statement(YangStatementParser.java:229) at org.opendaylight.yangtools.yang.parser.antlr.YangStatementParser.statement(YangStatementParser.java:229) at org.opendaylight.yangtools.yang.parser.antlr.YangStatementParser.statement(YangStatementParser.java:229) at org.opendaylight.yangtools.yang.parser.antlr.YangStatementParser.statement(YangStatementParser.java:229) at org.opendaylight.yangtools.yang.parser.antlr.YangStatementParser.statement(YangStatementParser.java:229) at org.opendaylight.yangtools.yang.parser.antlr.YangStatementParser.statement(YangStatementParser.java:229) at org.opendaylight.yangtools.yang.parser.antlr.YangStatementParser.statement(YangStatementParser.java:229) at org.opendaylight.yangtools.yang.parser.rfc7950.repo.YangStatementStreamSource.parseYangSource(YangStatementStreamSource.java:171) at org.opendaylight.yangtools.yang.parser.rfc7950.repo.YangStatementStreamSource.create(YangStatementStreamSource.java:98) at org.opendaylight.yangtools.yang.parser.impl.YangParserImpl.sourceToStatementStream(YangParserImpl.java:119) at org.opendaylight.yangtools.yang.parser.impl.YangParserImpl.addLibSource(YangParserImpl.java:73) at org.opendaylight.yangtools.yang.validator.SystemTestUtils.parseYangSources(SystemTestUtils.java:103) at org.opendaylight.yangtools.yang.validator.SystemTestUtils.parseYangSources(SystemTestUtils.java:87) at org.opendaylight.yangtools.yang.validator.Main.runSystemTest(Main.java:177) at org.opendaylight.yangtools.yang.validator.Main.main(Main.java:136) </pre> </div></div> <p>Notice the above command is using 12G heap size. Obviously, it's much faster to hit<br/> with a smaller heap. I will link a heap dump from a 1G recreation</p> YANGTOOLS-1093

yang-model-validator tool crashes with OOM

Bug Medium Resolved Won't Do Unassigned Jamo Luhrsen Tue, 31 Mar 2020 15:40:20 +0000 Wed, 28 Oct 2020 17:39:18 +0000 Fri, 12 Jun 2020 17:01:12 +0000 0 3 <p>Attached is a screen shot from the yourkit profiler showing an expanded view of the objects taking 99% of allocated memory in case<br/> that helps. It was taken from <a href="https://drive.google.com/file/d/1YHgRiJoTqibl5QlSIcjbeN0582-1ZWkB/view?usp=sharing" class="external-link" target="_blank" rel="nofollow noopener">this heap dump </a></p> <p>Some of the juniper models are very large and I noticed the life profiler indicated some kind fo deadlock when it noticed some threads<br/> not moving and mentioned some recursion happening. Possibly if we are reading and holding on to the same large models over<br/> and over in duplicate during recursion we could end up in this trouble?</p> <p>The stack trace indicates 'addLibSource', i.e. when a source is being added to the library. We are creating an ASTSchemaSource to determine the real identifier of the source and are retaining it for future use (same as we do for normal sources).</p> <p>For normal sources this makes sense, as they are required and guaranteed to be touched during assembly. For library sources this is not as clear-cut, as they may or may not be referenced.</p> <p>We need to instantiate ASTSchemaSource lazily when it is actually needed. We also should not retain a strong reference to it, so as to allow it to be GC'd if it ends up being unneeded.</p> <p>This is going to take a bit more effort, as it requires reworking how SOURCE_PRE_LINKAGE phase works – which is part of <a href="https://jira.opendaylight.org/browse/YANGTOOLS-1112" title="Separate out ModelProcessingPhase.SOURCE_PRE_LINKAGE phase handling" class="issue-link" data-issue-key="YANGTOOLS-1112">YANGTOOLS-1112</a>. Without that refactor, any laziness in file parsing would be immediately negated by the indiscriminate loading of statements.</p> <p> </p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jluhrsen" class="user-hover" rel="jluhrsen">jluhrsen</a> while this issue is a valid improvement, I think the command uses a vastly over-specified library, which pulls in a <b>ton</b> of unrelated models – hence we should be able to limit the impact.</p> <p>agreed. Eventually I found ways to limit unneeded models and not kill things. Something like this is in a testtool is<br/> low priority and likely not anything to push for a fix at this point. Let's close it as won't do for now and if someone<br/> else comes along with a need we can look again.</p> Issue split YANGTOOLS-1112 Development Rank 0|i03rtz: