https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/YANGTOOLS-694 yangtools <p>Analysis of a memory snapshot with vendor models shows we end up with a SchemaContext which weighs about 240MB. Further analysis of duplicate objects shows we have ~4000 instances of DescriptionEffectiveStatementImpl wasting ~400kB, which refer to a single description statement defined in a grouping.</p> <p>This hints at us not having the understanding that a statement has been copied and unmodified from a grouping. Analyze the exact cause for this and fix it either by not performing a copy, or squashing resulting duplicates.</p> <p>This theory is reinforced by the fact that the same dump shows ~64K WhenEffectiveStatementImpl objects (23x groups of 2769 instances each) coming from a construct like:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> grouping foo { leaf bar { when <span class="code-quote">"../x"</span>; } } </pre> </div></div> <p>A similar situation happens with UnitsEffectiveStatementImpl – 15K instances wasting 450kB.</p> <p>Operating System: All<br/> Platform: All</p> YANGTOOLS-694

Eliminate duplicate DescriptionEffectiveStatementImpl objects

Improvement Highest Resolved Done Miroslav Kovac Robert Varga Tue, 18 Oct 2016 12:59:40 +0000 Sun, 10 Apr 2022 18:35:54 +0000 Wed, 20 Jan 2021 18:36:39 +0000 7.0.0 0 7 <p>I think the root cause for this is in SubstatementContext.copy</p> {Declared,Effective} <p>Statements and GroupingUtils.needToCopyByUses(). The check is a weird and undocumented one and does not take effect in situations like:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> grouping foo { leaf foo { description <span class="code-quote">"foo"</span>; } } </pre> </div></div> <p>Note leaf is copied, but since description's getParentContext() points to a leaf, the second part check fails and the description ends up being copied.</p> <p>If this is true, the fix for this issue will also mean we will cut down temporary object use, too.</p> <p>A simple unit test for this can be created. With the following models:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>module foo { grouping foo { leaf foo { type string; units foo; } } grouping export { uses grouping foo; } } module bar { import foo; container bar-export { uses foo:export; } container bar-foo { uses foo:foo; } } module baz { import foo; container baz-export { uses foo:export; } container baz-foo { uses foo:foo; } }</pre> </div></div> <p>All units visible from effective model should point to the same UnitsEffectiveStatementImpls bject.</p> <p>We have eliminated all obvious duplicates during assembly, with the exception of duplication of StatementContextBase, which is performed when statements are inlined via uses statement.</p> <p>This is caused by the fact we copy all declared/effective statements eagerly, no matter if they will be touched in future – just to update their parent, path and copy history.</p> <p>We need to create StatementContextBase equivalent proxy, which will perform the copy lazily as needed and add the changed CopyHistory and similar details to createEffective(), so it ends up reusing statement definitions if they are not touched by an augment, refine or similar.</p> <p><a href="https://jira.opendaylight.org/browse/YANGTOOLS-779" title="OOM expection while parsing 27.1 MB of YANG with 2GO RAM" class="issue-link" data-issue-key="YANGTOOLS-779"><del>YANGTOOLS-779</del></a> shows that this particular issue can be hit out in the field. ~28MB of models, which are presumably grouping/uses-heavy can cause us to run out 4GB of heap.</p> <p>A memory snapshot taken due to an OOM during FULL_DECLARATION parsing phase shows 1.5GB memory usage, with 6.2M SubstatementContext instances retaining ~1.1GB.</p> <p>While this memory usage is temporary (BuildGlobalContext shows 800MB retained at this point), we obviously need to address this to keep our resource usage within reasonable bounds.</p> <p>I am not sure we can deliver this by Carbon SR1.</p> <p>Solving this issue requires re-wiring 'uses', 'augment', 'refine' and 'deviate' mechanics, which deals with namespace management and child nodes. Namespace management there is tricky and we essentially want a copy-on-write StatementContextBase.</p> <p>There are three sides to this problem:</p> <p>1) the side we are investigating, which is reuse and lazily-populated StatementContextBase objects (which will need to be refactored). This involves dealing with CopyHistory and instantiation namespaces. Since this is traversed in a tree-like manner, this is a relatively simple problem, as all it boils down to is using forwarding objects in UsesStatementImpl's resolution phase.</p> <p>2) the side of augment/refine/deviate making modifications to objects, which really expands forwarding objects with lazy instantiation to cope with modifications.</p> <p>3) the side of actually triggering augment/deviate/uses phase actions, which is driven by reactor and SchemaNodeIdentifierBuildNamespace, which is populated by StmtContext.Mutable object coming and going.</p> <p>It is the combination of 1) and 3) which makes this hard, as there are competing requirements. If 'uses' stops instantiating StatementContextBases, it stops feeding the namespace, which means augment/deviate/uses triggers will no longer trigger when they should.</p> <p>Turning this into a milestone, as this will require multiple steps, which we do not want to do in one go simply because of the complexities involved.</p> <p>Part 3) would be solved with <a href="https://jira.opendaylight.org/browse/YANGTOOLS-859" title="Augmenting a container with if-feature" class="issue-link" data-issue-key="YANGTOOLS-859"><del>YANGTOOLS-859</del></a>, that issue needs to have a phase action trigger when an unsupported node is added – which can be also used to move a registration downwards until it hits the appropriate proxy, eventually either fully resolving the node, or knowing further resolution is impossible.</p> <p><a href="https://git.opendaylight.org/gerrit/73972" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/73972</a> solves the triggering problem. All triggers now follow ChildSchemaNodeNamespace, hence lazy StatementContextBase equivalent class can be plugged in: it can implement namespaces and it can also act as a statement – hence 3) will not be a problem.</p> <p>Part 2) is being solved in <a href="https://jira.opendaylight.org/browse/YANGTOOLS-724" title="Make AbstractEffectiveDataSchemaNode.augmenting private and final" class="issue-link" data-issue-key="YANGTOOLS-724"><del>YANGTOOLS-724</del></a>, and that is the next step we need to take.</p> <p>So, I've run unit test with next models:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> module bar { namespace bar-ns; prefix bar; <span class="code-keyword">import</span> foo { prefix foo;} container bar-export { uses foo:export; } container bar-foo { uses foo:foo; } } module baz { namespace baz-ns; prefix baz; <span class="code-keyword">import</span> foo { prefix foo;} container baz-export { uses foo:export; } container baz-foo { uses foo:foo; } } module foo { namespace foo-ns; prefix foo; grouping foo { leaf foo { type string; units foo; } } grouping export { uses foo; } } </pre> </div></div> <p>And memory usage less than 50 Mb.</p> <p><span class="image-wrap" style=""><a id="15404_thumb" href="https://jira.opendaylight.org/secure/attachment/15404/15404_Screen+Shot+2019-08-14+at+12.46.25.png" title="Screen Shot 2019-08-14 at 12.46.25.png" file-preview-type="image" file-preview-id="15404" file-preview-title="Screen Shot 2019-08-14 at 12.46.25.png"><img src="https://jira.opendaylight.org/secure/thumbnail/15404/_thumb_15404.png" style="border: 0px solid black" role="presentation"/></a></span></p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=rovarga" class="user-hover" rel="rovarga">rovarga</a> any ideas how to reproduce it?</p> <p> </p> <p>Well, you need to analyze a heap dump to see if there are duplicate objects – using YourKit, for example.</p> <p><span class="image-wrap" style=""><a id="15405_thumb" href="https://jira.opendaylight.org/secure/attachment/15405/15405_Screen+Shot+2019-08-15+at+10.07.46.png" title="Screen Shot 2019-08-15 at 10.07.46.png" file-preview-type="image" file-preview-id="15405" file-preview-title="Screen Shot 2019-08-15 at 10.07.46.png"><img src="https://jira.opendaylight.org/secure/thumbnail/15405/_thumb_15405.png" style="border: 0px solid black" role="presentation"/></a></span></p> <p>Profile looks normal. There aren't thousands of duplicates.</p> <p><span class="nobr"><a href="https://jira.opendaylight.org/secure/attachment/15406/15406_heap.bin" title="heap.bin attached to YANGTOOLS-694">heap.bin^{<img class="rendericon" src="https://jira.opendaylight.org/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span> here is heap memory dump</p> <p><span class="image-wrap" style=""><img src="https://jira.opendaylight.org/secure/attachment/15407/15407_yt694-dups.png" style="border: 0px solid black" /></span></p> <p>As you can see there are 6 identical objects, expanded from a single grouping. This may not look like much, but it certainly adds up when you work with something like <a href="https://github.com/Juniper/yang/tree/master/19.2/19.2R1/junos/conf" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/Juniper/yang/tree/master/19.2/19.2R1/junos/conf</a> .</p> <p> </p> <p>To add some colour to the picture: a SchemaContext can easily weigh 375MiB, as noted (as a matter of course) in <a href="https://jira.opendaylight.org/browse/YANGTOOLS-1004" title="four new netconf mounts using same schema not sharing same compiled schema" class="issue-link" data-issue-key="YANGTOOLS-1004"><del>YANGTOOLS-1004</del></a> – and these things are per device-type, so they directly affect scalability to no small extent.</p> <p>There are three remaining steps in this task:</p> <ol> <li>clarify StatementSupport.applyCopyPolicy() behavior</li> <li>audit all StatementSupport classes to check whether they are using correct CopyPolicy</li> <li>address the FIXME in CONTEXT_INDEPENDENT case of StatementContextBase.copyAsChildOf(), so that completely independent statements are not copied. Note the second FIXME should be taken care of by <a href="https://jira.opendaylight.org/browse/YANGTOOLS-784" title="Defer statement initialization in InferredStatementContext" class="issue-link" data-issue-key="YANGTOOLS-784"><del>YANGTOOLS-784</del></a>.</li> </ol> <p>So where are here:</p> <p><span class="image-wrap" style=""><img src="https://jira.opendaylight.org/secure/attachment/15909/15909_image-2020-12-03-18-31-00-574.png" height="741" width="1332" style="border: 0px solid black" /></span></p> <p>the highlighted statements should really be squashed, as at a number of them are duplicates, as confirmed by examining declared instance references:</p> <p><span class="image-wrap" style=""><img src="https://jira.opendaylight.org/secure/attachment/15910/15910_image-2020-12-03-18-56-27-486.png" height="624" width="1122" style="border: 0px solid black" /></span></p> <p>This issue should take care of at least these.</p> Blocks YANGTOOLS-1067 YANGTOOLS-784 YANGTOOLS-859 Duplicate YANGTOOLS-779 Relates YANGTOOLS-1065 YANGTOOLS-1066 YANGTOOLS-724 YANGTOOLS-743 YANGTOOLS-1121 YANGTOOLS-1163 YANGTOOLS-1164 YANGTOOLS-1195 Development Epic Link YANGTOOLS-652 External issue ID 6972 External issue URL Issue Type Priority Rank 0|i028t3: