https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/ODLPARENT-125 odlparent <p>SingleFeatureTest is failing on odl-integration-all feature for some contributions, preventing them from merging (as distribution-check fails).</p> <p>As reported in e-mail <span class="error">&#91;0&#93;</span>, there is a workaround (Distribution to set higher heap size in order for SingleFeatureTest to pass), which just needs Odlparent releasing 2.0.5 version.</p> <p>Local testing showed this might be just a consequence of <a href="https://jira.opendaylight.org/browse/ODLPARENT-115" title="Karaf is slow to start processing features after start" class="issue-link" data-issue-key="ODLPARENT-115"><del>ODLPARENT-115</del></a> Karaf 4.0.9 consumes not only time but also memory in excess. But it might be possible to fix memory consumption in a different way.</p> <p>Also, this Bug is more severe than <a href="https://jira.opendaylight.org/browse/ODLPARENT-115" title="Karaf is slow to start processing features after start" class="issue-link" data-issue-key="ODLPARENT-115"><del>ODLPARENT-115</del></a> as that has a workaround of just waiting for a longer time.</p> <p>Additional example of a Change failing distribution-check: <span class="error">&#91;1&#93;</span>.</p> <p><span class="error">&#91;0&#93;</span> <a href="https://lists.opendaylight.org/pipermail/odlparent-dev/2017-September/001341.html" class="external-link" target="_blank" rel="nofollow noopener">https://lists.opendaylight.org/pipermail/odlparent-dev/2017-September/001341.html</a><br/> <span class="error">&#91;1&#93;</span> <a href="https://git.opendaylight.org/gerrit/63329" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/63329</a></p> <p>Operating System: All<br/> Platform: All</p> ODLPARENT-125

Oxygen development is limited by SingleFeatureTest running out of heap space

Bug Resolved Done Robert Varga Vratko Polak Wed, 27 Sep 2017 08:45:16 +0000 Fri, 13 Apr 2018 08:47:45 +0000 Fri, 13 Apr 2018 08:47:45 +0000 2.0.5 3.1.0 General 0 1 <p>Based on a heap dump we've got, it is something between karaf and felix resolver. The resolver is tasked to resolve 24837 requirements, which quickly falls over due to OpenHashMapList cloning during permutations (recorded 978 copies) and 24M CandidateSelector objects.</p> <p>Moreover there are huge numbers of duplicate karaf's SimpleFilter objects - in groups of 400-5000 objects, wasting around 5MB memory.</p> <p>Furthermore there is a ton of duplicate (Linked)HashMap entries retained from RequirementImpl &#8211; typically containing only a single element, wasting quite a bit of memory there.</p> <p>To top it all off, there are 21K sparsely populated &#8211; pre-allocated to 10 entries, but holding 1 or 2 entries &#8211; most of which comes from SimpleFilter.</p> <p>A test case is quite simple:</p> <ul class="alternate" type="square"> <li>download Nitrogen</li> <li>JAVA_MAX_MEM=1g ./bin/karaf</li> <li>repo-add mvn:org.opendaylight.integration/odl-integration-all/0.7.0/xml/features</li> <li>feature:install odl-integration-all</li> </ul> <p>So this is definitely exposed by us packaging bundles multiple times. The problem is that felix resolver is performing SUBSTITUTE permutations, which are the memory hog.</p> <p>These are emitted each time a bundle is duplicated &#8211; e.g. bgp-parser-api, which is packaged three times adds two permutation candidates.</p> <p>I think suspect there is a bug in karaf, but we can definitely hide the issue by being careful with out packaging.</p> <p>So this issue is a combination of us multi-packaging our features and how Karaf implements Resource, Capability and Requirement interfaces &#8211; these have well-defined semantics and are tied together (i.e. they cross-reference each other). They are violating the API contract by not providing proper hashCode()/equals() as specified by the contract.</p> <p>While Capability and Requirement are easily retro-fitted, Karaf does not record Resource (i.e. bundle) content sufficiently to make Resource equals() not dependent on Capability.</p> <p>This has two impacts:<br/> 1) Each bundle declaration is treated as unique, hence multiple declarations in a feature (and even <b>within</b> the same feature) are not squashed, even when they point to the same bundle.</p> <p>2) Felix Resolver is led to believe that duplicate bundles are actually different potential candidates, which provide same capabilities (expanding the search space along the permutation axis) and also treating each such bundle's requirements as unique (expanding the search space along the requirement axis).</p> <p>The end result is that the resolver has a large working set for each permutation and also has a large number of permutations &#8211; quickly exhausting memory.</p> <p>Fixing this completely correctly on Karaf side is probably going to be a major surgery and a multi-week effort.</p> <p>Raising severity of this issue to blocker. Will re-target to Nitrogen SR1 and open per-project issues to once there is an SR1 milestone in bugzilla.</p> <p><a href="https://issues.apache.org/jira/browse/KARAF-5395" class="external-link" target="_blank" rel="nofollow noopener">https://issues.apache.org/jira/browse/KARAF-5395</a> upstream.</p> <p>Even with Odlparent 2.0.5 (Karaf 4.0.10) we can run out of heap when installing large enough tree of features. See <span class="error">&#91;2&#93;</span>.</p> <p>Note that SingleFeatureTest during distribution-check currently uses 4 GB of heap space. But subsequent boot (to verify Restconf) uses only 3 GB.</p> <p>Distribution-check currently does not archive heap dump. Even if it did, .gz would have almost 900 MB, so Nexus upload would fail.<br/> .xz would have around 120MB, it would take some time to make distribution-check archive that.</p> <p><span class="error">&#91;2&#93;</span> <a href="https://git.opendaylight.org/gerrit/#/c/61750/4" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/61750/4</a></p> <p>&gt; .xz would have around 120MB, it would take some time to make<br/> &gt; distribution-check archive that.</p> <p>Some time has passed, releng/builder improvement is ready: <span class="error">&#91;3&#93;</span>.</p> <p><span class="error">&#91;3&#93;</span> <a href="https://git.opendaylight.org/gerrit/63992" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/63992</a></p> <p>patches for individual projects are collected under <a href="https://git.opendaylight.org/gerrit/#/q/topic:bug9218" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/q/topic:bug9218</a></p> <p>&gt;&gt; Even with Odlparent 2.0.5 (Karaf 4.0.10) we can run out of heap when<br/> &gt;&gt; installing large enough tree of features. See <span class="error">&#91;2&#93;</span>.</p> <p>&gt; releng/builder improvement is ready: <span class="error">&#91;3&#93;</span>.</p> <p>Compressed heap dump is archived: <span class="error">&#91;4&#93;</span>.</p> <p><span class="error">&#91;4&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/distribution-check-nitrogen/425/hprof.tar.xz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/distribution-check-nitrogen/425/hprof.tar.xz</a></p> Development External issue ID 9218 External issue URL Issue Type ODL SR Target Milestone Rank 0|i023gn: