Operating System: All
Platform: All

Based on a heap dump we've got, it is something between karaf and felix resolver. The resolver is tasked to resolve 24837 requirements, which quickly falls over due to OpenHashMapList cloning during permutations (recorded 978 copies) and 24M CandidateSelector objects.

Moreover there are huge numbers of duplicate karaf's SimpleFilter objects - in groups of 400-5000 objects, wasting around 5MB memory.

Furthermore there is a ton of duplicate (Linked)HashMap entries retained from RequirementImpl – typically containing only a single element, wasting quite a bit of memory there.

To top it all off, there are 21K sparsely populated – pre-allocated to 10 entries, but holding 1 or 2 entries – most of which comes from SimpleFilter.

A test case is quite simple:

• download Nitrogen
• JAVA_MAX_MEM=1g ./bin/karaf
• repo-add mvn:org.opendaylight.integration/odl-integration-all/0.7.0/xml/features
• feature:install odl-integration-all

So this is definitely exposed by us packaging bundles multiple times. The problem is that felix resolver is performing SUBSTITUTE permutations, which are the memory hog.

These are emitted each time a bundle is duplicated – e.g. bgp-parser-api, which is packaged three times adds two permutation candidates.

I think suspect there is a bug in karaf, but we can definitely hide the issue by being careful with out packaging.

So this issue is a combination of us multi-packaging our features and how Karaf implements Resource, Capability and Requirement interfaces – these have well-defined semantics and are tied together (i.e. they cross-reference each other). They are violating the API contract by not providing proper hashCode()/equals() as specified by the contract.

While Capability and Requirement are easily retro-fitted, Karaf does not record Resource (i.e. bundle) content sufficiently to make Resource equals() not dependent on Capability.

This has two impacts:
1) Each bundle declaration is treated as unique, hence multiple declarations in a feature (and even within the same feature) are not squashed, even when they point to the same bundle.

2) Felix Resolver is led to believe that duplicate bundles are actually different potential candidates, which provide same capabilities (expanding the search space along the permutation axis) and also treating each such bundle's requirements as unique (expanding the search space along the requirement axis).

The end result is that the resolver has a large working set for each permutation and also has a large number of permutations – quickly exhausting memory.

Fixing this completely correctly on Karaf side is probably going to be a major surgery and a multi-week effort.

Raising severity of this issue to blocker. Will re-target to Nitrogen SR1 and open per-project issues to once there is an SR1 milestone in bugzilla.

https://issues.apache.org/jira/browse/KARAF-5395 upstream.

Even with Odlparent 2.0.5 (Karaf 4.0.10) we can run out of heap when installing large enough tree of features. See [2].

Note that SingleFeatureTest during distribution-check currently uses 4 GB of heap space. But subsequent boot (to verify Restconf) uses only 3 GB.

Distribution-check currently does not archive heap dump. Even if it did, .gz would have almost 900 MB, so Nexus upload would fail.
.xz would have around 120MB, it would take some time to make distribution-check archive that.

[2] https://git.opendaylight.org/gerrit/#/c/61750/4

> .xz would have around 120MB, it would take some time to make
> distribution-check archive that.

Some time has passed, releng/builder improvement is ready: [3].

[3] https://git.opendaylight.org/gerrit/63992

patches for individual projects are collected under https://git.opendaylight.org/gerrit/#/q/topic:bug9218

>> Even with Odlparent 2.0.5 (Karaf 4.0.10) we can run out of heap when
>> installing large enough tree of features. See [2].

> releng/builder improvement is ready: [3].

Compressed heap dump is archived: [4].

[4] https://logs.opendaylight.org/releng/jenkins092/distribution-check-nitrogen/425/hprof.tar.xz