https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-1559 controller <p>The references to service providers are not removed from memory on blueprint containers' restart. This produces a memory leak on each bundle tree restart (e.g. initiated by a change in *.cfg configs).<br/> Example of a provider reference chain left in memory after restart is in the attachement.</p> <p>Operating System: All<br/> Platform: All</p> CONTROLLER-1559

Memory leak during bundle tree restart

Bug Resolved Done Alexis de Talhouët Martin Dindoffer Tue, 18 Oct 2016 12:55:30 +0000 Tue, 25 Jul 2023 08:24:14 +0000 Thu, 17 Nov 2016 13:43:55 +0000 blueprint 0 6 <p>Attachment file_6969.txt has been added with description: Example of a provider leak reference chain</p> <p>Martin, could you try this fix please: <a href="https://git.opendaylight.org/gerrit/#/c/48284/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/48284/</a></p> <p>Attachment topoChain.txt has been added with description: Path to the GC roots of a leaked TopoProcessingProviderImpl after a proposed fix</p> <p>@Alexis<br/> This actually seems to make the problem even worse on current Topoprocessing master (546d29e92f78a97419eff784b863f9a39b0dbcbf). Now, after 2 restarts and a GC, I have 7 instances of TopoProcessingProviderImpl in memory. Is this related to the (not merged yet) patch of yours in topoprocessing <span class="error">&#91;0&#93;</span>?<br/> Anyway, I've attached a full path to GC roots of a leaked provider with the provided fix.</p> <p><span class="error">&#91;0&#93;</span> <a href="https://git.opendaylight.org/gerrit/#/c/48283/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/48283/</a></p> <p>(On a side note, I am currently on vacation till early December. So I don't guarantee I will be reachable.)</p> <p>Thanks Martin for the quick feedback. I understand you might not pick up the fix until early December, thanks for notifying as well.</p> <p>You are correct the fix I provided in topoprocessing is also part of the resolution of this bug, but was unfortunately uncomplete at the time of your testing, now it is.</p> <p>I have identified two root causes for the memory leak:</p> <p>1. Retention on BlueprintContainer when destroying them before "quiescing"<br/> them: <a href="https://git.opendaylight.org/gerrit/#/c/48284/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/48284/</a><br/> Restarting a set of bundles requires cooperation between them to avoid<br/> having retention on services due to on-going call, processing.<br/> As OSGi APIs don't provide such functionality, the Quiesce APIs where<br/> created, ensuring all calls to be finished before actually shutting down<br/> the bundle. Once the set of bundles are quiesced, e.g. shut down, they<br/> can safely be destroyed.</p> <ul class="alternate" type="square"> <li>Retention on the TopoProcessingProviderImpl due to unreleased resources:<br/> <a href="https://git.opendaylight.org/gerrit/#/c/48283/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/48283/</a></li> </ul> <p>Obversed behavior with both the changes:<br/> a. change the config value in the config file<br/> b. bundle is reloaded<br/> b. i. no more BlueprintContainer retention (opendaylight-user@root&gt;services show only one BlueprintContainer)<br/> b. ii. memory still have two instances of TopoProcessingProviderImpl, but one is marked as "unreachable from GC roots, but not yet collected." hence will be eventually collected.<br/> Forcing GC removes that instance.</p> <p>Let me know if those fixes are indeed solving your issues. Thanks.</p> <p>Martin, on the controller side, that bug is now resolved. I've opened a BUG in Topoprocessing to track the memory leak over there: <a href="https://bugs.opendaylight.org/show_bug.cgi?id=7197" class="external-link" target="_blank" rel="nofollow noopener">https://bugs.opendaylight.org/show_bug.cgi?id=7197</a></p> <p>Thanks,<br/> Alexis</p> Development External issue ID 6969 External issue URL Issue Type Rank 0|i02rbj: