Operating System: All
Platform: All

Attachment file_6969.txt has been added with description: Example of a provider leak reference chain

Martin, could you try this fix please: https://git.opendaylight.org/gerrit/#/c/48284/

Attachment topoChain.txt has been added with description: Path to the GC roots of a leaked TopoProcessingProviderImpl after a proposed fix

@Alexis
This actually seems to make the problem even worse on current Topoprocessing master (546d29e92f78a97419eff784b863f9a39b0dbcbf). Now, after 2 restarts and a GC, I have 7 instances of TopoProcessingProviderImpl in memory. Is this related to the (not merged yet) patch of yours in topoprocessing [0]?
Anyway, I've attached a full path to GC roots of a leaked provider with the provided fix.

[0] https://git.opendaylight.org/gerrit/#/c/48283/

(On a side note, I am currently on vacation till early December. So I don't guarantee I will be reachable.)

Thanks Martin for the quick feedback. I understand you might not pick up the fix until early December, thanks for notifying as well.

You are correct the fix I provided in topoprocessing is also part of the resolution of this bug, but was unfortunately uncomplete at the time of your testing, now it is.

I have identified two root causes for the memory leak:

1. Retention on BlueprintContainer when destroying them before "quiescing"
them: https://git.opendaylight.org/gerrit/#/c/48284/
Restarting a set of bundles requires cooperation between them to avoid
having retention on services due to on-going call, processing.
As OSGi APIs don't provide such functionality, the Quiesce APIs where
created, ensuring all calls to be finished before actually shutting down
the bundle. Once the set of bundles are quiesced, e.g. shut down, they
can safely be destroyed.

• Retention on the TopoProcessingProviderImpl due to unreleased resources:
  https://git.opendaylight.org/gerrit/#/c/48283/

Obversed behavior with both the changes:
a. change the config value in the config file
b. bundle is reloaded
b. i. no more BlueprintContainer retention (opendaylight-user@root>services show only one BlueprintContainer)
b. ii. memory still have two instances of TopoProcessingProviderImpl, but one is marked as "unreachable from GC roots, but not yet collected." hence will be eventually collected.
Forcing GC removes that instance.

Let me know if those fixes are indeed solving your issues. Thanks.

Martin, on the controller side, that bug is now resolved. I've opened a BUG in Topoprocessing to track the memory leak over there: https://bugs.opendaylight.org/show_bug.cgi?id=7197

Thanks,
Alexis