> I've started to work on this in:
> * more coming this week
hello, world - so is this KEWL, or WHAT... check out the output below from this new trace:transactions topic:bug/9060 tool!
So I can find 5 
transaction (thus memory) leaks JUST by installing odl-netvirt-openstack (master), and not doing ANYTHING yet... looks like our code is like Swiss Cheese, with many holes in it! It will be so much FUN to have this used e.g. in a CSIT and later during real scale testing.. and, likely, find many many more like this:
opendaylight-user@root>trace:transactions
TracingBroker found some not yet (or never..) closed transaction[chain]s!
[NB: If no stack traces are shown below, then enable transaction-debug-context-enabled in mdsaltrace_config.xml]
DataBroker : newReadOnlyTransaction()
1x transactions opened here, which are not closed:
(...)
Proxy52650ca9_3ae6_40f1_9c52_ec2047cecc51.newReadOnlyTransaction(Unknown Source)
org.opendaylight.openflowplugin.applications.topology.manager.FlowCapableTopologyProvider.isFlowTopologyExist(FlowCapableTopologyProvider.java:86)
org.opendaylight.openflowplugin.applications.topology.manager.FlowCapableTopologyProvider.start(FlowCapableTopologyProvider.java:57)
(...)
==> OPNFLWPLUG-934
DataBroker : createTransactionChain()
1x TransactionChains opened but not closed:
(...)
Proxyfaab8ae8_e5e3_4a72_a6ea_4ae4baef08d5.createTransactionChain(Unknown Source)
org.opendaylight.restconf.RestConnectorProvider.start(RestConnectorProvider.java:87)
(...)
==> NETCONF-462
1x TransactionChains opened but not closed:
(...)
Proxy52650ca9_3ae6_40f1_9c52_ec2047cecc51.createTransactionChain(Unknown Source)
org.opendaylight.openflowplugin.applications.topology.manager.OperationProcessor.<init>(OperationProcessor.java:36)
==> OPNFLWPLUG-935
1x TransactionChains opened but not closed:
(...)
Proxy5501955d_71e1_4829_a1c6_db3085042ff1.createTransactionChain(Unknown Source)
org.opendaylight.ovsdb.southbound.transactions.md.TransactionInvokerImpl.<init>(TransactionInvokerImpl.java:53)
org.opendaylight.ovsdb.southbound.SouthboundProvider.init(SouthboundProvider.java:79)
(...)
==> OVSDB-423
1x TransactionChains opened but not closed:
(...)
Proxyd6d64bb3_914d_4db8_9c2a_094278739686.createTransactionChain(Unknown Source)
org.opendaylight.ovsdb.hwvtepsouthbound.transactions.md.TransactionInvokerImpl.<init>(TransactionInvokerImpl.java:61)
org.opendaylight.ovsdb.hwvtepsouthbound.HwvtepSouthboundProvider.init(HwvtepSouthboundProvider.java:78)
==> OVSDB-424
Status update: This tool is now available in master. https://git.opendaylight.org/gerrit/#/c/62403/ would be great to have in netvirt master to make it much easier to use e.g. by QA folks (and in order to be able to start talking about integrating it into CSIT, which would be VERY cool).
Next step: I'll now start looking at the feasibility of back patching this stuff from master to stable/carbon, so that we can run it on internal QA environments under load - and use this to help us find the real culprit of CONTROLLER-1756!
> have this used e.g. in a CSIT
> start talking about integrating it into CSIT
==> NETVIRT-878
I've just (minutes ago) been able to merge the back-patch of new https://bugs.opendaylight.org/show_bug.cgi?id=9060 tooling from master into the upstream stable/carbon branch, and with that consider this issue fully done from my side. We'll be dealing with enhancements in new issues, such as CONTROLLER-1765 and hope to update CONTROLLER-1756 with the findings obtained thanks to this new tool soon (that was the main point of all of this).
http://blog2.vorburger.ch/2017/09/how-to-find-transaction-related-memory.html