https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/OPNFLWPLUG-982 OpenFlowPlugin <p><a href="https://jira.opendaylight.org/browse/NETVIRT-878" title="CSIT should help to detect possible memory leaks leading to OOM related to non-closed transactions (and tx chains) early" class="issue-link" data-issue-key="NETVIRT-878">NETVIRT-878</a> brings up <a href="https://jira.opendaylight.org/browse/OPNFLWPLUG-940" title="Suspected (not sure) TransactionChain leak in TransactionChainManager" class="issue-link" data-issue-key="OPNFLWPLUG-940"><del>OPNFLWPLUG-940</del></a> (or was it <a href="https://jira.opendaylight.org/browse/OPNFLWPLUG-961" title="New OOM due to more TX leaks seen in the nova-boot scenario" class="issue-link" data-issue-key="OPNFLWPLUG-961"><del>OPNFLWPLUG-961</del></a> or <a href="https://jira.opendaylight.org/browse/OPNFLWPLUG-933" title="IllegalStateException: Attempted to close chain with outstanding transaction PingPongTransaction at org.opendaylight.openflowplugin.impl.device.TransactionChainManager.createTxChain" class="issue-link" data-issue-key="OPNFLWPLUG-933"><del>OPNFLWPLUG-933</del></a> ?) again:</p> <p>This time we need to either fix this, or if we confirm this is not a real issue, then move this issue to the Controller project and change the org.opendaylight.controller.md.sal.trace.dom.impl.TracingBroker to ignore it:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>TracingBroker found some not yet (or never..) closed transaction[chain]s! [NB: If no stack traces are shown below, then enable transaction-debug-context-enabled in mdsaltrace_config.xml] DataBroker : createTransactionChain() 3x TransactionChains opened but not closed here: (...) org.opendaylight.controller.md.sal.dom.broker.impl.PingPongTransactionChain.&lt;init&gt;(PingPongTransactionChain.java:104) org.opendaylight.controller.md.sal.dom.broker.impl.PingPongDataBroker.createTransactionChain(PingPongDataBroker.java:49) org.opendaylight.controller.md.sal.dom.broker.impl.PingPongDataBroker.createTransactionChain(PingPongDataBroker.java:28) (...) org.opendaylight.controller.md.sal.binding.impl.BindingDOMTransactionChainAdapter.&lt;init&gt;(BindingDOMTransactionChainAdapter.java:45) org.opendaylight.controller.md.sal.binding.impl.BindingDOMDataBrokerAdapter.createTransactionChain(BindingDOMDataBrokerAdapter.java:74) (...) org.opendaylight.openflowplugin.common.txchain.TransactionChainManager.createTxChain(TransactionChainManager.java:81) org.opendaylight.openflowplugin.common.txchain.TransactionChainManager.activateTransactionManager(TransactionChainManager.java:109) org.opendaylight.openflowplugin.impl.device.DeviceContextImpl.lazyTransactionManagerInitialization(DeviceContextImpl.java:649) org.opendaylight.openflowplugin.impl.device.DeviceContextImpl.instantiateServiceInstance(DeviceContextImpl.java:592) org.opendaylight.openflowplugin.impl.lifecycle.GuardedContextImpl.instantiateServiceInstance(GuardedContextImpl.java:86) java.util.concurrent.CopyOnWriteArrayList.forEach(CopyOnWriteArrayList.java:891) org.opendaylight.openflowplugin.impl.lifecycle.ContextChainImpl.instantiateServiceInstance(ContextChainImpl.java:74) org.opendaylight.mdsal.singleton.dom.impl.ClusterSingletonServiceGroupImpl.lambda$startServices$0(ClusterSingletonServiceGroupImpl.java:648) java.util.ArrayList.forEach(ArrayList.java:1257) org.opendaylight.mdsal.singleton.dom.impl.ClusterSingletonServiceGroupImpl.startServices(ClusterSingletonServiceGroupImpl.java:645) org.opendaylight.mdsal.singleton.dom.impl.ClusterSingletonServiceGroupImpl.cleanupCandidateOwnershipChanged(ClusterSingletonServiceGroupImpl.java:506) org.opendaylight.mdsal.singleton.dom.impl.ClusterSingletonServiceGroupImpl.lockedOwnershipChanged(ClusterSingletonServiceGroupImpl.java:453) org.opendaylight.mdsal.singleton.dom.impl.ClusterSingletonServiceGroupImpl.ownershipChanged(ClusterSingletonServiceGroupImpl.java:433) org.opendaylight.mdsal.singleton.dom.impl.AbstractClusterSingletonServiceProviderImpl.ownershipChanged(AbstractClusterSingletonServiceProviderImpl.java:234) org.opendaylight.mdsal.singleton.dom.impl.DOMClusterSingletonServiceProviderImpl.ownershipChanged(DOMClusterSingletonServiceProviderImpl.java:23) org.opendaylight.controller.cluster.datastore.entityownership.EntityOwnershipListenerActor.onEntityOwnershipChanged(EntityOwnershipListenerActor.java:44) org.opendaylight.controller.cluster.datastore.entityownership.EntityOwnershipListenerActor.handleReceive(EntityOwnershipListenerActor.java:33) org.opendaylight.controller.cluster.common.actor.AbstractUntypedActor.onReceive(AbstractUntypedActor.java:38) (...)</pre> </div></div> OPNFLWPLUG-982

Suspected TransactionChain leak in TransactionChainManager (or fix false positive in TracingBroker)

Bug High Resolved Won't Do Stephen Kitt Michael Vorburger Mon, 26 Feb 2018 17:48:52 +0000 Thu, 12 Apr 2018 09:48:47 +0000 Thu, 12 Apr 2018 09:48:47 +0000 Oxygen Fluorine 0 3 <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=skitt" class="user-hover" rel="skitt">skitt</a> Stephen, are you looking at this issue ? If not, please unassign so we can triage it.</p> <p>I am indeed working on it, <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=Avishnoi" class="user-hover" rel="Avishnoi">Avishnoi</a>; I get the impression there’s a bunch of “synchronisation by magic” (aka GuardedBy statements with no corresponding reality) which could explain contribute to this, although it could also be perfectly innocent (if three devices are connected, there should be three outstanding transaction chains, right?).</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=Avishnoi" class="user-hover" rel="Avishnoi">Avishnoi</a> and <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=skitt" class="user-hover" rel="skitt">skitt</a> I meant to speak to you F2F last week at ONS about this, but forgot.. can we move on this?</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=skitt" class="user-hover" rel="skitt">skitt</a> This class was reworked meany times to take care of mutliple race condition. Plugin creates TransactionChainManager per device and it is used in various places in openflowplugin to process the southbound events coming from device like</p> <p>(1) Flow/Group/Port/Queue/Table statistics --&gt; they all are separate events and can come in any order and in parallel. Even Flow/Group statistics are divided in multiple openflow messages and each message end up generating once transaction. </p> <p>(2) Port status messages --&gt; these are asyn messages and can be send anytime from switch if any port goes down.</p> <p>(3) Flow removed messages –&gt; Async message send by switch whenever flow is removed from switch.</p> <p>(4) I may be forgetting one or two more events.</p> <p> </p> <p>So although TransactionChainManager instance is created per node, it needs to deal with the concurrency because of these various parallel events processing and hence "GuardedBy". But i would love someone to review the synchronization in this class and push a patch if it can be handle in better way.</p> <p>Regarding the open transaction chain, given that openflowplugin create TransactionChainManager instance per device, it opens the transaction chain per node after device is initialized properly.  I believe that's the reason you see these open transaction chains. I believe this can be further optimized by creating the transaction chain lazily, but as far as i can see this is not really a leak as such. </p> <p> </p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=skitt" class="user-hover" rel="skitt">skitt</a> if you agree with <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=Avishnoi" class="user-hover" rel="Avishnoi">Avishnoi</a> then let us close this? </p> <p>I would then implement <a href="https://jira.opendaylight.org/browse/CONTROLLER-1765" title="trace:transactions should suppress false TransactionChain positives, by white list, or showing only x2+ ?" class="issue-link" data-issue-key="CONTROLLER-1765"><del>CONTROLLER-1765</del></a> and then <a href="https://jira.opendaylight.org/browse/NETVIRT-878" title="CSIT should help to detect possible memory leaks leading to OOM related to non-closed transactions (and tx chains) early" class="issue-link" data-issue-key="NETVIRT-878">NETVIRT-878</a> could use that.</p> <p>Yes, I agree. After careful analysis, there doesn’t seem to be anything wrong in the TransactionChainManager, although I still have an upcoming patch to clarify things and help out static analysis tools.</p> <p>This is a false positive.</p> Blocks NETVIRT-878 Development Rank 0|i03asf: