https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/OPNFLWPLUG-961 OpenFlowPlugin <p>Following the (many!) fixes that we did in <a href="https://jira.opendaylight.org/browse/NETVIRT-985" title="java.lang.OutOfMemoryError: Java heap space" class="issue-link" data-issue-key="NETVIRT-985"><del>NETVIRT-985</del></a>, a new round of testing on the latest stable/carbon using the "nova-boot scenario" has hit an OOM again. A new "trace:transaction" output reveals more TX leaks which we need to plug; see attached.</p> <p>The biggest one with 273x open newReadOnlyTransaction likely causing the new OOM this time appears in (1) ovsdb; <a href="https://git.opendaylight.org/gerrit/#/c/65567/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/65567/</a> seems to not have helped for that one - curious... or we made a mess in downstream porting?!</p> <p>Filing this in JIRA project ovsdb due to above; but the attached shows a number of other leaks in (2) openflowplugin and (3) genius.lockmanager &amp; (4) genius.interfacemanager (x90!) which we need to plug as part of this JIRA as well.</p> OPNFLWPLUG-961

New OOM due to more TX leaks seen in the nova-boot scenario

Bug Highest Resolved Done Michael Vorburger Michael Vorburger Thu, 30 Nov 2017 14:21:49 +0000 Wed, 6 Dec 2017 15:28:41 +0000 Mon, 4 Dec 2017 14:37:26 +0000 0 2 <p><a href="https://git.opendaylight.org/gerrit/#/q/topic:OVSDB-435" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/q/topic:OVSDB-435</a> will have all changes re. this new OOM accross ovsdb, genius and openflowplugin.</p> <p>Extract of openflowplugin related leaks from attachments, WITHOUT the x1 which don't count:</p> <p>A. something in <tt>DeviceContextImpl</tt>:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>DataBroker : newReadOnlyTransaction() 5x transactions opened here, which are not closed: (...) org.opendaylight.controller.md.sal.dom.spi.ForwardingDOMDataBroker.newReadOnlyTransaction(ForwardingDOMDataBroker.java:42) (...) org.opendaylight.controller.md.sal.binding.impl.BindingDOMDataBrokerAdapter.newReadOnlyTransaction(BindingDOMDataBrokerAdapter.java:59) (...) org.opendaylight.openflowplugin.impl.device.DeviceContextImpl.getReadTransaction(DeviceContextImpl.java:244) org.opendaylight.openflowplugin.impl.statistics.StatisticsGatheringUtils.deleteAllKnownFlows(StatisticsGatheringUtils.java:208) org.opendaylight.openflowplugin.impl.services.AbstractMultipartRequestOnTheFlyCallback.startCollecting(AbstractMultipartRequestOnTheFlyCallback.java:135) org.opendaylight.openflowplugin.impl.services.AbstractMultipartRequestOnTheFlyCallback.onSuccess(AbstractMultipartRequestOnTheFlyCallback.java:84) org.opendaylight.openflowplugin.impl.services.AbstractMultipartRequestOnTheFlyCallback.onSuccess(AbstractMultipartRequestOnTheFlyCallback.java:35) org.opendaylight.openflowjava.protocol.impl.core.connection.OutboundQueueEntry.complete(OutboundQueueEntry.java:121) org.opendaylight.openflowjava.protocol.impl.core.connection.StackedSegment.completeEntry(StackedSegment.java:113) org.opendaylight.openflowjava.protocol.impl.core.connection.StackedSegment.pairRequest(StackedSegment.java:153) ...</pre> </div></div> <p>B. something re Ping Pong from <tt>TransactionChainManager</tt>:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>25x TransactionChains opened but not closed here: (...) org.opendaylight.controller.md.sal.dom.broker.impl.PingPongTransactionChain.&lt;init&gt;(PingPongTransactionChain.java:98) org.opendaylight.controller.md.sal.dom.broker.impl.PingPongDataBroker.createTransactionChain(PingPongDataBroker.java:47) org.opendaylight.controller.md.sal.dom.broker.impl.PingPongDataBroker.createTransactionChain(PingPongDataBroker.java:27) (...) org.opendaylight.controller.md.sal.binding.impl.BindingDOMTransactionChainAdapter.&lt;init&gt;(BindingDOMTransactionChainAdapter.java:45) org.opendaylight.controller.md.sal.binding.impl.BindingDOMDataBrokerAdapter.createTransactionChain(BindingDOMDataBrokerAdapter.java:74) (...) org.opendaylight.openflowplugin.common.txchain.TransactionChainManager.createTxChain(TransactionChainManager.java:80) org.opendaylight.openflowplugin.common.txchain.TransactionChainManager.onTransactionChainFailed(TransactionChainManager.java:278) org.opendaylight.controller.md.sal.binding.impl.BindingDOMTransactionChainAdapter.failTransactionChain(BindingDOMTransactionChainAdapter.java:110) org.opendaylight.controller.md.sal.binding.impl.BindingDOMTransactionChainAdapter.access$200(BindingDOMTransactionChainAdapter.java:31) org.opendaylight.controller.md.sal.binding.impl.BindingDOMTransactionChainAdapter$3.onFailure(BindingDOMTransactionChainAdapter.java:91) com.google.common.util.concurrent.Futures$6.run(Futures.java:1310) com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457) com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202) com.google.common.util.concurrent.SettableFuture.setException(SettableFuture.java:68) org.opendaylight.controller.md.sal.dom.broker.impl.PingPongTransaction.onFailure(PingPongTransaction.java:65) org.opendaylight.controller.md.sal.dom.broker.impl.PingPongTransactionChain.transactionFailed(PingPongTransactionChain.java:293) org.opendaylight.controller.md.sal.dom.broker.impl.PingPongTransactionChain$2.onFailure(PingPongTransactionChain.java:243) com.google.common.util.concurrent.Futures$6.run(Futures.java:1310) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748)</pre> </div></div> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=skitt" class="user-hover" rel="skitt">skitt</a> I had my (very very first, actually) look inside openflowplugin today, and .. find an entirely different way of writing code than in genius and netvirt in there; they have their own set up TX utils! <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/sad.png" height="16" width="16" align="absmiddle" alt="" border="0"/> Addressing isn't the 1h job I thought it would be. </p> <p>A. is easy to fix (if it is even a real leak...), see <a href="https://git.opendaylight.org/gerrit/#/c/66128/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/66128/</a> &amp; Co.</p> <p>But for B. I don't yet see how to easily address the leaks clearily found above. As these are minor leaks, compared to the other big ones we've seen, I am not planning to address this in short term. I'll email openflowplug-dev, but don't really expect them to fix this for us in the short term.</p> <p>PS: In the longer term, and with more effort, what could perhaps be interesting is to start exploring if openflowplugin could be changed to use the TX Utils we've now come up with in genius, like our ManagedNewTransactionRunner?</p> Blocks NETVIRT-878 NETVIRT-985 Cloners OVSDB-435 Development Rank 0|i038mf: