https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/NETVIRT-1295 netvirt <p><a href="https://jira.opendaylight.org/browse/NETVIRT-1089" title="Add trace:transactions to suite teardowns" class="issue-link" data-issue-key="NETVIRT-1089">NETVIRT-1089</a> found these in natservice, all of which need to be (separately) plugged:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>DataBroker : newReadOnlyTransaction() 28x transactions opened here, which are not closed: (...) org.opendaylight.controller.md.sal.binding.impl.BindingDOMDataBrokerAdapter.newReadOnlyTransaction(BindingDOMDataBrokerAdapter.java:59) (...) org.opendaylight.netvirt.natservice.internal.ExternalRoutersListener.read(ExternalRoutersListener.java:573) org.opendaylight.netvirt.natservice.internal.ExternalRoutersListener.subnetRegisterMapping(ExternalRoutersListener.java:422) org.opendaylight.netvirt.natservice.internal.NaptSwitchHA.subnetRegisterMapping(NaptSwitchHA.java:1063) org.opendaylight.netvirt.natservice.internal.RouterDpnChangeListener.handleSNATForDPN(RouterDpnChangeListener.java:333) org.opendaylight.netvirt.natservice.internal.RouterDpnChangeListener.lambda$add$0(RouterDpnChangeListener.java:198) org.opendaylight.infrautils.jobcoordinator.internal.JobCoordinatorImpl$MainTask.runWithUncheckedExceptionLogging(JobCoordinatorImpl.java:416)</pre> </div></div> <p>as well as this:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> 2x transactions opened here, which are not closed: (...) org.opendaylight.controller.md.sal.binding.impl.BindingDOMDataBrokerAdapter.newReadOnlyTransaction(BindingDOMDataBrokerAdapter.java:59) (...) org.opendaylight.netvirt.natservice.internal.ExternalRoutersListener.read(ExternalRoutersListener.java:573) org.opendaylight.netvirt.natservice.internal.ExternalRoutersListener.subnetRegisterMapping(ExternalRoutersListener.java:422) org.opendaylight.netvirt.natservice.internal.ExternalRoutersListener.handleEnableSnat(ExternalRoutersListener.java:300) org.opendaylight.netvirt.natservice.internal.ExternalRoutersListener.lambda$add$0(ExternalRoutersListener.java:281) org.opendaylight.infrautils.jobcoordinator.internal.JobCoordinatorImpl$MainTask.runWithUncheckedExceptionLogging(JobCoordinatorImpl.java:416)</pre> </div></div> <p>and this:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> DataBroker : newWriteOnlyTransaction() 59x transactions opened here, which are not closed: (...) org.opendaylight.controller.md.sal.binding.impl.BindingDOMDataBrokerAdapter.newWriteOnlyTransaction(BindingDOMDataBrokerAdapter.java:69) (...) org.opendaylight.netvirt.natservice.internal.NatInterfaceStateChangeListener$NatInterfaceStateUpdateWorker.call(NatInterfaceStateChangeListener.java:377) org.opendaylight.netvirt.natservice.internal.NatInterfaceStateChangeListener$NatInterfaceStateUpdateWorker.call(NatInterfaceStateChangeListener.java:356) org.opendaylight.infrautils.jobcoordinator.internal.JobCoordinatorImpl$MainTask.runWithUncheckedExceptionLogging(JobCoordinatorImpl.java:416) org.opendaylight.infrautils.utils.concurrent.LoggingUncaughtThreadDeathContextRunnable.run(LoggingUncaughtThreadDeathContextRunnable.java:60) java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402) java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)</pre> </div></div> <p>and:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>49x transactions opened here, which are not closed: (...) org.opendaylight.controller.md.sal.binding.impl.BindingDOMDataBrokerAdapter.newWriteOnlyTransaction(BindingDOMDataBrokerAdapter.java:69) (...) org.opendaylight.netvirt.natservice.internal.ExternalRoutersListener.lambda$remove$3(ExternalRoutersListener.java:1685) org.opendaylight.infrautils.jobcoordinator.internal.JobCoordinatorImpl$MainTask.runWithUncheckedExceptionLogging(JobCoordinatorImpl.java:416) org.opendaylight.infrautils.utils.concurrent.LoggingUncaughtThreadDeathContextRunnable.run(LoggingUncaughtThreadDeathContextRunnable.java:60) java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402) java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)</pre> </div></div> NETVIRT-1295

Transaction leak in ExternalRoutersListener

Bug Medium Resolved Done Karthikeyan Krishnan Michael Vorburger feedback Mon, 4 Jun 2018 16:30:53 +0000 Wed, 22 Jan 2020 13:20:44 +0000 Wed, 22 Jan 2020 13:20:34 +0000 Fluorine Fluorine-SR2 Neon natservice 0 5 <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jluhrsen" class="user-hover" rel="jluhrsen">jluhrsen</a> in chat with <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=skitt" class="user-hover" rel="skitt">skitt</a> he rightfully requested we set the right Affects Version here - are your <a href="https://jira.opendaylight.org/browse/NETVIRT-1089" title="Add trace:transactions to suite teardowns" class="issue-link" data-issue-key="NETVIRT-1089">NETVIRT-1089</a> CSIT jobs where I copy/pasted above from running on master (Fluorin) or stable/oxygen?</p> <p>latest open transactions from a fluorine job:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> DataBroker : newReadOnlyTransaction() 28x transactions opened here, which are not closed: (...) org.opendaylight.controller.md.sal.binding.impl.BindingDOMDataBrokerAdapter.newReadOnlyTransaction(BindingDOMDataBrokerAdapter.java:57) (...) org.opendaylight.netvirt.natservice.internal.ExternalRoutersListener.read(ExternalRoutersListener.java:576) org.opendaylight.netvirt.natservice.internal.ExternalRoutersListener.subnetRegisterMapping(ExternalRoutersListener.java:425) org.opendaylight.netvirt.natservice.internal.NaptSwitchHA.subnetRegisterMapping(NaptSwitchHA.java:1063) org.opendaylight.netvirt.natservice.internal.RouterDpnChangeListener.handleSNATForDPN(RouterDpnChangeListener.java:325) org.opendaylight.netvirt.natservice.internal.RouterDpnChangeListener.lambda$<span class="code-keyword">null</span>$0(RouterDpnChangeListener.java:194) org.opendaylight.genius.infra.ManagedNewTransactionRunnerImpl.callWithNewWriteOnlyTransactionAndSubmit(ManagedNewTransactionRunnerImpl.java:45) org.opendaylight.netvirt.natservice.internal.RouterDpnChangeListener.lambda$<span class="code-keyword">null</span>$1(RouterDpnChangeListener.java:193) org.opendaylight.genius.infra.ManagedNewTransactionRunnerImpl.callWithNewWriteOnlyTransactionAndSubmit(ManagedNewTransactionRunnerImpl.java:45) org.opendaylight.netvirt.natservice.internal.RouterDpnChangeListener.lambda$add$2(RouterDpnChangeListener.java:151) org.opendaylight.infrautils.jobcoordinator.internal.JobCoordinatorImpl$MainTask.runWithUncheckedExceptionLogging(JobCoordinatorImpl.java:401) org.opendaylight.infrautils.utils.concurrent.LoggingUncaughtThreadDeathContextRunnable.run(LoggingUncaughtThreadDeathContextRunnable.java:60) java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402) java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157) 2x transactions opened here, which are not closed: (...) org.opendaylight.controller.md.sal.binding.impl.BindingDOMDataBrokerAdapter.newReadOnlyTransaction(BindingDOMDataBrokerAdapter.java:57) (...) org.opendaylight.netvirt.natservice.internal.ExternalRoutersListener.read(ExternalRoutersListener.java:576) org.opendaylight.netvirt.natservice.internal.ExternalRoutersListener.subnetRegisterMapping(ExternalRoutersListener.java:425) org.opendaylight.netvirt.natservice.internal.ExternalRoutersListener.handleEnableSnat(ExternalRoutersListener.java:303) org.opendaylight.netvirt.natservice.internal.ExternalRoutersListener.lambda$<span class="code-keyword">null</span>$0(ExternalRoutersListener.java:287) org.opendaylight.genius.infra.ManagedNewTransactionRunnerImpl.callWithNewWriteOnlyTransactionAndSubmit(ManagedNewTransactionRunnerImpl.java:45) org.opendaylight.netvirt.natservice.internal.ExternalRoutersListener.lambda$add$1(ExternalRoutersListener.java:270) org.opendaylight.infrautils.jobcoordinator.internal.JobCoordinatorImpl$MainTask.runWithUncheckedExceptionLogging(JobCoordinatorImpl.java:401) org.opendaylight.infrautils.utils.concurrent.LoggingUncaughtThreadDeathContextRunnable.run(LoggingUncaughtThreadDeathContextRunnable.java:60) java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402) java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157) DataBroker : createTransactionChain() 1x TransactionChains opened but not closed here: (...) org.opendaylight.controller.md.sal.binding.impl.BindingDOMTransactionChainAdapter.&lt;init&gt;(BindingDOMTransactionChainAdapter.java:45) org.opendaylight.controller.md.sal.binding.impl.BindingDOMDataBrokerAdapter.createTransactionChain(BindingDOMDataBrokerAdapter.java:72) (...) org.opendaylight.ovsdb.hwvtepsouthbound.transactions.md.TransactionInvokerImpl.&lt;init&gt;(TransactionInvokerImpl.java:60) org.opendaylight.ovsdb.hwvtepsouthbound.HwvtepSouthboundProvider.init(HwvtepSouthboundProvider.java:81) (...) java.util.concurrent.FutureTask.run(FutureTask.java:266) (...) java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) java.util.concurrent.FutureTask.run(FutureTask.java:266) (...) java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) java.util.concurrent.FutureTask.run(FutureTask.java:266) java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.<span class="code-object">Thread</span>.run(<span class="code-object">Thread</span>.java:748) 3x TransactionChains opened but not closed here: (...) org.opendaylight.controller.md.sal.dom.broker.impl.PingPongTransactionChain.&lt;init&gt;(PingPongTransactionChain.java:101) org.opendaylight.controller.md.sal.dom.broker.impl.PingPongDataBroker.createTransactionChain(PingPongDataBroker.java:49) org.opendaylight.controller.md.sal.dom.broker.impl.PingPongDataBroker.createTransactionChain(PingPongDataBroker.java:28) (...) org.opendaylight.controller.md.sal.binding.impl.BindingDOMTransactionChainAdapter.&lt;init&gt;(BindingDOMTransactionChainAdapter.java:45) org.opendaylight.controller.md.sal.binding.impl.BindingDOMDataBrokerAdapter.createTransactionChain(BindingDOMDataBrokerAdapter.java:72) (...) org.opendaylight.openflowplugin.common.txchain.TransactionChainManager.createTxChain(TransactionChainManager.java:81) org.opendaylight.openflowplugin.common.txchain.TransactionChainManager.activateTransactionManager(TransactionChainManager.java:109) org.opendaylight.openflowplugin.impl.device.DeviceContextImpl.lazyTransactionManagerInitialization(DeviceContextImpl.java:686) org.opendaylight.openflowplugin.impl.device.DeviceContextImpl.instantiateServiceInstance(DeviceContextImpl.java:629) org.opendaylight.openflowplugin.impl.lifecycle.GuardedContextImpl.instantiateServiceInstance(GuardedContextImpl.java:86) java.util.concurrent.CopyOnWriteArrayList.forEach(CopyOnWriteArrayList.java:891) org.opendaylight.openflowplugin.impl.lifecycle.ContextChainImpl.instantiateServiceInstance(ContextChainImpl.java:74) org.opendaylight.mdsal.singleton.dom.impl.ClusterSingletonServiceGroupImpl.lambda$startServices$0(ClusterSingletonServiceGroupImpl.java:648) java.util.ArrayList.forEach(ArrayList.java:1257) org.opendaylight.mdsal.singleton.dom.impl.ClusterSingletonServiceGroupImpl.startServices(ClusterSingletonServiceGroupImpl.java:645) org.opendaylight.mdsal.singleton.dom.impl.ClusterSingletonServiceGroupImpl.cleanupCandidateOwnershipChanged(ClusterSingletonServiceGroupImpl.java:506) org.opendaylight.mdsal.singleton.dom.impl.ClusterSingletonServiceGroupImpl.lockedOwnershipChanged(ClusterSingletonServiceGroupImpl.java:453) org.opendaylight.mdsal.singleton.dom.impl.ClusterSingletonServiceGroupImpl.ownershipChanged(ClusterSingletonServiceGroupImpl.java:433) org.opendaylight.mdsal.singleton.dom.impl.AbstractClusterSingletonServiceProviderImpl.ownershipChanged(AbstractClusterSingletonServiceProviderImpl.java:234) org.opendaylight.mdsal.singleton.dom.impl.DOMClusterSingletonServiceProviderImpl.ownershipChanged(DOMClusterSingletonServiceProviderImpl.java:23) org.opendaylight.controller.cluster.datastore.entityownership.EntityOwnershipListenerActor.onEntityOwnershipChanged(EntityOwnershipListenerActor.java:44) org.opendaylight.controller.cluster.datastore.entityownership.EntityOwnershipListenerActor.handleReceive(EntityOwnershipListenerActor.java:33) org.opendaylight.controller.cluster.common.actor.AbstractUntypedActor.onReceive(AbstractUntypedActor.java:38) (...) 1x TransactionChains opened but not closed here: (...) org.opendaylight.controller.md.sal.binding.impl.BindingDOMTransactionChainAdapter.&lt;init&gt;(BindingDOMTransactionChainAdapter.java:45) org.opendaylight.controller.md.sal.binding.impl.BindingDOMDataBrokerAdapter.createTransactionChain(BindingDOMDataBrokerAdapter.java:72) (...) org.opendaylight.ovsdb.southbound.transactions.md.TransactionInvokerImpl.&lt;init&gt;(TransactionInvokerImpl.java:54) org.opendaylight.ovsdb.southbound.SouthboundProvider.init(SouthboundProvider.java:89) (...) java.util.concurrent.FutureTask.run(FutureTask.java:266) (...) java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) java.util.concurrent.FutureTask.run(FutureTask.java:266) (...) java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) java.util.concurrent.FutureTask.run(FutureTask.java:266) java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.<span class="code-object">Thread</span>.run(<span class="code-object">Thread</span>.java:748) </pre> </div></div> <p>the below comment was from the fluorine job I ran on 6/5. I don't remember what I used for the original<br/> logs. sorry <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/sad.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Thanks; I've set the Affects Version field to Fluorine.</p> <p>Can I suggest that we treat only the leak in netvirt.natservice here, and ignore the other leaks shown above in this issue.</p> <p>Hi All,</p> <p>I have addressed the issue on ReadTransaction not getting closed in ExternalRouterListener.</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/72710/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/72710/</a></p> <p>Where I can see these kind of exception when CSIT(test-netvirt-current) is ranned as I didn't observe any such error in below ODL log location.</p> <p><a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-queens-gate-stateful-fluorine/549/odl_1/" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-queens-gate-stateful-fluorine/549/odl_1/</a>  </p> <p>&gt; Where I can see these kind of exception when CSIT(test-netvirt-current) is run</p> <p>I'm not sure if and how you could see this on a CSIT for an un-merged changed. If I were you, I wouldn't worry and just get it in... If this is still seen, this JIRA issue will simply get re-openend! <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/wink.png" height="16" width="16" align="absmiddle" alt="" border="0"/> But if you do want to check, reach out to <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jluhrsen" class="user-hover" rel="jluhrsen">jluhrsen</a> re. where you can see the trace:transaction results on a CSIT.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=xcheara" class="user-hover" rel="xcheara">xcheara</a> could you plz cherry-pick c/72710 to stable/oxygen as well (once and after it has +1/2 and merged to master) ? I just had a look and the bad code was already there as well.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jluhrsen" class="user-hover" rel="jluhrsen">jluhrsen</a> <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=xcheara" class="user-hover" rel="xcheara">xcheara</a> I was just discussing this one with <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=skitt" class="user-hover" rel="skitt">skitt</a> who mentioned re. the newWriteOnlyTransaction that they were already using the SingleTransactionDataBroker there (the c/72710/ fix is just for the ReadTransaction), so that's very curious... <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jluhrsen" class="user-hover" rel="jluhrsen">jluhrsen</a> are we sure that we're doing "trace:transaction" at the "end" in that CIST in <a href="https://jira.opendaylight.org/browse/NETVIRT-1089" title="Add trace:transactions to suite teardowns" class="issue-link" data-issue-key="NETVIRT-1089">NETVIRT-1089</a>, when the system should be "calmed down" and "done" ? I understand this is a vague- but it's of course "normal" for some transactions to still be open while it's still "doing something". Otherwise, I guess what could be happening is that for some reason things are "stuck" inside a job. We would ideally need to see a jstack taken at the same time the "trace:transaction" was taken to be really sure what's going on - would that be possible?</p> <p>May be we should split this Jira into separate issues for the WriteOnlyTransaction VS the ReadTransaction problems listed above - does someone want to do that? </p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=vorburger" class="user-hover" rel="vorburger">vorburger</a>, we are doing the trace:transaction at the end of <b>every</b> test case, as well as in every suite's teardown stage So, <br/> I'm guessing there will be times where the system is still chugging away at something so we'd end up getting some listing of<br/> transactions open that should be and will be closed shortly after. But, that means someone could go sift through all the test<br/> cases if they wanted to see when a transaction first showed up and see when it eventually went away, if that's the idea.</p> <p>However, you could also just look at the <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/89/jamo-traces-netvirt-csit-1node-openstack-queens-upstream-stateful-fluorine/1/robot-plugin/log_full.html.gz#s1-s10-k2-k3-k1-k1-k7" class="external-link" target="_blank" rel="nofollow noopener">final suite's teardown</a> which happened aprox 3s after what I think is the last external event<br/> that could possibly tickle things, which is just a router list. And before that, it's just other reads for some time. It was probably<br/> ~30s before there was anything other than reads happening (a security group delete happened).</p> <p>as for a "jstack", where is that done and what are the details for doing it? Maybe I can add it to my patch and run<br/> this job in the sandbox again.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jluhrsen" class="user-hover" rel="jluhrsen">jluhrsen</a>, based on above logs, we have addressed the closing on</p> <p>newReadOnlyTransaction(). Will it be possible to ran the suite with this patchset changes - <a href="https://git.opendaylight.org/gerrit/#/c/72710/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/72710/</a></p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=xcheara" class="user-hover" rel="xcheara">xcheara</a>, you want the suite with "trace:transactions" to be run, correct?</p> <p>Yes. Basically want to verify no more occurrence of Read Transcations opened with this patch.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=xcheara" class="user-hover" rel="xcheara">xcheara</a> is thsi still being worked?</p> <p>Yes Sam. I will rebase this patch and make it ready for review/merge.</p> <p>This identified issue been merged. </p> <p><a href="https://git.opendaylight.org/gerrit/#/c/netvirt/+/72710/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/netvirt/+/72710/</a></p> <p>Please veriify if it's still an open issue in magnesium.</p> Blocks NETVIRT-1089 Development Rank 0|i03fa7: