https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/NETVIRT-878 netvirt <p><a href="https://jira.opendaylight.org/browse/CONTROLLER-1760" title="Tooling to find the real root cause culprit of memory leaks related to non-closed transactions (and tx chains)" class="issue-link" data-issue-key="CONTROLLER-1760"><del>CONTROLLER-1760</del></a>'s new "tooling to find the real root cause culprit of memory leaks related to non-closed transactions" should ideally be run during CSIT already, something like this:</p> <p>At the start, before you install netvirt odl-netvirt-openstack, if you could add a "feature:install odl-mdsal-trace" and then after <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/warning.png" height="16" width="16" align="absmiddle" alt="" border="0"/> that's through a "feature:install --no-auto-refresh odl-netvirt-openstack" (due to <a href="https://jira.opendaylight.org/browse/CONTROLLER-1764" title="Karaf 4: odl-mdsal-trace cannot &quot;just&quot; be installed e.g. before odl-netvirt-openstack anymore, as it used to" class="issue-link" data-issue-key="CONTROLLER-1764"><del>CONTROLLER-1764</del></a>). Then do loads of interesting stuff, and then at the end do "trace:transactions", and if that returns anything else than that 1 line "all good" message, then start crying like a baby and fail CSIT (and show the output from trace:transactions, which will have long details about what transactions were not closed) ...</p> <p>Operating System: All<br/> Platform: All</p> NETVIRT-878

CSIT should help to detect possible memory leaks leading to OOM related to non-closed transactions (and tx chains) early

Improvement Medium In Progress Unresolved Srinivas Rachakonda Michael Vorburger Thu, 31 Aug 2017 13:53:51 +0000 Wed, 22 Jan 2020 16:24:03 +0000 Nitrogen Fluorine-SR2 Neon General 0 5 <p>I'm not 100% sure if on master after we're fully through with <a href="https://jira.opendaylight.org/browse/NETVIRT-985" title="java.lang.OutOfMemoryError: Java heap space" class="issue-link" data-issue-key="NETVIRT-985"><del>NETVIRT-985</del></a> you'll truly hit 0 output on the output of the trace:transaction CLI command - but I think the best is to "just do it" and then before merging the CSIT extension doing this shout and let me know - I'll then at that point fix or whitelist / exclude whatever you still see.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jluhrsen" class="user-hover" rel="jluhrsen">jluhrsen</a> has now started work related to this on <a href="https://git.opendaylight.org/gerrit/#/q/topic:feature-install" class="external-link" target="_blank" rel="nofollow noopener">Gerrit topic feature-install</a>.</p> <p>It looks like the changes to install this mdsal trace and then netvirt afterward are causing some<br/> CSIT failures. Here is a sandbox [link|<a href="https://jenkins.opendaylight.org/sandbox/job/netvirt-csit-1node-openstack-ocata-jamo-upstream-stateful-nitrogen/1/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/sandbox/job/netvirt-csit-1node-openstack-ocata-jamo-upstream-stateful-nitrogen/1/</a>] (it will be invalid in one week).</p> <p>Essentially, the high level problem is that openstack instances are not getting ips. Digging a little deeper it doesn't look like<br/> the OVS instances are getting programmed with flows.</p> <p>any ideas?</p> <p> </p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jluhrsen" class="user-hover" rel="jluhrsen">jluhrsen</a> as per private IRC chat, more as a FTR for myself: That job (above) is for nitrogen, but the trace:transactions is not available for Nitrogen anyway (I've never back patched it). I therefore propose to focus this only on Oxygen - even if your first phase isn't even about trace:transactions just yet, it will later have to be, so might as well focus only on one release.</p> <p>The only thing in <a href="https://logs.opendaylight.org/sandbox/vex-yul-odl-jenkins-2/netvirt-csit-1node-openstack-ocata-jamo-upstream-stateful-nitrogen/1/odl_1/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">the Karaf log of your Nitrogen CSIT job for this</a> is this error:</p> <p><tt>2018-01-08 19:00:27,797 | WARN | ender-2-thread-1 | AbstractLifeCycle | 282 - org.eclipse.jetty.util - 9.2.21.v20170120 | FAILED HttpServiceContext ... httpContext=WebAppHttpContext ... org.opendaylight.neutron.northbound-api - 374 ... java.lang.NoClassDefFoundError: javax/ws/rs/ext/MessageBodyReader java.lang.NoClassDefFoundError: javax/ws/rs/ext/MessageBodyReader</tt></p> <p>perhaps that is causing the 404 you wrote on IRC you saw in neutron log:</p> <p><tt>DEBUG networking_odl.common.client <span class="error">&#91;-&#93;</span> Exception from ODL: 404 Client Error: Not Found for url: <a href="http://10.30.170.91:8080/controller/nb/v2/neutron/ports" class="external-link" target="_blank" rel="nofollow noopener">http://10.30.170.91:8080/controller/nb/v2/neutron/ports</a></tt></p> <p>If we are seeing this NoClassDefFoundError on Oxygen master as well, and if we are only seeing it when you first feature:install the odl-mdsal-trace then we would have to dig into that... but the easiest then is probably just to wait for <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=skitt" class="user-hover" rel="skitt">skitt</a>'s fix for <a href="https://jira.opendaylight.org/browse/CONTROLLER-1764" title="Karaf 4: odl-mdsal-trace cannot &quot;just&quot; be installed e.g. before odl-netvirt-openstack anymore, as it used to" class="issue-link" data-issue-key="CONTROLLER-1764"><del>CONTROLLER-1764</del></a> to land and then just add odl-mdsal-trace as the first in the list of boot features...</p> <p>if using featuresBoot, I don't see the problem loading neutron NB API, but it does show up for me even locally doing the<br/> steps with feature:install. I know we can just change our scripts to simply use featuresBoot, but it's probably a good excuse<br/> for us to figure out why this is happening when we feature:install. Maybe there is some bug we can fix.</p> <p>I do like having the option of doing feature:install vs featuresBoot in our infra, so no need to stop that work. It's pretty much<br/> ready at this point, but I don't want to merge the netvirt part of it since we know it will break things.</p> <p>btw, just to be clear, we did run this job with oxygen and saw the same 14 failures:<br/> <a href="https://jenkins.opendaylight.org/sandbox/job/netvirt-csit-1node-openstack-ocata-jamo-upstream-stateful-oxygen/2/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/sandbox/job/netvirt-csit-1node-openstack-ocata-jamo-upstream-stateful-oxygen/2/</a></p> <p><a href="https://jira.opendaylight.org/browse/NEUTRON-153" title="Neutron NB API feature fails to install when combining with odl-mdsal-trace" class="issue-link" data-issue-key="NEUTRON-153"><del>NEUTRON-153</del></a> will look into the problem loading neutron NB API with features:install.</p> <p>I suggest that in this issue we move forward using featuresBoot instead of feature:install.</p> <p>with this <a href="https://git.opendaylight.org/gerrit/c/68448/" class="external-link" target="_blank" rel="nofollow noopener">builder patch</a> to install odl-mdsal-trace as an additional feature, and running the l2 connectivity suite with this <a href="https://git.opendaylight.org/gerrit/c/68447/" class="external-link" target="_blank" rel="nofollow noopener">int/test patch</a> we get the following <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/17/jamo-netvirt-csit-1node-openstack-pike-upstream-stateful-oxygen/11/robot-plugin/log_full.html.gz#s1-k2-k3-k1-k1-k6" class="external-link" target="_blank" rel="nofollow noopener">output</a>:</p> <blockquote><p><tt>[?1l&gt;[?2004lTracingBroker found some not yet (or never..) closed transaction<span class="error">&#91;chain&#93;</span>s! <span class="error">&#91;NB: If no stack traces are shown below, then enable transaction-debug-context-enabled in mdsaltrace_config.xml&#93;</span> DataBroker : createTransactionChain() 3x TransactionChains opened but not closed here: (...) org.opendaylight.controller.md.sal.dom.broker.impl.PingPongTransactionChain.&lt;init&gt;(PingPongTransactionChain.java:104) org.opendaylight.controller.md.sal.dom.broker.impl.PingPongDataBroker.createTransactionChain(PingPongDataBroker.java:49) org.opendaylight.controller.md.sal.dom.broker.impl.PingPongDataBroker.createTransactionChain(PingPongDataBroker.java:28) (...) org.opendaylight.controller.md.sal.binding.impl.BindingDOMTransactionChainAdapter.&lt;init&gt;(BindingDOMTransactionChainAdapter.java:45) org.opendaylight.controller.md.sal.binding.impl.BindingDOMDataBrokerAdapter.createTransactionChain(BindingDOMDataBrokerAdapter.java:74) (...) org.opendaylight.openflowplugin.common.txchain.TransactionChainManager.createTxChain(TransactionChainManager.java:81) org.opendaylight.openflowplugin.common.txchain.TransactionChainManager.activateTransactionManager(TransactionChainManager.java:109) org.opendaylight.openflowplugin.impl.device.DeviceContextImpl.lazyTransactionManagerInitialization(DeviceContextImpl.java:649) org.opendaylight.openflowplugin.impl.device.DeviceContextImpl.instantiateServiceInstance(DeviceContextImpl.java:592) org.opendaylight.openflowplugin.impl.lifecycle.GuardedContextImpl.instantiateServiceInstance(GuardedContextImpl.java:86) java.util.concurrent.CopyOnWriteArrayList.forEach(CopyOnWriteArrayList.java:891) org.opendaylight.openflowplugin.impl.lifecycle.ContextChainImpl.instantiateServiceInstance(ContextChainImpl.java:74) org.opendaylight.mdsal.singleton.dom.impl.ClusterSingletonServiceGroupImpl.lambda$startServices$0(ClusterSingletonServiceGroupImpl.java:648) java.util.ArrayList.forEach(ArrayList.java:1257) org.opendaylight.mdsal.singleton.dom.impl.ClusterSingletonServiceGroupImpl.startServices(ClusterSingletonServiceGroupImpl.java:645) org.opendaylight.mdsal.singleton.dom.impl.ClusterSingletonServiceGroupImpl.cleanupCandidateOwnershipChanged(ClusterSingletonServiceGroupImpl.java:506) org.opendaylight.mdsal.singleton.dom.impl.ClusterSingletonServiceGroupImpl.lockedOwnershipChanged(ClusterSingletonServiceGroupImpl.java:453) org.opendaylight.mdsal.singleton.dom.impl.ClusterSingletonServiceGroupImpl.ownershipChanged(ClusterSingletonServiceGroupImpl.java:433) org.opendaylight.mdsal.singleton.dom.impl.AbstractClusterSingletonServiceProviderImpl.ownershipChanged(AbstractClusterSingletonServiceProviderImpl.java:234) org.opendaylight.mdsal.singleton.dom.impl.DOMClusterSingletonServiceProviderImpl.ownershipChanged(DOMClusterSingletonServiceProviderImpl.java:23) org.opendaylight.controller.cluster.datastore.entityownership.EntityOwnershipListenerActor.onEntityOwnershipChanged(EntityOwnershipListenerActor.java:44) org.opendaylight.controller.cluster.datastore.entityownership.EntityOwnershipListenerActor.handleReceive(EntityOwnershipListenerActor.java:33) org.opendaylight.controller.cluster.common.actor.AbstractUntypedActor.onReceive(AbstractUntypedActor.java:38) (...) [?1h=[?2004hopendaylight-user@root&gt;</tt></p></blockquote> <p> </p> <p>what can we make of this so far? I think we can probably merge the jjb and test<br/> patches as a start and then figure out if we want to start failing things if we<br/> get certain outputs. I think the idea was that we should never see any output,<br/> unless there is a bug being uncovered.</p> <p>&gt; what can we make of this so far?</p> <p>a real possible transaction leak in openflowplugin which could lead to OOM on longevity, or a false positive of the <tt>TracingBroker</tt> (<tt>trace:transaction</tt>) .. watch <a href="https://jira.opendaylight.org/browse/OPNFLWPLUG-982" title="Suspected TransactionChain leak in TransactionChainManager (or fix false positive in TracingBroker)" class="issue-link" data-issue-key="OPNFLWPLUG-982"><del>OPNFLWPLUG-982</del></a> for more on that front.</p> <p>&gt; I think we can probably merge the jjb and test<br/> &gt; patches as a start and then figure out if we want to start failing things if we<br/> &gt; get certain outputs. I think the idea was that we should never see any output,<br/> &gt; unless there is a bug being uncovered.</p> <p>confirming this, that is the goal here (and we should keep this issue open until we have achieved that). But I would start merging things gradually, and not wait for the full solution, but have as a goal to enforce failing CSIT when we see any output from <tt>trace:transactions</tt> after test run within a reasonable time frame - couple of weeks, perhaps?</p> <p>But a question: Is <a href="https://jira.opendaylight.org/browse/OPNFLWPLUG-982" title="Suspected TransactionChain leak in TransactionChainManager (or fix false positive in TracingBroker)" class="issue-link" data-issue-key="OPNFLWPLUG-982"><del>OPNFLWPLUG-982</del></a> just the first or the only output, after all of / a full netvirt CSIT? If it's the latter, then things re. Tx leaks (=OOMs) currently aren't so bad actually... that would mean that no OOM related regressions have been introduced since our last scale test (or they are in scenarios which the CSIT does not cover).</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=vorburger" class="user-hover" rel="vorburger">vorburger</a> <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jluhrsen" class="user-hover" rel="jluhrsen">jluhrsen</a> is there anything else for this issue?</p> <blockquote><p>Michael Vorburger Jamo Luhrsen is there anything else for this issue?</p></blockquote> <p>yeah, I need to finish <a href="https://jira.opendaylight.org/browse/NETVIRT-1089" class="external-link" rel="nofollow">this</a>. We had it in for a bit, but it caused the logging<br/> issues, so we reverted it. I'll take a look at getting it back soon.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=shague" class="user-hover" rel="shague">shague</a> yeah please don't close this one, this is <b>NOT</b> done - and we really have to finally do this!!</p> <p>Thanks for confirming what I said, <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=vorburger" class="user-hover" rel="vorburger">vorburger</a>. Just FYI, this is <b>NOT</b> at the top of my to-do list. I <b>DID</b> get<br/> it finished once before, but we hit a bug with a gagillion log messages ruining the deployment, and that quickly<br/> scared us in to reverting it. I know that logging bug was resolved. "I'll take a look at getting it back soon."</p> <p>actually, I forgot about this <a href="https://git.opendaylight.org/gerrit/#/c/68980/" class="external-link" target="_blank" rel="nofollow noopener">patch</a>. Running it in the sandbox<br/> now and depending on what that looks like, we might be able to get this in without much effort.</p> <p>the recent two sandbox jobs I ran with this <a href="https://git.opendaylight.org/gerrit/#/c/68980/" class="external-link" target="_blank" rel="nofollow noopener">CSIT patch</a> and odl-mdsal-trace feature have failures which we<br/> do not expect to see. I have not had time to dig in to the failures, but until we get those figured out, we<br/> can't move forward.</p> <p>two sandbox failures:<br/> <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/35/jamo-netvirt-csit-1node-openstack-queens-upstream-stateful-oxygen/2/robot-plugin/log_full.html.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/35/jamo-netvirt-csit-1node-openstack-queens-upstream-stateful-oxygen/2/robot-plugin/log_full.html.gz</a><br/> <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/34/jamo-netvirt-csit-1node-openstack-queens-upstream-stateful-oxygen/3/robot-plugin/log_full.html.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/34/jamo-netvirt-csit-1node-openstack-queens-upstream-stateful-oxygen/3/robot-plugin/log_full.html.gz</a></p> <p>I'll run it a few more times, as that's not really any work; just a few clicks.</p> <p>We really should pick this old and long overdue idea up and finish it, somehow. </p> <p>In parallel (and with a for me higher short term priority) we will also finish <a href="https://jira.opendaylight.org/browse/NETVIRT-1318" title="MDSAL best practices" class="issue-link" data-issue-key="NETVIRT-1318">NETVIRT-1318</a> and start <a href="https://jira.opendaylight.org/browse/GENIUS-176" title="Convert all of genius to managed transactions (datastore-constrained)" class="issue-link" data-issue-key="GENIUS-176">GENIUS-176</a>.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jluhrsen" class="user-hover" rel="jluhrsen">jluhrsen</a> did we want to get this in?</p> <p>yes, but we need two things.</p> <p>1) run csit with this trace:transactions stuff going to verify no weird failures. In the past, it was causing<br/> extra failures. The last time I had time to check, it was not causing trouble.</p> <p>2) modify the step to fail when too many open transactions are there.</p> <p>any update here?</p> <p>the work went in to add "trace:transactions" to each test case teardown, but it looks like it's <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-neon/1541/robot-plugin/log_full.html.gz#s1-s18-t3-k50-k2-k5-k1-k1-k1-k7" class="external-link" target="_blank" rel="nofollow noopener">not working now</a>, but maybe the feature is not installed properly any more? We'd need to find someone to own this jira now and take it forward.</p> <p>once that command is working again, the next step would be to understand if extra transactions are not being closed and then mark the<br/> test case as failed. That will take some thinking and understanding to get right though because there will always be some transactions opened<br/> when you call the command. What we are looking for is the case when transaction count increases over time without going back down which<br/> would be a memory leak scenario and could blow up as an OOM in a long running production environment.</p> <p>Nishchya, can you please look into this?</p> <p>Hi Srini, please checkif it still valid.</p> <p>this jira is not about fixing a bug that you need to check if it's still valid or not. It's about doing the work to add functionality to existing CSIT code<br/> to find memory leaks coming from non-closed transactions. That work never really got finished.</p> Blocks NETVIRT-1010 NETVIRT-1089 CONTROLLER-1760 CONTROLLER-1764 GENIUS-102 OPNFLWPLUG-961 OVSDB-435 OPNFLWPLUG-982 NETVIRT-883 NETVIRT-985 NEUTRON-153 Development External issue ID 9082 External issue URL Rank 0|i01uin: