https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/NETVIRT-1089 netvirt <p>initially, just add the call. we can save the results to a file and log them like we do with other<br/> things like packet traces, exceptions, etc.</p> <p>but hopefully, at some point, we can fail if the output is non-empty, which probably<br/> indicates some leftover unfinished (aka leaking) transactions</p> NETVIRT-1089

Add trace:transactions to suite teardowns

Sub-task NETVIRT-1086 Medium In Progress Unresolved Jamo Luhrsen Jamo Luhrsen Tue, 13 Feb 2018 22:33:56 +0000 Fri, 21 Dec 2018 19:24:26 +0000 Fluorine-SR2 Neon 0 3 <p>I was remembering a jira ticket about this task that should have comments or updates. this one is<br/> blank. Oh, I see it. It's <a href="https://jira.opendaylight.org/browse/NETVIRT-878" title="CSIT should help to detect possible memory leaks leading to OOM related to non-closed transactions (and tx chains) early" class="issue-link" data-issue-key="NETVIRT-878">NETVIRT-878</a> which is mentioned on this ticket to be blocked by this.</p> <p>Anyway, <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=smalleni" class="user-hover" rel="smalleni">smalleni</a> mentioned he might want to look at this as a task to get familiar with CSIT,<br/> jenkins, robot, sandbox, etc etc. So, I added him as a watcher.</p> <p>However, we were able to quickly run this <a href="https://git.opendaylight.org/gerrit/#/c/68980/" class="external-link" target="_blank" rel="nofollow noopener">patch</a> in the sandbox again and I can confirm again that it <br/> introduces some <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/67/jamo-netvirt-csit-1node-openstack-queens-upstream-stateful-oxygen/3/robot-plugin/log_full.html.gz" class="external-link" target="_blank" rel="nofollow noopener">troubles/failures </a> that we don't expect. we ran the same job without the trace:transactions<br/> happening on every test case teardown and it's <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/68/jamo-netvirt-csit-1node-openstack-queens-upstream-stateful-oxygen/4/robot-plugin/log_full.html.gz" class="external-link" target="_blank" rel="nofollow noopener">100% passing </a>. I have not dug in to the what/why/how, but<br/> clearly doing this trace:transactions is having some kind of negative effect on our jobs. Maybe there is<br/> some simple explanation and fix, but we have to figure this out before we can get this in to our tests.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=smalleni" class="user-hover" rel="smalleni">smalleni</a>, feel like digging around?</p> <p>I'm interested in getting this done, so motivated to help. Had a look at the "troubles/failures" linked above - 13 CSIT failures? What could perhaps be useful as a next step is if one of you could look through the karaf.log (<a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/67/jamo-netvirt-csit-1node-openstack-queens-upstream-stateful-oxygen/3/odl_1/" class="external-link" target="_blank" rel="nofollow noopener">is it here?</a>), and somehow <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/help_16.png" height="16" width="16" align="absmiddle" alt="" border="0"/> compare a failing one with trace:transaction with one without where CSIT passes 100% - if you spot any errors / exceptions there which only happen with trace:transactions, I may be able to understand the cause. (In theory, it should not cause any functional difference, as the trace feature just installs a kind of wrapper.)</p> <p>ok, so first of all <b>MY BAD</b> in that the job I linked to with failures did not actually even install the odl-mdsal-trace feature.<br/> it was running "trace:transactions" on each test case though resulting in a command not found. And just doing that was<br/> enough to cause failures.</p> <p>I did look at the karaf.log (yes, <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=vorburger" class="user-hover" rel="vorburger">vorburger</a>, you have the right place for those logs) but nothing stood out.</p> <p>The first few failures are end-user kind of failures where openstack instances have connectivity problems. We could<br/> try to dig in on the debugs there to see why that is. But, the later failures, like this <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/67/jamo-netvirt-csit-1node-openstack-queens-upstream-stateful-oxygen/3/robot-plugin/log_full.html.gz#s1-s3" class="external-link" target="_blank" rel="nofollow noopener">one</a> looks like a test system<br/> issue where we lost the ssh connection to one of our test VMs. That seems like something to chase. I wonder if our<br/> step to do trace:transactions, which is done via ssh, is building up more connection cruft than we can handle eventually.<br/> Or we are hitting some ssh bug in robot. Btw, we do <a href="https://github.com/opendaylight/integration-test/blob/master/csit/libraries/KarafKeywords.robot#L43" class="external-link" target="_blank" rel="nofollow noopener">close</a> the ssh connection each time we run trace:transactions.</p> <p>Anyway, just thinking out loud for now.</p> <p>I am running this again <a href="https://jenkins.opendaylight.org/sandbox/job/jamo-mdsaltrace-netvirt-csit-1node-openstack-queens-upstream-stateful-oxygen/" class="external-link" target="_blank" rel="nofollow noopener">here</a> so lets wait on that before we dig any further. yes, I am running it with odl-mdsal-trace.</p> <p>the new job finished, had failures and really did do the mdsal-trace feature install.</p> <p>All the logs are <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/79/" class="external-link" target="_blank" rel="nofollow noopener">here</a></p> <p>I have not dug very deep, but I did notice a few things:</p> <ul> <li>lots of <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/79/jamo-mdsaltrace-netvirt-csit-1node-openstack-queens-upstream-stateful-oxygen/1/odl_1/odl1_exception.log.gz" class="external-link" target="_blank" rel="nofollow noopener">exceptions </a> and to compare this is the <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-queens-upstream-stateful-oxygen/675/odl_1/odl1_exception.log.gz" class="external-link" target="_blank" rel="nofollow noopener">same job</a> without this trace stuff to look at. <ul> <li>note that one is a "frozen class" which I recall seeing some email thread about. I think we have a red hat bugzilla around that (if it's the same)</li> <li>a pretty serious <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/79/jamo-mdsaltrace-netvirt-csit-1node-openstack-queens-upstream-stateful-oxygen/1/robot-plugin/log_full.html.gz#s1-s1-t8-k6" class="external-link" target="_blank" rel="nofollow noopener">netvirt functional failure </a> where all instances in one network did not receive their IPs but another network was ok.</li> <li>looks like the same problem with ssh connections that I mentioned in the previous comment is also happening here as well</li> <li>in the final call to trace:transactions we can see a LOT (I think it's a lot) of <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/79/jamo-mdsaltrace-netvirt-csit-1node-openstack-queens-upstream-stateful-oxygen/1/robot-plugin/log_full.html.gz#s1-s10-t12-k11-k3-k1-k1-k6" class="external-link" target="_blank" rel="nofollow noopener">unclosed transactions </a></li> </ul> </li> </ul> <p>let me know what you think <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=vorburger" class="user-hover" rel="vorburger">vorburger</a> and we can think of ideas to keep digging.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jluhrsen" class="user-hover" rel="jluhrsen">jluhrsen</a> as per internal / private email: What I'm most interested in first here (we'll get back to transaction leaks after) is that you've managed to hit "frozen class" <a href="https://jira.opendaylight.org/browse/MDSAL-213" title="Serializing DataObject to JSON causes frozen class exception" class="issue-link" data-issue-key="MDSAL-213"><del>MDSAL-213</del></a> here! Could your re-run this and turn but up the logging for class org.opendaylight.mdsal.binding.dom.codec.gen.impl.AbstractStreamWriterGenerator all the way up to TRACE (hoping that doesn't lead to too much, we'll see; otherwise just do DEBUG), because I <a href="https://git.opendaylight.org/gerrit/#/c/71890/" class="external-link" target="_blank" rel="nofollow noopener">recently added additional new logging re. Frozen</a> (not the movie but the bug), and provide that log on <a href="https://jira.opendaylight.org/browse/MDSAL-213" title="Serializing DataObject to JSON causes frozen class exception" class="issue-link" data-issue-key="MDSAL-213"><del>MDSAL-213</del></a> ?</p> <p>Just for the record here and as previously stated elsewhere: It COULD be that frozen ONLY happens with features-mdsal-trace... If you are able to try it once completely without and once with odl-mdsal-trace, we (<a href="https://jira.opendaylight.org/browse/MDSAL-213" title="Serializing DataObject to JSON causes frozen class exception" class="issue-link" data-issue-key="MDSAL-213"><del>MDSAL-213</del></a>) would be most curious whether that makes any difference. See also <a href="https://jira.opendaylight.org/browse/CONTROLLER-1832" title="Transaction Trace tool wiring creates second BindingToNormalizedNodeCodec" class="issue-link" data-issue-key="CONTROLLER-1832"><del>CONTROLLER-1832</del></a> and <a href="https://jira.opendaylight.org/browse/CONTROLLER-1834" title="Transaction Trace tool wiring for ping-pong DataBroker is wrong" class="issue-link" data-issue-key="CONTROLLER-1834"><del>CONTROLLER-1834</del></a> which are vaguely related.</p> <p>&gt; in the final call to trace:transactions we can see a LOT (I think it's a lot) of <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/79/jamo-mdsaltrace-netvirt-csit-1node-openstack-queens-upstream-stateful-oxygen/1/robot-plugin/log_full.html.gz#s1-s10-t12-k11-k3-k1-k1-k6" class="external-link" target="_blank" rel="nofollow noopener">unclosed transactions</a></p> <p>I've opened <a href="https://jira.opendaylight.org/browse/NETVIRT-1295" title="Transaction leak in ExternalRoutersListener" class="issue-link" data-issue-key="NETVIRT-1295"><del>NETVIRT-1295</del></a> and <a href="https://jira.opendaylight.org/browse/NETVIRT-1296" title="Transaction leak in VpnOpStatusListener" class="issue-link" data-issue-key="NETVIRT-1296"><del>NETVIRT-1296</del></a> <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=shague" class="user-hover" rel="shague">shague</a> will you drive someone to "plug" those in netvirt weekly meetings?</p> <p>working on it with this <a href="https://jenkins.opendaylight.org/sandbox/job/jamo-traces-netvirt-csit-1node-openstack-queens-upstream-stateful-fluorine/" class="external-link" target="_blank" rel="nofollow noopener">sandbox job</a></p> <p>I ran one job with the <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/89/jamo-traces-netvirt-csit-1node-openstack-queens-upstream-stateful-fluorine/1/" class="external-link" target="_blank" rel="nofollow noopener">mdsal-trace feature installed</a> and one job <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/90/jamo-traces-netvirt-csit-1node-openstack-queens-upstream-stateful-fluorine/2/" class="external-link" target="_blank" rel="nofollow noopener">without that feature</a>. It looks like the failures are the same, so I no longer suspect this feature is causing any more problems. I'm guessing the failures (which aren't expected) are coming because of all the extra trace:transactions commands via ssh to karaf console. We'll have to dig in to that idea next.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=vorburger" class="user-hover" rel="vorburger">vorburger</a> pinged me on IRC if this jira was about fluorine and/or oxygen. this is an int/test patch and by<br/> default will apply to both. If we don't want it in one or the other, that's possible. what's the thinking, <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=vorburger" class="user-hover" rel="vorburger">vorburger</a>?</p> <p>Running this on top Fluorine AND Oxygen is great IMHO!</p> <p><a href="https://jira.opendaylight.org/browse/CONTROLLER-1765" title="trace:transactions should suppress false TransactionChain positives, by white list, or showing only x2+ ?" class="issue-link" data-issue-key="CONTROLLER-1765"><del>CONTROLLER-1765</del></a> is needed to finish up being able to fully automate this.</p> <p>I finally had some time to look at this one again.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=vorburger" class="user-hover" rel="vorburger">vorburger</a>, over in <a href="https://jira.opendaylight.org/browse/CONTROLLER-1765" title="trace:transactions should suppress false TransactionChain positives, by white list, or showing only x2+ ?" class="issue-link" data-issue-key="CONTROLLER-1765"><del>CONTROLLER-1765</del></a> you guessed at using some number 7 as a threshold for which we might want to flag a<br/> problem with a failure in csit. in our standard netvirt csit job, by the 2nd suite we are already seeing <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/494/jamo-netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-snat-conntrack-oxygen/4/robot-plugin/log_full.html.gz#s1-s2-k1-k37-k1-k5-k1-k1-k7" class="external-link" target="_blank" rel="nofollow noopener">14 open transactions</a>. By the final suite, that number has grown to 114. I'm guessing that indicates a minor leak. what do you think?</p> <p>Looks like there are other open transactions from the get go, but they do not increase. So possibly those are leak-able, but just not with the normal netvirt csit flow.</p> <p>Unfortunately, something about this work (installing odl-mdsal-trace and issuing that karaf cli on each test case teardown) is causing a problem<br/> with something unrelated. I have to figure that out before we can get this really working.</p> <p>but, in the meantime we probably want to figure out if we have a leak to fix, and if not, how to deal with it (growing from<br/> 7-114) in our csit job.</p> <p>I've opened <a href="https://jira.opendaylight.org/browse/OPNFLWPLUG-1043" title="Transaction leak in ForwardingRulesManagerImpl (ReadOnly)" class="issue-link" data-issue-key="OPNFLWPLUG-1043"><del>OPNFLWPLUG-1043</del></a> but FYI it's newReadOnlyTransaction - less critical, AFAIK.</p> <p>great thanks.</p> <p>if it's less critical (as in, wont get fixed) then we'll need to ignore it when checking because we don't want to have<br/> these things failing our csit which is our gates. maybe that makes it slightly more critical <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>~2m later, taking another look. Trying to put this in the apex job first.</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/78883/" class="external-link" target="_blank" rel="nofollow noopener">step one complete </a><br/> <a href="https://git.opendaylight.org/gerrit/#/c/78970/" class="external-link" target="_blank" rel="nofollow noopener">step two looks good so far</a><br/> <a href="https://git.opendaylight.org/gerrit/#/c/77363/" class="external-link" target="_blank" rel="nofollow noopener">step three needs some iterations </a></p> Blocks NETVIRT-878 MDSAL-213 CONTROLLER-1765 NETVIRT-1296 CONTROLLER-1832 CONTROLLER-1834 NETVIRT-1295 OPNFLWPLUG-1043 Development Rank 0|i03abb: