https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/NETVIRT-512 netvirt <p><a href="https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-mitaka-upstream-transparent-carbon/407/archives/log.html.gz#s1-s2-s1-t11" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-mitaka-upstream-transparent-carbon/407/archives/log.html.gz#s1-s2-s1-t11</a></p> <p>Operating System: All<br/> Platform: All</p> NETVIRT-512

CSIT Sporadic failures - tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_stop_start

Bug Medium Confirmed Unresolved Srinivas Rachakonda Jamo Luhrsen Mon, 6 Mar 2017 17:17:30 +0000 Wed, 27 Nov 2019 07:26:18 +0000 Oxygen General 0 8 <p><a href="https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-newton-nodl-v2-upstream-transparent-carbon/568/log.html.gz#s1-s2-s1-t11" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-newton-nodl-v2-upstream-transparent-carbon/568/log.html.gz#s1-s2-s1-t11</a></p> <p><a href="https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-newton-nodl-v2-upstream-learn-carbon/334/log.html.gz#s1-s2-s1-t11" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-newton-nodl-v2-upstream-learn-carbon/334/log.html.gz#s1-s2-s1-t11</a></p> <p>no longer seen in CSIT</p> <p>This is back now, and we need to track it. Starting it out as an Oxygen Blocker since it</p> <p>was introduced recently and very consistent.</p> <p>it's consistent in <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-queens-upstream-stateful-oxygen/201/robot-plugin/log_05_tempest.html.gz#s1-t6" class="external-link" target="_blank" rel="nofollow noopener">queens</a> as well as <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-pike-upstream-stateful-oxygen/198/robot-plugin/log_05_tempest.html.gz#s1-t6" class="external-link" target="_blank" rel="nofollow noopener">pike</a>. Pike is easier to debug because the openstack services have their logs<br/> separate. Apparently, in devstack queens, the logs are all munged together in the system messages file.</p> <p>some clues:</p> <ul> <li>the tempest test stops the instance and then tries to start it to check connectivity remains. But, the instance<br/> never comes back and is stuck in SHUTOFF state. We collect debugs in each tempest scenario test before<br/> the test case tears down. Here you can see the <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-pike-upstream-stateful-oxygen/198/robot-plugin/log_05_tempest.html.gz#s1-t6-k2-k2-k1-k13-k4-k1-k2" class="external-link" target="_blank" rel="nofollow noopener">server list</a> which shows the instance stuck in SHUTOFF<br/> state.</li> <li>in the compute node <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-pike-upstream-stateful-oxygen/198/compute_1/n-cpu.log.2018-03-05-042326.gz" class="external-link" target="_blank" rel="nofollow noopener">nova-cpu log</a> there is a line: <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> 2018-03-05 05:24:08.014 27550 WARNING nova.virt.libvirt.driver [req-9f7b4144-e42a-49fd-b17b-6a53de727ac8 service nova] [instance: 11442853-e52d-43ff-af61-e028b62cd5ea] Timeout waiting <span class="code-keyword">for</span> vif plugging callback <span class="code-keyword">for</span> instance with vm_state stopped and task_state powering-on.: Timeout: 300 seconds</pre> </div></div> <p>that is coming a couple of minutes after the test has given up and marking it a failure.</p></li> <li>it looks like this showed up sometime between <a href="https://jenkins.opendaylight.org/releng/user/jluhrsen/my-views/view/netvirt%20csit/job/netvirt-csit-1node-openstack-pike-upstream-stateful-oxygen/192/robot/openstack/05%20tempest/tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_stop_start/" class="external-link" target="_blank" rel="nofollow noopener">Feb 28th</a> and <a href="https://jenkins.opendaylight.org/releng/user/jluhrsen/my-views/view/netvirt%20csit/job/netvirt-csit-1node-openstack-pike-upstream-stateful-oxygen/198/robot/openstack/05%20tempest/tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_stop_start/" class="external-link" target="_blank" rel="nofollow noopener">Mar 1st</a> Be careful when you look at the <a href="https://jenkins.opendaylight.org/releng/user/jluhrsen/my-views/view/netvirt%20csit/job/netvirt-csit-1node-openstack-pike-upstream-stateful-oxygen/198/robot/openstack/05%20tempest/tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_stop_start/" class="external-link" target="_blank" rel="nofollow noopener">Mar 1st</a> link because it still shows a failure, but it's not the SHUTOFF failure. It's failing because of one our sporadic connectivity issues. If you look at the <a href="https://jenkins.opendaylight.org/releng/user/jluhrsen/my-views/view/netvirt%20csit/job/netvirt-csit-1node-openstack-pike-upstream-stateful-oxygen/203/robot/openstack/05%20tempest/tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_stop_start/graph?zoomSignificant=false&amp;failedOnly=false&amp;criticalOnly=false&amp;maxBuildsToShow=0&amp;hd=true" class="external-link" target="_blank" rel="nofollow noopener">graph</a> you can see that the test case was always passing, then two sporadic connectivity failures, and finally the rest are all for this SHUTOFF issue.</li> </ul> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=k.faseela" class="user-hover" rel="k.faseela">k.faseela</a> - Any update on this? Is someone actively working on it? Friendly reminder that we need to get this resolved ASAP as it's blocking the imminent Oxygen release.</p> <p>Jamo, the new failure is not anything on the flow programming side, rather the VM is not coming up, right?</p> <p>I don't think I am the right person to debug this, some openstack expertise will be needed.</p> <p>Vishal or Vivekanandan might be able to help?</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=shague" class="user-hover" rel="shague">shague</a> <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jluhrsen" class="user-hover" rel="jluhrsen">jluhrsen</a> : Had a chat with Vishal, he is telling this needs to be looked upon by someone who has nova/neutron expertise. He was suggesting to check with <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=yamahata" class="user-hover" rel="yamahata">yamahata</a></p> <p>Has someone started digging into this? <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=yamahata" class="user-hover" rel="yamahata">yamahata</a>? Maybe <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jhershbe" class="user-hover" rel="jhershbe">jhershbe</a> could help?</p> <p>I've dug around a lot, looking for something obvious, but not really finding anything blatant.</p> <p>There is a difference in some of the nova logs, when I compare a passing job to a failing job.<br/> (attached are the outputs of the compute nodes journalctl.log file for all entries matching the<br/> relevant instances UUID)</p> <p>in the passing job (timestamp Feb 08 07:03:54) you will see that the instance had the lifecycle<br/> event for "VM Resumed". That is not there in the failing job (timestamp: Mar 07 23:00:51), where<br/> lifecycle event is "VM Started". In the failing log, you can next see (less than a second later) that<br/> the instance gets a "VM Paused" event. I'm assuming that's what sticks it to SHUTOFF state<br/> and ruins the test.</p> <p>I'm trying to think of ways to rule out OpenDaylight as the culprit, so we can move this from<br/> blocker state, but we need to know why that instance got quickly cycled off after trying to be<br/> restarted.</p> <p> </p> <p>I was wondering if this could somehow be a bug outside of ODL, maybe in the openstack side of things.<br/> I tried to run the Queens jobs, but rolling things back to mid-Feb-ish (when the bug wasn't happening)<br/> but that did not help (I rolled back devstack, networking-odl, neutron and nova).</p> <p>However, I did run Oxygen with the Ocata branch of openstack and the bug did not show up. log <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/22/jamo-netvirt-csit-1node-openstack-ocata-upstream-stateful-oxygen/1/robot-plugin/log_04_tempest.html.gz" class="external-link" target="_blank" rel="nofollow noopener">here</a></p> <p>I'm not sure what to make of this yet.</p> <p>So I also suspect that this might be an openstack issue or even a libvirt issue. When I looked at this last week I noticed this test was failing on the start (unpause) of the VM after it was stopped. Turns out, neutron and ODL are not involved in that process at all.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jluhrsen" class="user-hover" rel="jluhrsen">jluhrsen</a> - Is pushing on this hard, looking like it's an OpenStack bug so far, will update by EOD PT.</p> <blockquote><p>Jamo Luhrsen - Is pushing on this hard, looking like it's an OpenStack bug so far, will update by EOD PT.</p></blockquote> <p>It took me all day to get my local setup working, and did not get this specific test to run yet. I'll hopefully get some time in my<br/> late evening to work on it, but no specific news to update with now as I had intended. stay tuned...</p> <p>There are earlier jobs with the failure back to 2/8/18:</p> <p><a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-queens-upstream-stateful-oxygen/144/robot-plugin/log_05_tempest.html.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-queens-upstream-stateful-oxygen/144/robot-plugin/log_05_tempest.html.gz</a></p> <p>pike: <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-pike-upstream-stateful-oxygen/190/robot-plugin/log_05_tempest.html.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-pike-upstream-stateful-oxygen/190/robot-plugin/log_05_tempest.html.gz</a></p> <ul class="alternate" type="square"> <li>but the pike looks like a different failure in the test</li> </ul> <p>This is not a bug with OpenDaylight itself. I have removed this from blocker, but leaving the bug open for now<br/> until we have the appropriate openstack bug filed and tracked.</p> <p>It appears this <a href="https://review.openstack.org/#/c/542738/" class="external-link" target="_blank" rel="nofollow noopener">patch</a> in nova is what introduced the failure. However, it's possible (and theorized by Mohamed Naser) that<br/> this could be a problem in networking_odl not actually unplugging the port when it should. I'm not totally understanding those<br/> details yet. However, when we run our <a href="https://logs.opendaylight.org/sandbox/vex-yul-odl-jenkins-2/jamo-netvirt-csit-1node-openstack-queens-upstream-stateful-oxygen/4/robot-plugin/log_01_tempest.html.gz#s1-t2" class="external-link" target="_blank" rel="nofollow noopener">CSIT</a> by rolling back nova to a commit hash just before the patch mentioned earlier,<br/> the failure goes away.</p> <p>We've also noticed that non-voting tempest tests that run against networking-odl also started failing around the same time<br/> that this nova patch landed.</p> <p><a href="https://review.openstack.org/553035" class="external-link" target="_blank" rel="nofollow noopener">https://review.openstack.org/553035</a> was merged to revert the original nova patch that was causing the problem. running new gates to reenable the test and see if it is fixed.</p> <p>New patch to fix original issue in nova: <a href="https://review.openstack.org/558001" class="external-link" target="_blank" rel="nofollow noopener">https://review.openstack.org/558001</a></p> <p>any update here?</p> <p>I'm not keeping track of netvirt jobs any more. Can we get someone to go through the tempest jobs and see if this test case is still failing sporadically or not?</p> <p>actually it's not that hard to figure out. here is a <a href="https://jenkins.opendaylight.org/releng/view/netvirt-csit/job/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-rocky-tempest-stateful-sodium/210/robot/openstack/tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps.test_server_connectivity_stop_start/" class="external-link" target="_blank" rel="nofollow noopener">link for the sodium job</a> that shows that it does still fail sporadically.</p> Development External issue ID 7909 External issue URL Rank 0|i01s9b: Status Whiteboard csit:sporadic_failures