https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/NETVIRT-1212 netvirt <p>OVS 2.9.0 has issues with group tables collecting packets. This fails many of the csit tests. The bug below details the problem and the fix. The fix has been upstreamed but a newer OVS version has not been released yet.</p> <p><a href="https://bugzilla.redhat.com/show_bug.cgi?id=1544892" class="external-link" target="_blank" rel="nofollow noopener">https://bugzilla.redhat.com/show_bug.cgi?id=1544892</a></p> <p>The upstream fix is: <a href="https://github.com/openvswitch/ovs/commit/058d3a4afddd4db6bf052c71ff431a59fa5a0335" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/openvswitch/ovs/commit/058d3a4afddd4db6bf052c71ff431a59fa5a0335</a>. Erics comment about the patches pushed upstream:<br/> Applied upstream:</p> <p> 14ebc6c199a7 ("tests/ofproto-dpif: New test for action_set after traversing patch port")<br/> 118b21d93f55 ("ofproto-dpif-xlate: translate action_set in clone action")</p> <p>It has also been backported to branch-2.9.</p> <p>A possible workaround is to install the ovs package mentioned in the bugzilla. The basic idea is to add the repo config and then install ovs from that repo. This could be done in a couple ways:</p> <p>1. In the current csit vm images, ovs is installed via the packer scripts. Add the repo config there and install ovs. Also remove the "sudo yum versionlock add openvswitch" from the run.sh script which is what is locking the ovs version at 2.7.3.</p> <p>2. After the vms are started, ssh over to the vms, config the repo and install ovs.</p> <p>I suspect updating with <span class="error">&#91;1&#93;</span> is the easier solution, but it does require building the image.</p> NETVIRT-1212

CSIT fails with OVS 2.9.0 when group table fails to collect packets

Bug Medium In Progress Unresolved Srinivas Rachakonda Sam Hague Thu, 12 Apr 2018 14:45:56 +0000 Thu, 28 Nov 2019 06:01:21 +0000 0 4 <p>according to this <a href="https://lists.opendaylight.org/pipermail/integration-dev/2018-April/011586.html" class="external-link" target="_blank" rel="nofollow noopener">email</a> I think we can <a href="#/c/71045/" target="_blank" rel="noopener">revert the patch</a> to not upgrade ovs.</p> <p>Leaving this open until we have a package. The patch has been upstreamed but it needs to find it's way into an rpm that we can install. </p> <p>for kicks, I ran a <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/46/jamo-netvirt-csit-1node-openstack-queens-upstream-stateful-fluorine/2/robot-plugin/log_full.html.gz" class="external-link" target="_blank" rel="nofollow noopener">job today</a> that ran with ovs2.9.0 and it still has trouble, so safe to assume<br/> we still don't have the fix in the packages used by csit.</p> <p>Upstream ovs has release 9.0.1 and 9.0.2 that have the fix. RDO has the following patch to include those builds in RDO: <a href="https://review.rdoproject.org/r/#/c/13839/" class="external-link" target="_blank" rel="nofollow noopener">https://review.rdoproject.org/r/#/c/13839/</a>. Once this patch merges we will have the fixes in RDO. Rebuild our packer images to pick up the new RDO package for openvswitch, remove the lock to 2.7 and we are good.</p> <p>do we still need to keep this open?</p> <p>you will need to run some jobs with a newer ovs version to see. looking at a main job <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-neon/1541/control_1/ovsdb-server.log.gz" class="external-link" target="_blank" rel="nofollow noopener">here</a> which is<br/> queens and neon, it's still running ovs 2.8</p> <p>Thanks JamO.<br/> Srinivas, let's run a few jobs with OVS 2.9</p> <p>Scheduled a job to run with OVS 2.9.2</p> <p><a href="https://jenkins.opendaylight.org/sandbox/job/srini-netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-neon/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/sandbox/job/srini-netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-neon/</a></p> <p>Stack is failing after OVS 2.9.2 is installed.</p> <p> </p> <p><a href="https://jenkins.opendaylight.org/sandbox/job/srini-netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-neon/2/console" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/sandbox/job/srini-netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-neon/2/console</a></p> <p>I don't think it ever even tries to run devstack. Looks like it's failing near the custom ovs install. here is the snippet<br/> that caught my eye from the console log:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">+ install_ovs_from_repo 10.30.170.102 file:/tmp/ovs_rpms + local -r ip=10.30.170.102 + local -r rpm_repo=file:/tmp/ovs_rpms + echo <span class="code-quote">'Installing OVS from repo file:/tmp/ovs_rpms on 10.30.170.102 ...'</span> Installing OVS from repo file:/tmp/ovs_rpms on 10.30.170.102 ... + cat + scp /w/workspace/srini-netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-neon/install_ovs.sh 10.30.170.102:/tmp Warning: Permanently added <span class="code-quote">'10.30.170.102'</span> (ECDSA) to the list of known hosts. + ssh -t -t 10.30.170.102 <span class="code-quote">'bash /tmp/install_ovs.sh &gt;&gt; /tmp/install_ovs.txt 2&gt;&amp;1'</span> Warning: Permanently added <span class="code-quote">'10.30.170.102'</span> (ECDSA) to the list of known hosts. Connection to 10.30.170.102 closed. Build step <span class="code-quote">'Execute shell'</span> marked build as failure </pre> </div></div> <p>btw, I didn't see a stack.log file in the compute node logs, which is how I knew that devstack didn't run. look <a href="https://logs.opendaylight.org/sandbox/vex-yul-odl-jenkins-2/srini-netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-neon/2/control_1/" class="external-link" target="_blank" rel="nofollow noopener">here</a>.</p> <p>In the below log:</p> <p><a href="https://logs.opendaylight.org/sandbox/vex-yul-odl-jenkins-2/srini-netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-neon/2/control_1/install_ovs.txt.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/vex-yul-odl-jenkins-2/srini-netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-neon/2/control_1/install_ovs.txt.gz</a></p> <p>there is a message saying openvswitch installed.</p> <p><em>Installed:</em><br/> <em>openvswitch.x86_64 0:2.9.2-1.el7</em> <br/> <em>openvswitch-dkms.x86_64 0:2.9.2-1.el7</em> <br/> <em>openvswitch-selinux-policy.noarch 0:2.9.2-1.el7</em></p> <p> </p> <p>But there is an error for kernel not updated:</p> <p> </p> <p><em>++ sudo modinfo -n openvswitch</em><br/> <em>+ NEW_MOD=/lib/modules/3.10.0-1062.1.2.el7.x86_64/kernel/net/openvswitch/openvswitch.ko.xz</em><br/> <em>+ '<span class="error">&#91;&#39; /lib/modules/3.10.0-1062.1.2.el7.x86_64/kernel/net/openvswitch/openvswitch.ko.xz &#39;!=&#39; /lib/modules/3.10.0-1062.1.2.el7.x86_64/kernel/net/openvswitch/openvswitch.ko.xz &#39;&#93;</span>'</em><br/> <em>+ echo 'Kernel module was not updated'</em><br/> <em>Kernel module was not updated</em><br/> <em>+ exit 1</em></p> <p> </p> <p>Please kindly suggest if kernel needs to be updated to a new version</p> <p>1. Why did we take 2.9.2? I checked that 2.9.5 is the latest with most bug fixes in 2.9.x track. Any specific reason?<br/> 2. Linux kernel range supported is 3.10 to 4.13. So 3.10 should work, however you can try upgrading the kernel to newer versions in this range.</p> Development Rank 0|i03dav: