according to this email I think we can revert the patch to not upgrade ovs.

Leaving this open until we have a package. The patch has been upstreamed but it needs to find it's way into an rpm that we can install.

for kicks, I ran a job today that ran with ovs2.9.0 and it still has trouble, so safe to assume
we still don't have the fix in the packages used by csit.

Upstream ovs has release 9.0.1 and 9.0.2 that have the fix. RDO has the following patch to include those builds in RDO: https://review.rdoproject.org/r/#/c/13839/. Once this patch merges we will have the fixes in RDO. Rebuild our packer images to pick up the new RDO package for openvswitch, remove the lock to 2.7 and we are good.

do we still need to keep this open?

you will need to run some jobs with a newer ovs version to see. looking at a main job here which is
queens and neon, it's still running ovs 2.8

Thanks JamO.
Srinivas, let's run a few jobs with OVS 2.9

Scheduled a job to run with OVS 2.9.2

https://jenkins.opendaylight.org/sandbox/job/srini-netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-neon/

Stack is failing after OVS 2.9.2 is installed.

https://jenkins.opendaylight.org/sandbox/job/srini-netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-neon/2/console

I don't think it ever even tries to run devstack. Looks like it's failing near the custom ovs install. here is the snippet
that caught my eye from the console log:

+ install_ovs_from_repo 10.30.170.102 file:/tmp/ovs_rpms
+ local -r ip=10.30.170.102
+ local -r rpm_repo=file:/tmp/ovs_rpms
+ echo 'Installing OVS from repo file:/tmp/ovs_rpms on 10.30.170.102 ...'
Installing OVS from repo file:/tmp/ovs_rpms on 10.30.170.102 ...
+ cat
+ scp /w/workspace/srini-netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-neon/install_ovs.sh 10.30.170.102:/tmp
Warning: Permanently added '10.30.170.102' (ECDSA) to the list of known hosts.
+ ssh -t -t 10.30.170.102 'bash /tmp/install_ovs.sh >> /tmp/install_ovs.txt 2>&1'
Warning: Permanently added '10.30.170.102' (ECDSA) to the list of known hosts.
Connection to 10.30.170.102 closed.
Build step 'Execute shell' marked build as failure

btw, I didn't see a stack.log file in the compute node logs, which is how I knew that devstack didn't run. look here.

In the below log:

https://logs.opendaylight.org/sandbox/vex-yul-odl-jenkins-2/srini-netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-neon/2/control_1/install_ovs.txt.gz

there is a message saying openvswitch installed.

Installed:
openvswitch.x86_64 0:2.9.2-1.el7
openvswitch-dkms.x86_64 0:2.9.2-1.el7
openvswitch-selinux-policy.noarch 0:2.9.2-1.el7

But there is an error for kernel not updated:

++ sudo modinfo -n openvswitch
+ NEW_MOD=/lib/modules/3.10.0-1062.1.2.el7.x86_64/kernel/net/openvswitch/openvswitch.ko.xz
+ '[' /lib/modules/3.10.0-1062.1.2.el7.x86_64/kernel/net/openvswitch/openvswitch.ko.xz '!=' /lib/modules/3.10.0-1062.1.2.el7.x86_64/kernel/net/openvswitch/openvswitch.ko.xz ']'
+ echo 'Kernel module was not updated'
Kernel module was not updated
+ exit 1

Please kindly suggest if kernel needs to be updated to a new version

1. Why did we take 2.9.2? I checked that 2.9.5 is the latest with most bug fixes in 2.9.x track. Any specific reason?
2. Linux kernel range supported is 3.10 to 4.13. So 3.10 should work, however you can try upgrading the kernel to newer versions in this range.