[GBP-285] NSH headers not recognized Created: 17/May/17  Updated: 19/Oct/17

Status: Confirmed
Project: groupbasedpolicy
Component/s: Integration:SFC
Affects Version/s: unspecified
Fix Version/s: None

Type: Bug
Reporter: Tomas Cechvala Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: All
Platform: All


External issue ID: 8488
Priority: Normal

 Description   

There is a problem on Jenkins in groupbasedpolicy jobs. We can't upgrade OVS properly.

We moved from
curl https://raw.githubusercontent.com/pritesh/ovs/nsh-v8/third-party/start-ovs-deb.sh | bash
to
curl https://raw.githubusercontent.com/yyang13/ovs_nsh_patches/master/start-ovs-deb.sh | bash
since this patch got merged in Openflowplugin
https://git.opendaylight.org/gerrit/#/c/43697/

The installation script for new OVS should uninstall existing version of OVS and the install the new one.
This seems to finish successfully.

But when trying to add the following flows, an error is returned.

sudo ovs-ofctl add-flow $sw "priority=1000,nsi=255,actions=move:NXM_NX_NSH_MDTYPE[]>NXM_NX_NSH_MDTYPE[],move:NXM_NX_NSH_NP[]>NXM_NX_NSH_NP[],move:NXM_NX_NSP[]>NXM_NX_NSP[],load:254>NXM_NX_NSI[],move:NXM_NX_NSH_C1[]>NXM_NX_NSH_C1[],move:NXM_NX_NSH_C2[]>NXM_NX_NSH_C2[],move:NXM_NX_TUN_ID[0..31]>NXM_NX_TUN_ID[0..31],load:$TUNNEL>NXM_NX_TUN_IPV4_DST[],load:0x4->NXM_NX_TUN_GPE_NP[],IN_PORT" -OOpenFlow13

ovs-ofctl: unknown keyword nsi

sudo ovs-ofctl add-flow $sw "priority=1000,actions=move:NXM_NX_NSH_MDTYPE[]>NXM_NX_NSH_MDTYPE[],move:NXM_NX_NSH_NP[]>NXM_NX_NSH_NP[],move:NXM_NX_NSP[]>NXM_NX_NSP[],load:254>NXM_NX_NSI[],move:NXM_NX_NSH_C1[]>NXM_NX_NSH_C1[],move:NXM_NX_NSH_C2[]>NXM_NX_NSH_C2[],move:NXM_NX_TUN_ID[0..31]>NXM_NX_TUN_ID[0..31],load:$TUNNEL>NXM_NX_TUN_IPV4_DST[],load:0x4->NXM_NX_TUN_GPE_NP[],IN_PORT" -OOpenFlow13

ovs-ofctl: NXM_NX_NSH_MDTYPE[]->NXM_NX_NSH_MDTYPE[]: unknown field `NXM_NX_NSH_MDTYPE

The problem seems to be that the new OVS does not recognize NSH headers. The problem seems to be localy significant to Jenkins, haven't been spot elsewhere.
I was wondering if someone from integration project can help us with this. Maybe it's just a matter of upgrading ovs in releng/builder and not while test is running.

I don't think that we can just replace github link or rewrite versions in releng builder, since installing this new ovs version require some extra work.

Thanks
Tomas



 Comments   
Comment by Jamo Luhrsen [ 25/May/17 ]

(In reply to Tomas Cechvala from comment #0)
> There is a problem on Jenkins in groupbasedpolicy jobs. We can't upgrade OVS
> properly.
>
> We moved from
> curl
> https://raw.githubusercontent.com/pritesh/ovs/nsh-v8/third-party/start-ovs-
> deb.sh | bash
> to
> curl
> https://raw.githubusercontent.com/yyang13/ovs_nsh_patches/master/start-ovs-
> deb.sh | bash
> since this patch got merged in Openflowplugin
> https://git.opendaylight.org/gerrit/#/c/43697/
>
> The installation script for new OVS should uninstall existing version of OVS
> and the install the new one.
> This seems to finish successfully.
>
> But when trying to add the following flows, an error is returned.
>
> sudo ovs-ofctl add-flow $sw
> "priority=1000,nsi=255,actions=move:NXM_NX_NSH_MDTYPE[]->NXM_NX_NSH_MDTYPE[],
> move:NXM_NX_NSH_NP[]>NXM_NX_NSH_NP[],move:NXM_NX_NSP[]>NXM_NX_NSP[],load:
> 254->NXM_NX_NSI[],move:NXM_NX_NSH_C1[]>NXM_NX_NSH_C1[],move:NXM_NX_NSH_C2[]
> >NXM_NX_NSH_C2[],move:NXM_NX_TUN_ID[0..31]->NXM_NX_TUN_ID[0..31],load:
> $TUNNEL->NXM_NX_TUN_IPV4_DST[],load:0x4->NXM_NX_TUN_GPE_NP[],IN_PORT"
> -OOpenFlow13
>
> ovs-ofctl: unknown keyword nsi
>
> sudo ovs-ofctl add-flow $sw
> "priority=1000,actions=move:NXM_NX_NSH_MDTYPE[]->NXM_NX_NSH_MDTYPE[],move:
> NXM_NX_NSH_NP[]>NXM_NX_NSH_NP[],move:NXM_NX_NSP[]>NXM_NX_NSP[],load:254-
> >NXM_NX_NSI[],move:NXM_NX_NSH_C1[]>NXM_NX_NSH_C1[],move:NXM_NX_NSH_C2[]
> >NXM_NX_NSH_C2[],move:NXM_NX_TUN_ID[0..31]->NXM_NX_TUN_ID[0..31],load:
> $TUNNEL->NXM_NX_TUN_IPV4_DST[],load:0x4->NXM_NX_TUN_GPE_NP[],IN_PORT"
> -OOpenFlow13
>
> ovs-ofctl: NXM_NX_NSH_MDTYPE[]->NXM_NX_NSH_MDTYPE[]: unknown field
> `NXM_NX_NSH_MDTYPE
>
> The problem seems to be that the new OVS does not recognize NSH headers. The
> problem seems to be localy significant to Jenkins, haven't been spot
> elsewhere.
> I was wondering if someone from integration project can help us with this.
> Maybe it's just a matter of upgrading ovs in releng/builder and not while
> test is running.
>
> I don't think that we can just replace github link or rewrite versions in
> releng builder, since installing this new ovs version require some extra
> work.
>
> Thanks
> Tomas

Tomas,

I took a look and something is not quite right with the description here.
Well, at least I could not find anywhere in our int/test or releng/builder
repo that is going the "curl" install of ovs. I saw something in the sfc
int/test folders that was cloning some nsh repo from yyang. where is that
happening.

Also, I could not find the two errors you mention above ("unknown keyword
nsi" an d"unknown field 'NXM_NX_NSH_MDTYPE') in the robot or console logs.
Maybe I missed it? Can you point me there?

btw, we are talking about this type of job right?
https://jenkins.opendaylight.org/releng/view/groupbasedpolicy/job/groupbasedpolicy-csit-1node-6node-only-carbon/

In the sandbox, I took a random job and made it use the same tools_system
image (vm_1_image: Ubuntu 14.04 - gbp - 20170210-0536) that the gbp 6node
job is using. Then I hacked the setup scripts to do this:

ssh ${TOOLS_SYSTEM_IP} "curl https://raw.githubusercontent.com/yyang13/ovs_nsh_patches/master/start-ovs-deb.sh | bash"
ssh ${TOOLS_SYSTEM_IP} "sudo ovs-vsctl show"
ssh ${TOOLS_SYSTEM_IP} "sudo ovs-vsctl add-br br-jamo"
ssh ${TOOLS_SYSTEM_IP} "sudo ovs-ofctl dump-flows br-jamo -OOpenFlow13"
ssh ${TOOLS_SYSTEM_IP} "sudo ovs-ofctl add-flow br-jamo 'priority=1000,nsi=255,actions=move:NXM_NX_NSH_MDTYPE[]->NXM_NX_NSH_MDTYPE[],\
move:NXM_NX_NSH_NP[]>NXM_NX_NSH_NP[],move:NXM_NX_NSP[]>NXM_NX_NSP[],\
load:254->NXM_NX_NSI[],move:NXM_NX_NSH_C1[]->NXM_NX_NSH_C1[],\
move:NXM_NX_NSH_C2[]>NXM_NX_NSH_C2[],move:NXM_NX_TUN_ID[0..31]>NXM_NX_TUN_ID[0..31],\
load:81->NXM_NX_TUN_IPV4_DST[],load:0x4->NXM_NX_TUN_GPE_NP[],IN_PORT' -OOpenFlow13"
ssh ${TOOLS_SYSTEM_IP} "sudo ovs-vsctl show"
ssh ${TOOLS_SYSTEM_IP} "sudo ovs-ofctl dump-flows br-jamo -OOpenFlow13"

the resulting output seemed ok:

11:52:47 + ssh 10.29.12.18 'curl https://raw.githubusercontent.com/yyang13/ovs_nsh_patches/master/start-ovs-deb.sh | bash'
11:52:47 Warning: Permanently added '10.29.12.18' (ECDSA) to the list of known hosts.

11:52:47 % Total % Received % Xferd Average Speed Time Time Time Current
11:52:47 Dload Upload Total Spent Left Speed
11:52:47
0 0 0 0 0 0 0 0 -::- -::- -::- 0
65 2931 65 1926 0 0 2660 0 0:00:01 -::- 0:00:01 2660
100 2931 100 2931 0 0 4038 0 -::- -:-:-Removing old ovs configuration.
11:52:48 - -::- 4037
11:52:48 * Killing ovs-vswitchd (6329)
11:52:49 * Killing ovsdb-server (6319)

11:52:49 bash: line 5: cd: /usr/local/var/run/openvswitch: No such file or directory
11:52:49
11:52:49 Usage:
11:52:49 kill [options] <pid> [...]
11:52:49
11:52:49 Options:
11:52:49 <pid> [...] send signal to every <pid> listed
11:52:49 -<signal>, -s, --signal <signal>
11:52:49 specify the <signal> to be sent
11:52:49 -l, --list=[<signal>] list all signal names, or convert one to a name
11:52:49 -L, --table list all signal names in a nice table
11:52:49
11:52:49 -h, --help display this help and exit
11:52:49 -V, --version output version information and exit
11:52:49
11:52:49 For more details see kill(1).
11:52:49 rmmod: ERROR: Module vxlan is not currently loaded
11:52:49 rmmod: ERROR: Module openvswitch is not currently loaded
11:52:49 (Reading database ... 75791 files and directories currently installed.)
11:52:49 Removing openvswitch-switch (2.3.90-1) ...
11:52:49 * ovs-vswitchd is not running
11:52:49 * ovsdb-server is not running
11:52:49 Purging configuration files for openvswitch-switch (2.3.90-1) ...
11:52:49 Processing triggers for man-db (2.6.7.1-1ubuntu1) ...
11:52:49 (Reading database ... 75755 files and directories currently installed.)
11:52:49 Removing openvswitch-common (2.3.90-1) ...
11:52:49 dpkg: warning: while removing openvswitch-common, directory '/var/log/openvswitch' not empty so not removed
11:52:49 Processing triggers for man-db (2.6.7.1-1ubuntu1) ...
11:52:49 (Reading database ... 75709 files and directories currently installed.)
11:52:50 Removing openvswitch-datapath-dkms (2.3.90-1) ...
11:52:50
11:52:50 -------- Uninstall Beginning --------
11:52:50 Module: openvswitch
11:52:50 Version: 2.3.90
11:52:50 Kernel: 3.13.0-106-generic (x86_64)
11:52:50 -------------------------------------
11:52:50
11:52:50 Status: Before uninstall, this module version was ACTIVE on this kernel.
11:52:50
11:52:50 openvswitch.ko:

.
.
.
<SNIP>
.
.
.
11:57:25 /etc/rc5.d/S20openvswitch-switch -> ../init.d/openvswitch-switch
11:57:25 0955b887-a14c-4ee4-8ffe-a9ce182950cc
11:57:25 ovs_version: "2.5.90"
11:57:25 Install Complete!
11:57:25 + ssh 10.29.12.18 'sudo ovs-vsctl show'
11:57:25 Warning: Permanently added '10.29.12.18' (ECDSA) to the list of known hosts.

11:57:26 0955b887-a14c-4ee4-8ffe-a9ce182950cc
11:57:26 ovs_version: "2.5.90"
11:57:26 + ssh 10.29.12.18 'sudo ovs-vsctl add-br br-jamo'
11:57:26 Warning: Permanently added '10.29.12.18' (ECDSA) to the list of known hosts.
11:57:26 + ssh 10.29.12.18 'sudo ovs-ofctl dump-flows br-jamo -OOpenFlow13'
11:57:26 Warning: Permanently added '10.29.12.18' (ECDSA) to the list of known hosts.
11:57:27 OFPST_FLOW reply (OF1.3) (xid=0x2):
11:57:27 cookie=0x0, duration=0.457s, table=0, n_packets=0, n_bytes=0, priority=0 actions=NORMAL
11:57:27 + ssh 10.29.12.18 'sudo ovs-ofctl add-flow br-jamo '\''priority=1000,nsi=255,actions=move:NXM_NX_NSH_MDTYPE[]>NXM_NX_NSH_MDTYPE[], move:NXM_NX_NSH_NP[]>NXM_NX_NSH_NP[],move:NXM_NX_NSP[]>NXM_NX_NSP[], load:254>NXM_NX_NSI[],move:NXM_NX_NSH_C1[]>NXM_NX_NSH_C1[], move:NXM_NX_NSH_C2[]>NXM_NX_NSH_C2[],move:NXM_NX_TUN_ID[0..31]>NXM_NX_TUN_ID[0..31], load:81>NXM_NX_TUN_IPV4_DST[],load:0x4->NXM_NX_TUN_GPE_NP[],IN_PORT'\'' -OOpenFlow13'
11:57:27 Warning: Permanently added '10.29.12.18' (ECDSA) to the list of known hosts.

11:57:27 + ssh 10.29.12.18 'sudo ovs-vsctl show'
11:57:27 Warning: Permanently added '10.29.12.18' (ECDSA) to the list of known hosts.
11:57:28 0955b887-a14c-4ee4-8ffe-a9ce182950cc
11:57:28 Bridge br-jamo
11:57:28 Port br-jamo
11:57:28 Interface br-jamo
11:57:28 type: internal
11:57:28 ovs_version: "2.5.90"
11:57:28 + ssh 10.29.12.18 'sudo ovs-ofctl dump-flows br-jamo -OOpenFlow13'
11:57:28 Warning: Permanently added '10.29.12.18' (ECDSA) to the list of known hosts.

11:57:28 OFPST_FLOW reply (OF1.3) (xid=0x2):
11:57:28 cookie=0x0, duration=0.934s, table=0, n_packets=0, n_bytes=0, priority=1000,nsi=255 actions=move:NXM_NX_NSH_MDTYPE[]>NXM_NX_NSH_MDTYPE[],move:NXM_NX_NSH_NP[]>NXM_NX_NSH_NP[],move:NXM_NX_NSP[]>NXM_NX_NSP[],load:0xfe>NXM_NX_NSI[],move:NXM_NX_NSH_C1[]>NXM_NX_NSH_C1[],move:NXM_NX_NSH_C2[]>NXM_NX_NSH_C2[],move:NXM_NX_TUN_ID[0..31]>NXM_NX_TUN_ID[0..31],load:0x51>NXM_NX_TUN_IPV4_DST[],load:0x4->NXM_NX_TUN_GPE_NP[],IN_PORT
11:57:28 cookie=0x0, duration=1.860s, table=0, n_packets=0, n_bytes=0, priority=0 actions=NORMAL

hopefully this helps more than confuses.

Generated at Wed Feb 07 20:01:35 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.