[OVSDB-213] Inter-network East-west traffic does not work after a compute node is unstacked and re-stacked Created: 04/Oct/15  Updated: 29/May/18  Resolved: 15/Mar/16

Status: Resolved
Project: ovsdb
Component/s: openstack.net-virt
Affects Version/s: unspecified
Fix Version/s: None

Type: Bug
Reporter: Ravi Shankar S Assignee: Josh Hershberg
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: All
Platform: All


Issue Links:
Duplicate
is duplicated by OVSDB-215 East-west traffic (with different net... Resolved
External issue ID: 4408
Priority: High

 Description   

The Inter-network East West traffic is not workingm after a compute node is unstacked and restacked. The static Arp and L3 forwarding rules for inter-network east west traffic were not programmed after a unstack/restack for the compute node. This was the reason for failure.

Sample static Arp and L3 forwarding flows:

cookie=0x0, duration=14333.017s, table=20, n_packets=1, n_bytes=42, priority=1024,arp,tun_id=0x413,arp_tpa=1.0.0.3 actions=move:NXM_OF_ETH_SRC[]>NXM_OF_ETH_DST[],set_field:fa:16:3e:2f:11:a8>eth_src,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]>NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]>NXM_OF_ARP_TPA[],load:0xfa163e2f11a8->NXM_NX_ARP_SHA[],load:0x1000003->NXM_OF_ARP_SPA[],IN_PORT

cookie=0x0, duration=14333.049s, table=70, n_packets=15, n_bytes=1910, priority=1024,ip,tun_id=0x413,nw_dst=1.0.0.3 actions=set_field:fa:16:3e:2f:11:a8->eth_dst,goto_table:80



 Comments   
Comment by Flavio Fernandes [ 13/Oct/15 ]

can you give a little more info? My assumption is that when the node re-stacks
it will have all the mdsal flows re-programmed, so the cache in the ovsdb L3 netvirt is unaffected.

Clearing the cache doe not do anything about the programmed flows, thus I think I may be missing something here and gerrit [1] is not addressing the bug.

[1]: https://git.opendaylight.org/gerrit/#/c/27946/3/

Comment by Vasanthan Balasubramaniyan [ 03/Feb/16 ]

Tested on: Bery-RC1,openstack(kilo)

Steps to Reproduce:
===================
1.Stacked the Openstack Controller & compute node1,node2
2.Created 2 networks and spawn vm's on each compute node(s)/networks.
3.Associated the one Floating IP address on atleast one VM/networks (across compute node1,node2
4.East-West traffic(across compute node),North-South traffic(across compute node) working fine.
5.Unstack the compute node-2 and restack
6.Spawn VM-x on compute node-2 and associated floating IP address.
7.VM-x unable to reach VM on compute node-1(i.e created on step 3) using local ip address (i.e east-west traffic not reachable).However its reachable via Floating IP of VM-x to VM on compute node-1.

Workaround:
=============
If we reboot VM's (on compute node-1),then VM-x (on restacked compute node-2) manage to reach both local IP address (east-west) as well as floating ip address (north-south).

Openflow table entries captures on both compute node before & after reboot of VM.

Here: 3.3.3.5 (compute node-1 VM's local IP),201.1.1.5 is floating IP address.

Before reboot:
============
openflow@os-compute1:~/kilo/devstack$ sudo ovs-ofctl -O openflow13 dump-flows br-int | grep "3.3.3.5"
cookie=0x0, duration=546.107s, table=20, n_packets=0, n_bytes=0, priority=1024,arp,tun_id=0x446,arp_tpa=3.3.3.5,arp_op=1 actions=move:NXM_OF_ETH_SRC[]>NXM_OF_ETH_DST[],set_field:fa:16:3e:c9:60:51>eth_src,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]>NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]>NXM_OF_ARP_TPA[],load:0xfa163ec96051->NXM_NX_ARP_SHA[],load:0x3030305->NXM_OF_ARP_SPA[],IN_PORT
cookie=0x0, duration=529.757s, table=30, n_packets=105, n_bytes=11706, priority=1024,ip,in_port=2,nw_dst=201.1.1.5 actions=set_field:3.3.3.5->ip_dst,load:0x446->NXM_NX_REG3[],goto_table:40
cookie=0x0, duration=546.107s, table=40, n_packets=132, n_bytes=14900, priority=36001,ip,in_port=1,dl_src=fa:16:3e:c9:60:51,nw_src=3.3.3.5 actions=goto_table:50
cookie=0x0, duration=551.677s, table=70, n_packets=110, n_bytes=12728, priority=1024,ip,tun_id=0x446,nw_dst=3.3.3.5 actions=set_field:fa:16:3e:c9:60:51->eth_dst,goto_table:80
cookie=0x0, duration=529.718s, table=100, n_packets=109, n_bytes=12698, priority=512,ip,tun_id=0x446,dl_dst=fa:16:3e:05:a1:e8,nw_src=3.3.3.5 actions=set_field:fa:16:3e:79:94:9d->eth_src,dec_ttl,set_field:00:01:e8:00:ee:ee->eth_dst,set_field:201.1.1.5->ip_src,output:2

openflow@os-compute1:~/kilo/devstack$ sudo ovs-ofctl -O openflow13 dump-flows br-int | grep "201.1.1.5"
cookie=0x0, duration=613.934s, table=20, n_packets=1, n_bytes=60, priority=1024,arp,in_port=2,arp_tpa=201.1.1.5,arp_op=1 actions=move:NXM_OF_ETH_SRC[]>NXM_OF_ETH_DST[],set_field:fa:16:3e:79:94:9d>eth_src,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]>NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]>NXM_OF_ARP_TPA[],load:0xfa163e79949d->NXM_NX_ARP_SHA[],load:0xc9010105->NXM_OF_ARP_SPA[],IN_PORT
cookie=0x0, duration=613.903s, table=30, n_packets=105, n_bytes=11706, priority=1024,ip,in_port=2,nw_dst=201.1.1.5 actions=set_field:3.3.3.5->ip_dst,load:0x446->NXM_NX_REG3[],goto_table:40
cookie=0x0, duration=613.864s, table=100, n_packets=109, n_bytes=12698, priority=512,ip,tun_id=0x446,dl_dst=fa:16:3e:05:a1:e8,nw_src=3.3.3.5 actions=set_field:fa:16:3e:79:94:9d->eth_src,dec_ttl,set_field:00:01:e8:00:ee:ee->eth_dst,set_field:201.1.1.5->ip_src,output:2

==========================

openflow@os-controller:~/kilo/devstack$ sudo ovs-ofctl -O openflow13 dump-flows br-int | grep "3.3.3.5"
cookie=0x0, duration=564.976s, table=20, n_packets=2, n_bytes=84, priority=1024,arp,tun_id=0x446,arp_tpa=3.3.3.5,arp_op=1 actions=move:NXM_OF_ETH_SRC[]>NXM_OF_ETH_DST[],set_field:fa:16:3e:c9:60:51>eth_src,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]>NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]>NXM_OF_ARP_TPA[],load:0xfa163ec96051->NXM_NX_ARP_SHA[],load:0x3030305->NXM_OF_ARP_SPA[],IN_PORT
cookie=0x0, duration=565.007s, table=70, n_packets=5, n_bytes=1022, priority=1024,ip,tun_id=0x446,nw_dst=3.3.3.5 actions=set_field:fa:16:3e:c9:60:51->eth_dst,goto_table:80

openflow@os-controller:~/kilo/devstack$ sudo ovs-ofctl -O openflow13 dump-flows br-int | grep "201.1.1.3"
cookie=0x0, duration=3999.766s, table=20, n_packets=1, n_bytes=60, priority=1024,arp,in_port=2,arp_tpa=201.1.1.3,arp_op=1 actions=move:NXM_OF_ETH_SRC[]>NXM_OF_ETH_DST[],set_field:fa:16:3e:55:14:9a>eth_src,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]>NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]>NXM_OF_ARP_TPA[],load:0xfa163e55149a->NXM_NX_ARP_SHA[],load:0xc9010103->NXM_OF_ARP_SPA[],IN_PORT
cookie=0x0, duration=3999.684s, table=30, n_packets=113, n_bytes=13062, priority=1024,ip,in_port=2,nw_dst=201.1.1.3 actions=set_field:3.3.3.4->ip_dst,load:0x446->NXM_NX_REG3[],goto_table:40
cookie=0x0, duration=3999.679s, table=100, n_packets=108, n_bytes=12000, priority=512,ip,tun_id=0x446,dl_dst=fa:16:3e:05:a1:e8,nw_src=3.3.3.4 actions=set_field:fa:16:3e:55:14:9a->eth_src,dec_ttl,set_field:00:01:e8:00:ee:ee->eth_dst,set_field:201.1.1.3->ip_src,output:2
openflow@os-controller:~/kilo/devstack$

================================================================================

After reboot
=============

openflow@os-compute1:~/kilo/devstack$ sudo ovs-ofctl -O openflow13 dump-flows br-int | grep "201.1.1.5"
cookie=0x0, duration=694.066s, table=20, n_packets=1, n_bytes=60, priority=1024,arp,in_port=2,arp_tpa=201.1.1.5,arp_op=1 actions=move:NXM_OF_ETH_SRC[]>NXM_OF_ETH_DST[],set_field:fa:16:3e:79:94:9d>eth_src,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]>NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]>NXM_OF_ARP_TPA[],load:0xfa163e79949d->NXM_NX_ARP_SHA[],load:0xc9010105->NXM_OF_ARP_SPA[],IN_PORT
cookie=0x0, duration=694.035s, table=30, n_packets=105, n_bytes=11706, priority=1024,ip,in_port=2,nw_dst=201.1.1.5 actions=set_field:3.3.3.5->ip_dst,load:0x446->NXM_NX_REG3[],goto_table:40
cookie=0x0, duration=693.996s, table=100, n_packets=109, n_bytes=12698, priority=512,ip,tun_id=0x446,dl_dst=fa:16:3e:05:a1:e8,nw_src=3.3.3.5 actions=set_field:fa:16:3e:79:94:9d->eth_src,dec_ttl,set_field:00:01:e8:00:ee:ee->eth_dst,set_field:201.1.1.5->ip_src,output:2

openflow@os-compute1:~/kilo/devstack$ sudo ovs-ofctl -O openflow13 dump-flows br-int | grep "3.3.3.5"
cookie=0x0, duration=711.907s, table=20, n_packets=0, n_bytes=0, priority=1024,arp,tun_id=0x446,arp_tpa=3.3.3.5,arp_op=1 actions=move:NXM_OF_ETH_SRC[]>NXM_OF_ETH_DST[],set_field:fa:16:3e:c9:60:51>eth_src,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]>NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]>NXM_OF_ARP_TPA[],load:0xfa163ec96051->NXM_NX_ARP_SHA[],load:0x3030305->NXM_OF_ARP_SPA[],IN_PORT
cookie=0x0, duration=695.557s, table=30, n_packets=105, n_bytes=11706, priority=1024,ip,in_port=2,nw_dst=201.1.1.5 actions=set_field:3.3.3.5->ip_dst,load:0x446->NXM_NX_REG3[],goto_table:40
cookie=0x0, duration=711.907s, table=40, n_packets=132, n_bytes=14900, priority=36001,ip,in_port=1,dl_src=fa:16:3e:c9:60:51,nw_src=3.3.3.5 actions=goto_table:50
cookie=0x0, duration=717.477s, table=70, n_packets=110, n_bytes=12728, priority=1024,ip,tun_id=0x446,nw_dst=3.3.3.5 actions=set_field:fa:16:3e:c9:60:51->eth_dst,goto_table:80
cookie=0x0, duration=695.518s, table=100, n_packets=109, n_bytes=12698, priority=512,ip,tun_id=0x446,dl_dst=fa:16:3e:05:a1:e8,nw_src=3.3.3.5 actions=set_field:fa:16:3e:79:94:9d->eth_src,dec_ttl,set_field:00:01:e8:00:ee:ee->eth_dst,set_field:201.1.1.5->ip_src,output:2

==================
openflow@os-controller:~/kilo/devstack$ sudo ovs-ofctl O openflow13 dump-flows br-int | grep "201.1.1.3" cookie=0x0, duration=4078.356s, table=20, n_packets=1, n_bytes=60, priority=1024,arp,in_port=2,arp_tpa=201.1.1.3,arp_op=1 actions=move:NXM_OF_ETH_SRC[]>NXM_OF_ETH_DST[],set_field:fa:16:3e:55:14:9a->eth_src,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]>NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]>NXM_OF_ARP_TPA[],load:0xfa163e55149a->NXM_NX_ARP_SHA[],load:0xc9010103->NXM_OF_ARP_SPA[],IN_PORT
cookie=0x0, duration=4078.274s, table=30, n_packets=113, n_bytes=13062, priority=1024,ip,in_port=2,nw_dst=201.1.1.3 actions=set_field:3.3.3.4->ip_dst,load:0x446->NXM_NX_REG3[],goto_table:40
cookie=0x0, duration=4078.269s, table=100, n_packets=108, n_bytes=12000, priority=512,ip,tun_id=0x446,dl_dst=fa:16:3e:05:a1:e8,nw_src=3.3.3.4 actions=set_field:fa:16:3e:55:14:9a->eth_src,dec_ttl,set_field:00:01:e8:00:ee:ee->eth_dst,set_field:201.1.1.3->ip_src,output:2
openflow@os-controller:~/kilo/devstack$ sudo ovs-ofctl -O openflow13 dump-flows br-int | grep "3.3.3.5"
cookie=0x0, duration=688.149s, table=20, n_packets=2, n_bytes=84, priority=1024,arp,tun_id=0x446,arp_tpa=3.3.3.5,arp_op=1 actions=move:NXM_OF_ETH_SRC[]>NXM_OF_ETH_DST[],set_field:fa:16:3e:c9:60:51>eth_src,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]>NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]>NXM_OF_ARP_TPA[],load:0xfa163ec96051->NXM_NX_ARP_SHA[],load:0x3030305->NXM_OF_ARP_SPA[],IN_PORT
cookie=0x0, duration=688.180s, table=70, n_packets=5, n_bytes=1022, priority=1024,ip,tun_id=0x446,nw_dst=3.3.3.5 actions=set_field:fa:16:3e:c9:60:51->eth_dst,goto_table:80

Comment by Sam Hague [ 15/Mar/16 ]

master: https://git.opendaylight.org/gerrit/#/c/36067/
be: https://git.opendaylight.org/gerrit/#/c/36164/

Generated at Wed Feb 07 20:35:48 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.