Operating System: Linux
Platform: All

Steps to Reproduce

1. Ovsdb (Master, built from source)
karaf is up (ports: 6640,6633,6653 and the neutron ports)

2. openstack control node is up, br-int and br-ex created with the public ineterface

3. openstack compute node

4. Created router + networks (reference: http://www.flaviof.com/blog/work/how-to-odl-with-openstack-part3.html)

5. Created instances

6. Instances able to ping each other and also outside network to the L3 Gateway

7. Able to reach the instacnes from the l3 gateway and able to ssh

8. Restart ODL

9. Created new instances

10. The instacnes are able to ping the other instances from the internal network.

11. the new instances are not able to use the router and reach the external networks, also not reachable from the l3 gateway.

Other details

OVS Settings

Control Node

[stack@netvirt-control devstack]$ sudo ovs-vsctl show
b377158c-42d0-41b4-89ca-645017a33cd5
Manager "tcp:192.168.2.147:6640"
is_connected: true
Bridge br-ex
Controller "tcp:192.168.2.147:6653"
is_connected: true
fail_mode: secure
Port patch-int
Interface patch-int
type: patch
options:

Port br-ex
Interface br-ex
type: internal
Port "ens33"
Interface "ens33"
Bridge br-int
Controller "tcp:192.168.2.147:6653"
is_connected: true
fail_mode: secure
Port "tapb5795cb3-b9"
Interface "tapb5795cb3-b9"
Port "tap05b23c57-03"
Interface "tap05b23c57-03"
Port "vxlan-192.168.2.149"
Interface "vxlan-192.168.2.149"
type: vxlan
options:

Port br-int
Interface br-int
type: internal
Port "tape9efbabe-bd"
Interface "tape9efbabe-bd"
Port patch-ext
Interface patch-ext
type: patch
options:

ovs_version: "2.3.1"

Flows in Control node and compute node

Control node:

br-int

[stack@netvirt-control devstack]$ sudo ovs-ofctl O OpenFlow13 dump-flows br-int | cut -d',' -f3
OFPST_FLOW reply (OF1.3) (xid=0x2):
table=0, n_packets=29, n_bytes=5268, in_port=1,dl_src=fa:16:3e:13:08:4e actions=set_field:0x403->tun_id,load:0x1->NXM_NX_REG0[],goto_table:20
table=0, n_packets=87, n_bytes=6871, in_port=4,dl_src=fa:16:3e:08:5e:d8 actions=set_field:0x403->tun_id,load:0x1->NXM_NX_REG0[],goto_table:20
table=0, n_packets=346, n_bytes=19805, priority=0 actions=goto_table:20
table=0, n_packets=0, n_bytes=0, priority=8192,in_port=1 actions=drop
table=0, n_packets=0, n_bytes=0, priority=8192,in_port=4 actions=drop
table=0, n_packets=88, n_bytes=8336, tun_id=0x403,in_port=3 actions=load:0x2->NXM_NX_REG0[],goto_table:20
table=0, n_packets=1266, n_bytes=137994, dl_type=0x88cc actions=CONTROLLER:65535
table=20, n_packets=516, n_bytes=38852, priority=0 actions=goto_table:30
table=20, n_packets=26, n_bytes=1092, priority=1024,arp,tun_id=0x403,arp_tpa=10.1.0.4 actions=move:NXM_OF_ETH_SRC[]>NXM_OF_ETH_DST[],set_field:fa:16:3e:df:82:56>eth_src,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]>NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]>NXM_OF_ARP_TPA[],load:0xfa163edf8256->NXM_NX_ARP_SHA[],load:0xa010004->NXM_OF_ARP_SPA[],IN_PORT
table=20, n_packets=3, n_bytes=126, priority=1024,arp,tun_id=0x403,arp_tpa=10.1.0.2 actions=move:NXM_OF_ETH_SRC[]>NXM_OF_ETH_DST[],set_field:fa:16:3e:13:08:4e>eth_src,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]>NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]>NXM_OF_ARP_TPA[],load:0xfa163e13084e->NXM_NX_ARP_SHA[],load:0xa010002->NXM_OF_ARP_SPA[],IN_PORT
table=20, n_packets=1, n_bytes=42, priority=1024,arp,tun_id=0x403,arp_tpa=10.1.0.3 actions=move:NXM_OF_ETH_SRC[]>NXM_OF_ETH_DST[],set_field:fa:16:3e:49:d9:8c>eth_src,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]>NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]>NXM_OF_ARP_TPA[],load:0xfa163e49d98c->NXM_NX_ARP_SHA[],load:0xa010003->NXM_OF_ARP_SPA[],IN_PORT
table=20, n_packets=2, n_bytes=84, priority=1024,arp,tun_id=0x403,arp_tpa=10.1.0.1 actions=move:NXM_OF_ETH_SRC[]>NXM_OF_ETH_DST[],set_field:fa:16:3e:f2:49:0f>eth_src,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]>NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]>NXM_OF_ARP_TPA[],load:0xfa163ef2490f->NXM_NX_ARP_SHA[],load:0xa010001->NXM_OF_ARP_SPA[],IN_PORT
table=30, n_packets=516, n_bytes=38852, priority=0 actions=goto_table:40
table=40, n_packets=0, n_bytes=0, priority=36001,ip,in_port=4,dl_src=fa:16:3e:08:5e:d8,nw_src=10.1.0.8 actions=goto_table:50
table=40, n_packets=458, n_bytes=29967, priority=0 actions=goto_table:50
table=40, n_packets=14, n_bytes=4650, priority=61012,udp,tp_src=68,tp_dst=67 actions=goto_table:50
table=40, n_packets=44, n_bytes=4235, priority=61007,ip,dl_src=fa:16:3e:08:5e:d8 actions=goto_table:50
table=40, n_packets=0, n_bytes=0, priority=61011,udp,in_port=4,tp_src=67,tp_dst=68 actions=drop
table=50, n_packets=516, n_bytes=38852, priority=0 actions=goto_table:60
table=60, n_packets=516, n_bytes=38852, priority=0 actions=goto_table:70
table=60, n_packets=0, n_bytes=0, priority=2048,ip,reg3=0x403,nw_dst=10.1.0.0/24 actions=set_field:fa:16:3e:f2:49:0f->eth_src,dec_ttl,set_field:0x403->tun_id,goto_table:70
table=70, n_packets=483, n_bytes=33136, priority=0 actions=goto_table:80
table=70, n_packets=2, n_bytes=715, priority=1024,ip,tun_id=0x403,nw_dst=10.1.0.3 actions=set_field:fa:16:3e:49:d9:8c->eth_dst,goto_table:80
table=70, n_packets=2, n_bytes=602, priority=1024,ip,tun_id=0x403,nw_dst=10.1.0.2 actions=set_field:fa:16:3e:13:08:4e->eth_dst,goto_table:80
table=70, n_packets=25, n_bytes=2969, priority=1024,ip,tun_id=0x403,nw_dst=10.1.0.4 actions=set_field:fa:16:3e:df:82:56->eth_dst,goto_table:80
table=80, n_packets=516, n_bytes=38852, priority=0 actions=goto_table:90
table=90, n_packets=512, n_bytes=37941, priority=0 actions=goto_table:100
table=90, n_packets=4, n_bytes=911, priority=61007,ip,dl_dst=fa:16:3e:08:5e:d8 actions=goto_table:100
table=90, n_packets=0, n_bytes=0, priority=61006,udp,dl_src=fa:16:3e:13:08:4e,tp_src=67,tp_dst=68 actions=goto_table:100
table=100, n_packets=469, n_bytes=30726, priority=0 actions=goto_table:110
table=100, n_packets=47, n_bytes=8126, priority=1024,ip,tun_id=0x403,nw_dst=10.1.0.0/24 actions=goto_table:110
table=110, n_packets=19, n_bytes=1785, priority=8192,tun_id=0x403 actions=drop
table=110, n_packets=346, n_bytes=19805, priority=0 actions=drop
table=110, n_packets=19, n_bytes=1892, priority=16383,reg0=0x1,tun_id=0x403,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=output:1,output:3,output:4
table=110, n_packets=82, n_bytes=7454, priority=16384,reg0=0x2,tun_id=0x403,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=output:1,output:4
table=110, n_packets=2, n_bytes=715, tun_id=0x403,dl_dst=fa:16:3e:49:d9:8c actions=output:3
table=110, n_packets=8, n_bytes=1079, tun_id=0x403,dl_dst=fa:16:3e:f3:f4:be actions=output:3
table=110, n_packets=25, n_bytes=2969, tun_id=0x403,dl_dst=fa:16:3e:df:82:56 actions=output:3
table=110, n_packets=9, n_bytes=1121, tun_id=0x403,dl_dst=fa:16:3e:08:5e:d8 actions=output:4
table=110, n_packets=2, n_bytes=602, tun_id=0x403,dl_dst=fa:16:3e:13:08:4e actions=output:1

br-ex

[stack@netvirt-control devstack]$ sudo ovs-ofctl O OpenFlow13 dump-flows br-ex | cut -d',' -f3
OFPST_FLOW reply (OF1.3) (xid=0x2):
table=0, n_packets=630, n_bytes=58406, priority=0 actions=NORMAL
table=0, n_packets=1347, n_bytes=146539, dl_type=0x88cc actions=CONTROLLER:65535

Compute node

br-int

[stack@netvirt-compute devstack]$ sudo ovs-ofctl dump-flows br-int -OOpenFlow13
OFPST_FLOW reply (OF1.3) (xid=0x2):
cookie=0x0, duration=2516.604s, table=0, n_packets=95, n_bytes=7428, in_port=7,dl_src=fa:16:3e:f3:f4:be actions=set_field:0x403->tun_id,load:0x1->NXM_NX_REG0[],goto_table:20
cookie=0x0, duration=2545.369s, table=0, n_packets=95, n_bytes=8498, in_port=5,dl_src=fa:16:3e:df:82:56 actions=set_field:0x403->tun_id,load:0x1->NXM_NX_REG0[],goto_table:20
cookie=0x0, duration=2545.369s, table=0, n_packets=156, n_bytes=16964, in_port=6,dl_src=fa:16:3e:49:d9:8c actions=set_field:0x403->tun_id,load:0x1->NXM_NX_REG0[],goto_table:20
cookie=0x0, duration=2545.369s, table=0, n_packets=431, n_bytes=36231, priority=0 actions=goto_table:20
cookie=0x0, duration=2516.580s, table=0, n_packets=0, n_bytes=0, priority=8192,in_port=7 actions=drop
cookie=0x0, duration=2545.369s, table=0, n_packets=0, n_bytes=0, priority=8192,in_port=5 actions=drop
cookie=0x0, duration=2545.369s, table=0, n_packets=0, n_bytes=0, priority=8192,in_port=6 actions=drop
cookie=0x0, duration=2545.369s, table=0, n_packets=47, n_bytes=6097, tun_id=0x403,in_port=1 actions=load:0x2->NXM_NX_REG0[],goto_table:20
cookie=0x0, duration=2545.369s, table=0, n_packets=1294, n_bytes=141078, dl_type=0x88cc actions=CONTROLLER:65535
cookie=0x0, duration=2545.397s, table=20, n_packets=843, n_bytes=81806, priority=0 actions=goto_table:30
cookie=0x0, duration=2545.369s, table=20, n_packets=13, n_bytes=546, priority=1024,arp,tun_id=0x403,arp_tpa=10.1.0.4 actions=move:NXM_OF_ETH_SRC[]>NXM_OF_ETH_DST[],set_field:fa:16:3e:df:82:56>eth_src,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]>NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]>NXM_OF_ARP_TPA[],load:0xfa163edf8256->NXM_NX_ARP_SHA[],load:0xa010004->NXM_OF_ARP_SPA[],IN_PORT
cookie=0x0, duration=2545.369s, table=20, n_packets=5, n_bytes=210, priority=1024,arp,tun_id=0x403,arp_tpa=10.1.0.2 actions=move:NXM_OF_ETH_SRC[]>NXM_OF_ETH_DST[],set_field:fa:16:3e:13:08:4e>eth_src,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]>NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]>NXM_OF_ARP_TPA[],load:0xfa163e13084e->NXM_NX_ARP_SHA[],load:0xa010002->NXM_OF_ARP_SPA[],IN_PORT
cookie=0x0, duration=2545.379s, table=20, n_packets=10, n_bytes=420, priority=1024,arp,tun_id=0x403,arp_tpa=10.1.0.3 actions=move:NXM_OF_ETH_SRC[]>NXM_OF_ETH_DST[],set_field:fa:16:3e:49:d9:8c>eth_src,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]>NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]>NXM_OF_ARP_TPA[],load:0xfa163e49d98c->NXM_NX_ARP_SHA[],load:0xa010003->NXM_OF_ARP_SPA[],IN_PORT
cookie=0x0, duration=2545.369s, table=20, n_packets=16, n_bytes=672, priority=1024,arp,tun_id=0x403,arp_tpa=10.1.0.1 actions=move:NXM_OF_ETH_SRC[]>NXM_OF_ETH_DST[],set_field:fa:16:3e:f2:49:0f>eth_src,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]>NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]>NXM_OF_ARP_TPA[],load:0xfa163ef2490f->NXM_NX_ARP_SHA[],load:0xa010001->NXM_OF_ARP_SPA[],IN_PORT
cookie=0x0, duration=2545.369s, table=20, n_packets=5, n_bytes=300, priority=1024,arp,in_port=3,arp_tpa=192.168.35.7 actions=move:NXM_OF_ETH_SRC[]>NXM_OF_ETH_DST[],set_field:fa:16:3e:5e:43:c7>eth_src,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]>NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]>NXM_OF_ARP_TPA[],load:0xfa163e5e43c7->NXM_NX_ARP_SHA[],load:0xc0a82307->NXM_OF_ARP_SPA[],IN_PORT
cookie=0x0, duration=2545.379s, table=20, n_packets=8, n_bytes=480, priority=1024,arp,in_port=3,arp_tpa=192.168.35.6 actions=move:NXM_OF_ETH_SRC[]>NXM_OF_ETH_DST[],set_field:fa:16:3e:f1:10:3f>eth_src,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]>NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]>NXM_OF_ARP_TPA[],load:0xfa163ef1103f->NXM_NX_ARP_SHA[],load:0xc0a82306->NXM_OF_ARP_SPA[],IN_PORT
cookie=0x0, duration=2545.377s, table=30, n_packets=692, n_bytes=62495, priority=0 actions=goto_table:40
cookie=0x0, duration=2545.373s, table=30, n_packets=94, n_bytes=11101, priority=1024,ip,in_port=3,nw_dst=192.168.35.6 actions=set_field:10.1.0.3->ip_dst,load:0x403->NXM_NX_REG3[],goto_table:40
cookie=0x0, duration=2545.373s, table=30, n_packets=57, n_bytes=8210, priority=1024,ip,in_port=3,nw_dst=192.168.35.7 actions=set_field:10.1.0.4->ip_dst,load:0x403->NXM_NX_REG3[],goto_table:40
cookie=0x0, duration=2516.665s, table=40, n_packets=0, n_bytes=0, priority=36001,ip,in_port=7,dl_src=fa:16:3e:f3:f4:be,nw_src=10.1.0.7 actions=goto_table:50
cookie=0x0, duration=2545.378s, table=40, n_packets=0, n_bytes=0, priority=36001,ip,in_port=6,dl_src=fa:16:3e:49:d9:8c,nw_src=10.1.0.3 actions=goto_table:50
cookie=0x0, duration=2545.378s, table=40, n_packets=0, n_bytes=0, priority=36001,ip,in_port=5,dl_src=fa:16:3e:df:82:56,nw_src=10.1.0.4 actions=goto_table:50
cookie=0x0, duration=2545.379s, table=40, n_packets=552, n_bytes=47544, priority=0 actions=goto_table:50
cookie=0x0, duration=2545.369s, table=40, n_packets=51, n_bytes=6434, priority=61007,ip,dl_src=fa:16:3e:df:82:56 actions=goto_table:50
cookie=0x0, duration=2516.629s, table=40, n_packets=50, n_bytes=5322, priority=61007,ip,dl_src=fa:16:3e:f3:f4:be actions=goto_table:50
cookie=0x0, duration=2545.378s, table=40, n_packets=135, n_bytes=15866, priority=61007,ip,dl_src=fa:16:3e:49:d9:8c actions=goto_table:50
cookie=0x0, duration=2545.379s, table=40, n_packets=0, n_bytes=0, priority=61011,udp,in_port=5,tp_src=67,tp_dst=68 actions=drop
cookie=0x0, duration=2516.677s, table=40, n_packets=0, n_bytes=0, priority=61011,udp,in_port=7,tp_src=67,tp_dst=68 actions=drop
cookie=0x0, duration=2545.379s, table=40, n_packets=0, n_bytes=0, priority=61011,udp,in_port=6,tp_src=67,tp_dst=68 actions=drop
cookie=0x0, duration=2545.375s, table=50, n_packets=843, n_bytes=81806, priority=0 actions=goto_table:60
cookie=0x0, duration=2545.377s, table=60, n_packets=692, n_bytes=62495, priority=0 actions=goto_table:70
cookie=0x0, duration=2545.376s, table=60, n_packets=151, n_bytes=19311, priority=2048,ip,reg3=0x403,nw_dst=10.1.0.0/24 actions=set_field:fa:16:3e:f2:49:0f->eth_src,dec_ttl,set_field:0x403->tun_id,goto_table:70
cookie=0x0, duration=2545.421s, table=70, n_packets=642, n_bytes=55113, priority=0 actions=goto_table:80
cookie=0x0, duration=2545.421s, table=70, n_packets=90, n_bytes=11198, priority=1024,ip,tun_id=0x403,nw_dst=10.1.0.3 actions=set_field:fa:16:3e:49:d9:8c->eth_dst,goto_table:80
cookie=0x0, duration=2545.420s, table=70, n_packets=2, n_bytes=602, priority=1024,ip,tun_id=0x403,nw_dst=10.1.0.2 actions=set_field:fa:16:3e:13:08:4e->eth_dst,goto_table:80
cookie=0x0, duration=2545.421s, table=70, n_packets=82, n_bytes=11134, priority=1024,ip,tun_id=0x403,nw_dst=10.1.0.4 actions=set_field:fa:16:3e:df:82:56->eth_dst,goto_table:80
cookie=0x0, duration=2545.375s, table=80, n_packets=843, n_bytes=81806, priority=0 actions=goto_table:90
cookie=0x0, duration=2545.379s, table=90, n_packets=632, n_bytes=54020, priority=0 actions=goto_table:100
cookie=0x0, duration=2516.640s, table=90, n_packets=12, n_bytes=1695, priority=61007,ip,dl_dst=fa:16:3e:f3:f4:be actions=goto_table:100
cookie=0x0, duration=2545.379s, table=90, n_packets=82, n_bytes=11134, priority=61007,ip,dl_dst=fa:16:3e:df:82:56 actions=goto_table:100
cookie=0x0, duration=2545.379s, table=90, n_packets=90, n_bytes=11198, priority=61007,ip,dl_dst=fa:16:3e:49:d9:8c actions=goto_table:100
cookie=0x0, duration=2545.379s, table=90, n_packets=0, n_bytes=0, priority=61006,udp,dl_src=fa:16:3e:13:08:4e,tp_src=67,tp_dst=68 actions=goto_table:100
cookie=0x0, duration=2545.397s, table=100, n_packets=144, n_bytes=16248, priority=512,ip,tun_id=0x403,dl_dst=fa:16:3e:f2:49:0f,nw_src=10.1.0.3 actions=set_field:fa:16:3e:f1:10:3f->eth_src,dec_ttl,set_field:00:0c:29:2f:91:39->eth_dst,set_field:192.168.35.6->ip_src,output:3
cookie=0x0, duration=2545.398s, table=100, n_packets=48, n_bytes=5627, priority=512,ip,tun_id=0x403,dl_dst=fa:16:3e:f2:49:0f,nw_src=10.1.0.4 actions=set_field:fa:16:3e:5e:43:c7->eth_src,dec_ttl,set_field:00:0c:29:2f:91:39->eth_dst,set_field:192.168.35.7->ip_src,output:3
cookie=0x0, duration=2545.420s, table=100, n_packets=404, n_bytes=28211, priority=0 actions=goto_table:110
cookie=0x0, duration=2545.419s, table=100, n_packets=247, n_bytes=31720, priority=1024,ip,tun_id=0x403,nw_dst=10.1.0.0/24 actions=goto_table:110
cookie=0x0, duration=2545.373s, table=110, n_packets=45, n_bytes=4358, priority=8192,tun_id=0x403 actions=drop
cookie=0x0, duration=2545.374s, table=110, n_packets=267, n_bytes=16140, priority=0 actions=drop
cookie=0x0, duration=2545.369s, table=110, n_packets=66, n_bytes=5178, priority=16383,reg0=0x1,tun_id=0x403,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=output:5,output:1,output:6,output:7
cookie=0x0, duration=2545.375s, table=110, n_packets=12, n_bytes=1334, priority=16384,reg0=0x2,tun_id=0x403,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=output:5,output:6,output:7
cookie=0x0, duration=2545.373s, table=110, n_packets=90, n_bytes=11198, tun_id=0x403,dl_dst=fa:16:3e:49:d9:8c actions=output:6
cookie=0x0, duration=2516.569s, table=110, n_packets=25, n_bytes=2241, tun_id=0x403,dl_dst=fa:16:3e:f3:f4:be actions=output:7
cookie=0x0, duration=2545.369s, table=110, n_packets=82, n_bytes=11134, tun_id=0x403,dl_dst=fa:16:3e:df:82:56 actions=output:5
cookie=0x0, duration=2207.602s, table=110, n_packets=4, n_bytes=280, tun_id=0x403,dl_dst=fa:16:3e:08:5e:d8 actions=output:1
cookie=0x0, duration=2545.373s, table=110, n_packets=2, n_bytes=602, tun_id=0x403,dl_dst=fa:16:3e:13:08:4e actions=output:1

br-ex

[stack@netvirt-compute devstack]$ sudo ovs-ofctl dump-flows br-ex -OOpenFlow13
OFPST_FLOW reply (OF1.3) (xid=0x2):
cookie=0x0, duration=2566.494s, table=0, n_packets=776, n_bytes=67166, priority=0 actions=NORMAL
cookie=0x0, duration=2566.506s, table=0, n_packets=1374, n_bytes=149446, dl_type=0x88cc actions=CONTROLLER:65535

Md-Sal Stuff for debugging...

odl@odl-ovsdb-test:~/ovsdb$ python ./resources/commons/showOvsdbMdsal.py
aliasMap:
alpha -> openflow:52237137456 br-ex 00:00:00:0c:29:93:76:30
bravo -> openflow:129967535536201 br-int 00:00:76:34:6c:c9:b8:49
charlie -> openflow:52243196707 br-ex 00:00:00:0c:29:ef:eb:23
delta -> openflow:191435814915136 br-int 00:00:ae:1c:1f:be:cc:40

ovsdbNode:192.168.2.149:35967 mgr:192.168.2.147:6640 version:2.3.1
alpha:br-ex
of:1 ens33
of:2 patch-int
delta:br-int
of:1 tap25963115-4e mac:fa:16:3e:13:08:4e ifaceId:25963115-4e16-4d9d-aa2c-e25492d8a271
of:2 patch-ext
of:3 vxlan-192.168.2.150
of:4 tap464c44ea-01 mac:fa:16:3e:08:5e:d8 ifaceId:464c44ea-0169-4bed-845f-ccf0f17d3dc0
ovsdbNode:192.168.2.150:55444 mgr:192.168.2.147:6640 version:2.3.1
bravo:br-int
of:1 vxlan-192.168.2.149
of:3 patch-ext
of:5 tape9efbabe-bd mac:fa:16:3e:df:82:56 ifaceId:e9efbabe-bda0-45f8-a1e8-937d349677c0
of:6 tapb5795cb3-b9 mac:fa:16:3e:49:d9:8c ifaceId:b5795cb3-b905-4e06-a322-c156f843ad62
of:7 tap05b23c57-03 mac:fa:16:3e:f3:f4:be ifaceId:05b23c57-0372-4f4e-91d4-625671f7a7f9
charlie:br-ex
of:1 ens33
of:2 patch-int

operational tree flows at alpha
table 0: LLDP
table 0: NORMAL

operational tree flows at bravo
table 0: DEFAULT_PIPELINE_FLOW_0
table 0: DropFilter_5
table 0: DropFilter_6
table 0: DropFilter_7
table 0: LLDP
table 0: LocalMac_1027_5_fa:16:3e:df:82:56
table 0: LocalMac_1027_6_fa:16:3e:49:d9:8c
table 0: LocalMac_1027_7_fa:16:3e:f3:f4:be
table 0: TunnelIn_1027_1
table 20: ArpResponder_1027_10.1.0.1
table 20: ArpResponder_1027_10.1.0.2
table 20: ArpResponder_1027_10.1.0.3
table 20: ArpResponder_1027_10.1.0.4
table 20: ArpResponder_OFPort|3_192.168.35.6
table 20: ArpResponder_OFPort|3_192.168.35.7
table 20: DEFAULT_PIPELINE_FLOW_20
table 30: DEFAULT_PIPELINE_FLOW_30
table 30: InboundNAT_3_1027_192.168.35.6
table 30: InboundNAT_3_1027_192.168.35.7
table 40: DEFAULT_PIPELINE_FLOW_40
table 40: Egress_Allow_VM_IP_MAC_5fa:16:3e:df:82:56_Permit_
table 40: Egress_Allow_VM_IP_MAC_6fa:16:3e:49:d9:8c_Permit_
table 40: Egress_Allow_VM_IP_MAC_7fa:16:3e:f3:f4:be_Permit_
table 40: Egress_DHCP_Server_5_DROP_
table 40: Egress_DHCP_Server_6_DROP_
table 40: Egress_DHCP_Server_7_DROP_
table 40: Egress_IP1027_fa:16:3e:49:d9:8c_Permit_
table 40: Egress_IP1027_fa:16:3e:df:82:56_Permit_
table 40: Egress_IP1027_fa:16:3e:f3:f4:be_Permit_
table 50: DEFAULT_PIPELINE_FLOW_50
table 60: DEFAULT_PIPELINE_FLOW_60
table 60: Routing_external_1027_10.1.0.0/24
table 70: DEFAULT_PIPELINE_FLOW_70
table 70: L3Forwarding_1027_10.1.0.2
table 70: L3Forwarding_1027_10.1.0.3
table 70: L3Forwarding_1027_10.1.0.4
table 80: DEFAULT_PIPELINE_FLOW_80
table 90: DEFAULT_PIPELINE_FLOW_90
table 90: Ingress_DHCP_Server1027_FA:16:3E:13:08:4E_Permit_
table 90: Ingress_IP1027_fa:16:3e:49:d9:8c_Permit_
table 90: Ingress_IP1027_fa:16:3e:df:82:56_Permit_
table 90: Ingress_IP1027_fa:16:3e:f3:f4:be_Permit_
table 100: DEFAULT_PIPELINE_FLOW_100
table 100: OutboundNATExclusion_1027_10.1.0.0/24
table 100: OutboundNAT_1027_10.1.0.3
table 100: OutboundNAT_1027_10.1.0.4
table 110: BcastOut_1027
table 110: DEFAULT_PIPELINE_FLOW_110
table 110: LocalTableMiss_1027
table 110: TunnelFloodOut_1027
table 110: TunnelOut_1027_1_fa:16:3e:08:5e:d8
table 110: TunnelOut_1027_1_fa:16:3e:13:08:4e
table 110: UcastOut_1027_5_fa:16:3e:df:82:56
table 110: UcastOut_1027_6_fa:16:3e:49:d9:8c
table 110: UcastOut_1027_7_fa:16:3e:f3:f4:be

operational tree flows at charlie
table 0: LLDP
table 0: NORMAL

operational tree flows at delta
table 0: DEFAULT_PIPELINE_FLOW_0
table 0: DropFilter_1
table 0: DropFilter_4
table 0: LLDP
table 0: LocalMac_1027_1_fa:16:3e:13:08:4e
table 0: LocalMac_1027_4_fa:16:3e:08:5e:d8
table 0: TunnelIn_1027_3
table 20: ArpResponder_1027_10.1.0.1
table 20: ArpResponder_1027_10.1.0.2
table 20: ArpResponder_1027_10.1.0.3
table 20: ArpResponder_1027_10.1.0.4
table 20: DEFAULT_PIPELINE_FLOW_20
table 30: DEFAULT_PIPELINE_FLOW_30
table 40: DEFAULT_PIPELINE_FLOW_40
table 40: Egress_Allow_VM_IP_MAC_4fa:16:3e:08:5e:d8_Permit_
table 40: Egress_DHCP_Client_Permit_
table 40: Egress_DHCP_Server_4_DROP_
table 40: Egress_IP1027_fa:16:3e:08:5e:d8_Permit_
table 50: DEFAULT_PIPELINE_FLOW_50
table 60: DEFAULT_PIPELINE_FLOW_60
table 60: Routing_external_1027_10.1.0.0/24
table 70: DEFAULT_PIPELINE_FLOW_70
table 70: L3Forwarding_1027_10.1.0.2
table 70: L3Forwarding_1027_10.1.0.3
table 70: L3Forwarding_1027_10.1.0.4
table 80: DEFAULT_PIPELINE_FLOW_80
table 90: DEFAULT_PIPELINE_FLOW_90
table 90: Ingress_DHCP_Server1027_FA:16:3E:13:08:4E_Permit_
table 90: Ingress_IP1027_fa:16:3e:08:5e:d8_Permit_
table 100: DEFAULT_PIPELINE_FLOW_100
table 100: OutboundNATExclusion_1027_10.1.0.0/24
table 110: BcastOut_1027
table 110: DEFAULT_PIPELINE_FLOW_110
table 110: LocalTableMiss_1027
table 110: TunnelFloodOut_1027
table 110: TunnelOut_1027_3_fa:16:3e:49:d9:8c
table 110: TunnelOut_1027_3_fa:16:3e:df:82:56
table 110: TunnelOut_1027_3_fa:16:3e:f3:f4:be
table 110: UcastOut_1027_1_fa:16:3e:13:08:4e
table 110: UcastOut_1027_4_fa:16:3e:08:5e:d8

ofLinks (discover via lldp):
bravo:1 <-> delta:3
bravo:3 <-> charlie:2
delta:2 <-> alpha:2
alpha:1 <-> charlie:1

This issues seems to be related to gateway mac address resolution. We don't really persist all the gateway ip that need to be resolved for the MAC, so when we restart the controller we loose those caches. This is my first guess, i need to investigate more.

Hi Anil, Hi Venkatrangan,

Please, let me extend the description of this bug. If you have a clean neutron + clean ODL with "odl-ovsdb-sfc" feature and you create networks, subnetworks and instances using GRE/VXLAN. They will be able to ping, to enforce security groups, multi-tenancy isolation, etc. However, if you "reboot" the infrastructure (soft shutdown with proper shuting down of all services). Then, you will see how the "neutron gateway" cannot react anymore any instances. In fact, you will not receive any IP address via DHCP, etc from the gateway.

It means that the problem is not related to the "creation of instances". It seems that there is a mismatch sync between the namespaces, tunnels and ODL controller. This bug is weird because jenkins can pass it even if the bug still there.

I have been working in the bug for hours so happy to provide any information you may required. i can reproduce it many times.

Using: liberty multi-node over trusty with GRE/VXLAN tunnels + ovs 2.3.2 + ODL 0.4.0-SNAPSHOP (generated 4 days ago).

No errors reported in OVS, Neutron-DHCP, Nova-Controller, Neutron-Server, ODL and Nova-Compute.

I know that I'm using "odl-ovsdb-sfc". However, notice that I said clean ODL so that no any SFC rules have been even defined. It seams that it is a problem related to OVSDB.

Thanks in advance,
Jose

Tested with stable/beryllium l3 features are working for instances as expected even after rebooting controller. Hence we are closing the bug.