Operating System: All
Platform: All

Additional details of the scenario

-> ODL in cluster mode

-> Control Node up

-> Created BNetwork in Openstack

-> Bring up compute node

-> create instances

-> IP for instances got assigned, but the instacne in compute node not able to reach the dhcp IP

Looking at flows, it seems the flow to get back from the compute node to the control node for the dhcp mac is not in table 110.

Can we enable TRACE level on the netvirt code (at least for Of13Provider) and that might show us why that flow is not programmed.

• • • On the control node, port 1 is the dhcp port with mac=dl_src=fa:16:3e:d0:1f:ad

ubuntu@odl-os-control-node:~/devstack$ sudo ovs-ofctl dump-flows br-int -OOpenFlow13
OFPST_FLOW reply (OF1.3) (xid=0x2):
cookie=0x0, duration=2004.335s, table=0, n_packets=67, n_bytes=8000, in_port=1,dl_src=fa:16:3e:d0:1f:ad actions=set_field:0x426->tun_id,load:0x1->NXM_NX_REG0[],goto_table:20
cookie=0x0, duration=1182.896s, table=0, n_packets=44, n_bytes=3754, cookie=0x0, duration=1179.898s, table=110, n_packets=13, n_bytes=1800,

• • • Flow to go out tunnel to reach vm on compute node:

tun_id=0x426,dl_dst=fa:16:3e:9e:f6:70 actions=output:3

• • • On the control node, port 2 is the vm port with mac=dl_src=fa:16:3e:9e:f6:70

ubuntu@odl-os-compute-node:~/devstack$ sudo ovs-ofctl dump-flows br-int -OOpenFlow13
OFPST_FLOW reply (OF1.3) (xid=0x2):
cookie=0x0, duration=1217.089s, table=0, n_packets=58, n_bytes=4928, in_port=2,dl_src=fa:16:3e:9e:f6:70 actions=set_field:0x426->tun_id,load:0x1->NXM_NX_REG0[],goto_table:20

• • • but on the compute node there is not a flow to send back to the dhcp mac over the tunnel. All we have are these two flows below. The first is to forward traffic over the tunnel for the vm on the control node at 1.2.3.6. The second flow is to forward traffic locally to the 1.2.3.5 vm.

cookie=0x0, duration=1219.169s, table=110, n_packets=3, n_bytes=294, tun_id=0x426,dl_dst=fa:16:3e:de:47:a5 actions=output:1
cookie=0x0, duration=1216.963s, table=110, n_packets=13, n_bytes=1800, tun_id=0x426,dl_dst=fa:16:3e:9e:f6:70 actions=output:2

Disabled iptables in all compute nodes, all intances are reachable now when using ODL without clustering configutation.

There seems to be problems exisitng when ODL is used as 3node, there are multiple exceptions observed while port event handling and other processing.

Link: https://jenkins.opendaylight.org/releng/view/netvirt/job/netvirt-csit-3node-openstack-liberty-openstack-beryllium/30/artifact/odl1_karaf.log.xz

Some important flows in pipeline is missed in all the openstack nodes. This needs to be investigated immediately to have a stable 3node job.

ODL1 log

Attachment ODL1.zip has been added with description: ODL1 log

Hi Sam,
i have created 3node cluster setup in local and tested below scenario.

1.Running 3 ODL controller as cluster with HA Proxy connected 1control and 2 compute nodes.
2.created 2 networks.
3.net1 subnet range 10.0.0.0/24.
4.net2 subnet range 20.0.0.0/24.
5.bring down ODL1.
6.created 3VM instance.

first 2 VM(10.0.0.3, 10.0.0.4) instance getting IP and the flows are created properly in openstack nodes.
3rd VM (10.0.0.5) Not getting IP address in the VM console also in DUMP flows
only ARP flows present no flow entries in table 40,90 and 110.

enabled Debug log for southboudhandler and porthandler.
in the log i am able to see some failure in southbound event dispatcher.

OFPST_FLOW reply (OF1.3) (xid=0x2):
cookie=0x0, duration=300.163s, table=20, n_packets=1, n_bytes=42, priority=1024,arp,tun_id=0x444,arp_tpa=10.0.0.5,arp_op=1 actions=move:NXM_OF_ETH_SRC[]>NXM_OF_ETH_DST[],set_field:fa:16:3e:f5:cf:02>eth_src,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]>NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]>NXM_OF_ARP_TPA[],load:0xfa163ef5cf02->NXM_NX_ARP_SHA[],load:0xa000005->NXM_OF_ARP_SPA[],IN_PORT
cookie=0x0, duration=1676.577s, table=20, n_packets=0, n_bytes=0, priority=1024,arp,tun_id=0x444,arp_tpa=10.0.0.2,arp_op=1 actions=move:NXM_OF_ETH_SRC[]>NXM_OF_ETH_DST[],set_field:fa:16:3e:ea:30:af>eth_src,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]>NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]>NXM_OF_ARP_TPA[],load:0xfa163eea30af->NXM_NX_ARP_SHA[],load:0xa000002->NXM_OF_ARP_SPA[],IN_PORT
cookie=0x0, duration=1590.239s, table=20, n_packets=0, n_bytes=0, priority=1024,arp,tun_id=0x3fe,arp_tpa=20.0.0.2,arp_op=1 actions=move:NXM_OF_ETH_SRC[]>NXM_OF_ETH_DST[],set_field:fa:16:3e:9f:9d:7f>eth_src,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]>NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]>NXM_OF_ARP_TPA[],load:0xfa163e9f9d7f->NXM_NX_ARP_SHA[],load:0x14000002->NXM_OF_ARP_SPA[],IN_PORT
cookie=0x0, duration=683.569s, table=20, n_packets=1, n_bytes=42, priority=1024,arp,tun_id=0x444,arp_tpa=10.0.0.4,arp_op=1 actions=move:NXM_OF_ETH_SRC[]>NXM_OF_ETH_DST[],set_field:fa:16:3e:43:9a:8a>eth_src,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]>NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]>NXM_OF_ARP_TPA[],load:0xfa163e439a8a->NXM_NX_ARP_SHA[],load:0xa000004->NXM_OF_ARP_SPA[],IN_PORT
cookie=0x0, duration=1135.884s, table=20, n_packets=1, n_bytes=42, priority=1024,arp,tun_id=0x444,arp_tpa=10.0.0.3,arp_op=1 actions=move:NXM_OF_ETH_SRC[]>NXM_OF_ETH_DST[],set_field:fa:16:3e:58:a9:49>eth_src,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]>NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]>NXM_OF_ARP_TPA[],load:0xfa163e58a949->NXM_NX_ARP_SHA[],load:0xa000003->NXM_OF_ARP_SPA[],IN_PORT
cookie=0x0, duration=2632.505s, table=20, n_packets=118, n_bytes=12483, priority=0 actions=goto_table:30
cookie=0x0, duration=2632.505s, table=30, n_packets=118, n_bytes=12483, priority=0 actions=goto_table:40
cookie=0x0, duration=2632.505s, table=40, n_packets=118, n_bytes=12483, priority=0 actions=goto_table:50
cookie=0x0, duration=2632.505s, table=50, n_packets=118, n_bytes=12483, priority=0 actions=goto_table:60
cookie=0x0, duration=2632.505s, table=60, n_packets=118, n_bytes=12483, priority=0 actions=goto_table:70
cookie=0x0, duration=2632.505s, table=70, n_packets=118, n_bytes=12483, priority=0 actions=goto_table:80
cookie=0x0, duration=2632.505s, table=80, n_packets=118, n_bytes=12483, priority=0 actions=goto_table:90
cookie=0x0, duration=2632.505s, table=90, n_packets=118, n_bytes=12483, priority=0 actions=goto_table:100
cookie=0x0, duration=2632.505s, table=100, n_packets=118, n_bytes=12483, priority=0 actions=goto_table:110
cookie=0x0, duration=1673.671s, table=110, n_packets=31, n_bytes=2882, tun_id=0x444,dl_dst=fa:16:3e:ea:30:af actions=output:1
cookie=0x0, duration=1565.622s, table=110, n_packets=0, n_bytes=0, tun_id=0x3fe,dl_dst=fa:16:3e:9f:9d:7f actions=output:5
cookie=0x0, duration=998.128s, table=110, n_packets=19, n_bytes=2311, tun_id=0x444,dl_dst=fa:16:3e:58:a9:49 actions=output:3
cookie=0x0, duration=643.071s, table=110, n_packets=16, n_bytes=2017, tun_id=0x444,dl_dst=fa:16:3e:43:9a:8a actions=output:4
cookie=0x0, duration=1672.376s, table=110, n_packets=25, n_bytes=2326, priority=16384,reg0=0x2,tun_id=0x444,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=output:1
cookie=0x0, duration=1672.378s, table=110, n_packets=0, n_bytes=0, priority=16383,reg0=0x1,tun_id=0x444,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=output:1,output:3,output:4
cookie=0x0, duration=1565.611s, table=110, n_packets=0, n_bytes=0, priority=16384,reg0=0x2,tun_id=0x3fe,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=output:5
cookie=0x0, duration=1565.601s, table=110, n_packets=0, n_bytes=0, priority=16383,reg0=0x1,tun_id=0x3fe,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=output:5,output:3,output:4
cookie=0x0, duration=1672.285s, table=110, n_packets=2, n_bytes=196, priority=8192,tun_id=0x444 actions=drop
cookie=0x0, duration=1564.369s, table=110, n_packets=0, n_bytes=0, priority=8192,tun_id=0x3fe actions=drop
cookie=0x0, duration=2632.505s, table=110, n_packets=25, n_bytes=2751, priority=0 actions=drop

Attachment ODL2.zip has been added with description: ODL2 log

Attachment ODL3.zip has been added with description: ODL3 log

Attachment 26-5 FLows.txt has been added with description: dump flows for VM creation

I think one of reasons which caused this issue was a performance issue of the OVSDB Southbound plugin.

The following patches have dramatically increased the performance of the OVSDB Southbound Plugin, so I think the rate of occurrence of this issue has been decreased now.
https://git.opendaylight.org/gerrit/#/c/43580/
https://git.opendaylight.org/gerrit/#/c/44374/

We have tested with latest distribution after below patches were merged

https://git.opendaylight.org/gerrit/#/c/43580/
https://git.opendaylight.org/gerrit/#/c/44374/

Issue was not observed during multiple runs.We have observed, all created VM instances has acquired ip address.

Following bugs has been raised for issues found in fail-over scenario testing.

https://bugs.opendaylight.org/show_bug.cgi?id=6596

https://bugs.opendaylight.org/show_bug.cgi?id=6601

Hence,this bug can be closed.