[#OPNFLWPLUG-835] Bridge br-int added but flows not created after OVS stop and start in control node and compute nodes

[OPNFLWPLUG-835] Bridge br-int added but flows not created after OVS stop and start in control node and compute nodes Created: 30/Dec/16 Updated: 27/Sep/21 Resolved: 06/Mar/17
Status:	Resolved
Project:	OpenFlowPlugin
Component/s:	General
Affects Version/s:	None
Fix Version/s:	None

Type:

Bug

Reporter:

ranjithkumar_t

Assignee:

Bertrand Low

Resolution:

Done

Votes:

Labels:

None

Remaining Estimate:

Not Specified

Time Spent:

Not Specified

Original Estimate:

Not Specified

Environment:

Operating System: All
Platform: All

Attachments:

NoFlows.tar.gz

ODL1karaf log -10.106.138.68.zip

ODL2 karaf log-10.106.138.110.zip

ODL3 karaf log-10.106.138.154.zip

OVS Control node details.jpg

OVS details.zip

Success.tar.gz

karaf logs.zip

External issue ID:

7462

Description

ODL Version : distribution-karaf-0.5.2-Boron-SR2

Openstack Version : mitaka

Setup Details : 3 ODL Nodes + 1 Control Node + 2 Compute Nodes

Steps to Reproduce:
1. OVS Clean
sudo systemctl stop openvswitch
sudo rm -rf /var/log/openvswitch/*
sudo rm -rf /etc/openvswitch/conf.db
sudo systemctl start openvswitch
sudo ovs-vsctl show

2. Stop All 3 ODLs
cd /root/distribution-karaf-0.5.2-Boron-SR2/bin
LANG=C;./stop
ps -ef | grep java
Note:
Verify controller is not running for all 3 ODL nodes

3. Clean data, journal and snapshot for all 3 ODL nodes and start ODL
cd /root/distribution-karaf-0.5.2-Boron-SR2/bin
LANG=C;./start

4. Wait for a while ODL starting log back to normal
Note:
Wait some times to verify whether ODLs are up, cluster forms and initial flows initiated in operational data store for all 3 ODL nodes.

5. OVS setting
sudo ovs-vsctl set-manager tcp:10.106.138.68:6640 tcp:10.106.138.110:6640 tcp:10.106.138.154:6640

6. OVS stop and start
sudo systemctl stop openvswitch
sudo systemctl start openvswitch

Observation:
When we run step (6) repeatedly in control and compute nodes without delay. The bridge "br-int" is added but flows not created in control node and in compute nodes bridge "br-int" is not getting created.

Please find the attached files for karaf logs and OVS details.

Comments

Comment by ranjithkumar_t [ 30/Dec/16 ]

Attachment karaf logs.zip has been added with description: Karaf logs for your reference

Comment by ranjithkumar_t [ 30/Dec/16 ]

Attachment OVS details.zip has been added with description: OVS details for your reference

Comment by Bertrand Low [ 13/Jan/17 ]

The problem is the immediate succession of starting the ovs-vswitchd process after stopping it when the device is connected to an ODL cluster; using “systemctl restart openvswitch” has the same effect.

This issue is not observed in a non-clustered setup, single ODL setup.

If there is even a 2s (two second) delay between stopping and starting, installation of br-int and flows is successful.

Resetting the ovs-manager appears to successfully install br-int and the flows.

Comment by Bertrand Low [ 19/Jan/17 ]

Please see related ~~NETVIRT-402~~ for a description of a possible similar timing issue that openflow plugin may be facing here.

The difference compared to ~~NETVIRT-402~~ seems to be that no request to delete br-int makes it to the device, so the br-int is present but FRM sees the node as removed so flows are not pushed down.

I've attached a couple of logs: one which shows what happens when the openvswitch process is restarted on the device and br-int gets successfully installed but not the flows, and also a successful case where the flows get installed as well (FRM sees the node as added due to the timing order of the events).
In the successful case, there is some difficulty in connecting with odl33 as observed in the logs, however odl34 ends up pushing the flows down to the device.

In both of these log captures, the controller has issues connecting to odl35, but this may prove not to be meaningful.

Comment by Bertrand Low [ 19/Jan/17 ]

Attachment NoFlows.tar.gz has been added with description: no flows installed

Comment by Bertrand Low [ 19/Jan/17 ]

Attachment Success.tar.gz has been added with description: successful flow installation after restarting openvswitch