[NETVIRT-402] Bridge br-int is not getting created after OVS stop and start in control node and compute nodes Created: 30/Dec/16  Updated: 09/Mar/18  Resolved: 29/Jan/17

Status: Resolved
Project: netvirt
Component/s: None
Affects Version/s: Boron
Fix Version/s: None

Type: Bug
Reporter: ranjithkumar_t Assignee: Bertrand Low
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: All
Platform: All

Attachments: File NoBridge-NoDpidChange.tar.gz     File NoBridge-WithDpidChange.tar.gz     Zip Archive OVS details.zip     Zip Archive karaf logs.zip    
External issue ID: 7461

 Description   

ODL Version : distribution-karaf-0.5.2-Boron-SR2

Openstack Version : mitaka

Setup Details : 3 ODL Nodes + 1 Control Node + 2 Compute Nodes

Steps to Reproduce:
1. OVS Clean
sudo systemctl stop openvswitch
sudo rm -rf /var/log/openvswitch/*
sudo rm -rf /etc/openvswitch/conf.db
sudo systemctl start openvswitch
sudo ovs-vsctl show

2. Stop All 3 ODLs
cd /root/distribution-karaf-0.5.2-Boron-SR2/bin
LANG=C;./stop
ps -ef | grep java
Note:
Verify controller is not running for all 3 ODL nodes

3. Clean data, journal and snapshot for all 3 ODL nodes and start ODL
cd /root/distribution-karaf-0.5.2-Boron-SR2/bin
LANG=C;./start

4. Wait for a while ODL starting log back to normal
Note:
Wait some time to verify whether ODLs are up, cluster forms and initial flows initiated in operational data store for all 3 ODL nodes.

5. OVS setting
sudo ovs-vsctl set-manager tcp:10.106.138.68:6640 tcp:10.106.138.110:6640 tcp:10.106.138.154:6640

6. OVS stop and start
sudo systemctl stop openvswitch
sudo systemctl start openvswitch

Observation:
When we run step (6) repeatedly in control and Compute nodes without delay. The bridge "br-int" is not getting created in control node and compute nodes.

Please find the attached files for karaf logs and OVS details.

 Comments   
Comment by ranjithkumar_t [ 30/Dec/16 ]

Attachment karaf logs.zip has been added with description: Karaf logs for your reference

Comment by ranjithkumar_t [ 30/Dec/16 ]

Attachment OVS details.zip has been added with description: OVS details for your reference

Comment by Bertrand Low [ 19/Jan/17 ]

As with OPNFLWPLUG-835 the problem is the immediate succession of starting the ovs-vswitchd process after stopping it when the device is connected to an ODL cluster; using “systemctl restart openvswitch” has the same effect.

This issue is not observed in a non-clustered setup, single ODL setup.

If there is even a 2s (two second) delay between stopping and starting, installation of br-int and flows is successful.

Resetting the ovs-manager appears to successfully install br-int and the flows.

For NETVIRT-402 where br-int is not installed, some variation of the following sequence of events is happening*:
*note that if there is an alphabetical suffix for the step, these steps may be reversed in some instances. E.g Sometimes, 2b) comes before 2a) and sometimes 3b) comes before 3a)

Karaf.log
1) Connection closed by device (due to "restart openvswitch")
2a) ODL deleteBridge br-int from config DS
2b) Device connected to controller (due to “restart openvswitch”)
3a) Connection closed by device
3b) ODL addBridge br-int to config DS
4) Device connected to controller (due to connection flap)
5a) ODL deleteBridge br-int from config DS
5b) Connection closed by device

Note that ODL processes a deleteBridge after the addBridge.

On the device-side, it appears that before Step 3a) the device receives a mutation request from ODL for “deleting ovsdbbridgename”. Moments later, the connection flaps and in some instances, the br-int dpid changes.

I've attached the logs for two instances of this bug, one where the dpid remains the same, and another one where the dpid changes.

Comment by Bertrand Low [ 19/Jan/17 ]

Attachment NoBridge-NoDpidChange.tar.gz has been added with description: no br-int but dpid does not change

Comment by Bertrand Low [ 19/Jan/17 ]

Attachment NoBridge-WithDpidChange.tar.gz has been added with description: no br-int but dpid changes

Comment by Bertrand Low [ 26/Jan/17 ]

https://git.opendaylight.org/gerrit/#/c/51058/

Comment by Bertrand Low [ 29/Jan/17 ]

cherry-picked into master:
https://git.opendaylight.org/gerrit/#/c/51161/

Generated at Wed Feb 07 20:21:28 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.