[OVSDB-144] plugin set two controller's for integration bridge Created: 31/Mar/15  Updated: 26/Jun/15  Resolved: 26/Jun/15

Status: Resolved
Project: ovsdb
Component/s: Plugin
Affects Version/s: unspecified
Fix Version/s: None

Type: Bug
Reporter: Anil Vishnoi Assignee: Anil Vishnoi
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: All
Platform: All


External issue ID: 2925
Priority: Normal

 Description   

In my two node openstack setup, when i try to connect my compute node ovsdb instance to controller, i see that controller sets two controller value for the integration bridge. I can easily recreate this issue, after 2-3 attempts.

root@ComputeNode:~/openstack-setup# ovs-vsctl show
4575bb26-b73b-4e0a-a62a-9b3ff06e19af
Manager "tcp:192.168.57.1:6640"
is_connected: true
Bridge br-int
Controller "tcp:192.168.57.1:6633"
is_connected: true
Controller "tcp:192.168.57.1:6633"
is_connected: true
fail_mode: secure
Port br-int
Interface br-int
ovs_version: "2.0.2"



 Comments   
Comment by Anil Vishnoi [ 31/Mar/15 ]

The root cause of this problem is a race condition while setting up the openflow controller for bridge. When ovsdb node connects to the controller, it generate NODE event, that results in creating br-int and setting up the OF controller. Once bridge table is modified with br-int entry, it generate table update for Bridge table. Ovsdb Plugin receives this table update and it's default behavior is to set of controller for any new bridge, so when it gets the bridge table update, it again tries to set the openflow controller for the same bridge (br-int) and that causes the race condition, given that both the actions happens in two separate thread. Following are the two stack that try to set the OF controller for the bridge when ovsdb node connects to the controller first time.

1)
java.lang.Exception: Stack trace
at java.lang.Thread.dumpStack(Thread.java:1365)
at org.opendaylight.ovsdb.plugin.impl.ConfigurationServiceImpl.setOFController(ConfigurationServiceImpl.java:407)
at org.opendaylight.ovsdb.openstack.netvirt.impl.BridgeConfigurationManagerImpl.addBridge(BridgeConfigurationManagerImpl.java:677)
at org.opendaylight.ovsdb.openstack.netvirt.impl.BridgeConfigurationManagerImpl.createIntegrationBridge(BridgeConfigurationManagerImpl.java:376)
at org.opendaylight.ovsdb.openstack.netvirt.impl.BridgeConfigurationManagerImpl.prepareNode(BridgeConfigurationManagerImpl.java:182)
at org.opendaylight.ovsdb.openstack.netvirt.SouthboundHandler.processNodeUpdate(SouthboundHandler.java:128)
at org.opendaylight.ovsdb.openstack.netvirt.SouthboundHandler.processEvent(SouthboundHandler.java:345)
at org.opendaylight.ovsdb.openstack.netvirt.impl.EventDispatcherImpl.dispatchEvent(EventDispatcherImpl.java:92)
at org.opendaylight.ovsdb.openstack.netvirt.impl.EventDispatcherImpl.access$100(EventDispatcherImpl.java:28)
at org.opendaylight.ovsdb.openstack.netvirt.impl.EventDispatcherImpl$1.run(EventDispatcherImpl.java:55)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

2)
java.lang.Exception: Stack trace
at java.lang.Thread.dumpStack(Thread.java:1365)
at org.opendaylight.ovsdb.plugin.impl.ConfigurationServiceImpl.setOFController(ConfigurationServiceImpl.java:407)
at org.opendaylight.ovsdb.plugin.impl.InventoryServiceImpl$1.run(InventoryServiceImpl.java:184)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

Second stack is the one that set's the OF controller for any new bridge.

Although setOFController check for the existing controller, but it checks the internal cache for that, and internal cache won't get updated till controller gets the update from ovsdb node for Controller table update. So this race condition happens in the duration between controller sent the "set controller for br-int" to ovsdb-node and it gets updated in internal cache.

From Openstack integration perspective, i think we should not explicitly set controller for each bridge that we find on ovsdb node. I am referring to following code in InventoryServiceImpl.java

private void handleOpenVSwitchSpecialCase(final Node node, final String databaseName, final String tableName, final UUID uuid) {
if (OvsVswitchdSchemaConstants.shouldConfigureController(databaseName, tableName)) {
Runnable updateControllerRunnable = new Runnable() {
@Override
public void run() {
try

{ if (ovsdbConfigurationService != null) ovsdbConfigurationService.setOFController(node, uuid.toString()); }

catch (InterruptedException | ExecutionException e)

{ e.printStackTrace(); }

}
};
executor.execute(updateControllerRunnable);
}
}

Above method is called from method processTableUpdates() in the same class

if (update.getNew(uuid) != null) {
boolean isNewRow = (tCache == null || tCache.get(uuid.toString()) == null) ? true : false;
db.updateRow(databaseName, tableName, uuid.toString(), update.getNew(uuid));
if (isNewRow) {
this.handleOpenVSwitchSpecialCase(n, databaseName, tableName, uuid);
if (!ovsdbInventoryListeners.isEmpty()) {
for (OvsdbInventoryListener listener : ovsdbInventoryListeners)

{ listener.rowAdded(n, tableName, uuid.toString(), update.getNew(uuid)); }

}

Once i disable "setting OF controller for every new bridge" behavior, it fixes the issue.

IMO assuming that compute node ovsdb instance will be clean (no bridges) before it connects to the controller is absolutely fine. After that ovsdb instance will only persist the configuration that is created by controller. This will give more controlled environment for controller to make sure that things work as expected.

Comment by Anil Vishnoi [ 31/Mar/15 ]

If any external project wants to enable the behavior of setting OF Controller for every new bridge, they can override the value in controller config file to enable that. Although it's good if that can be exposed through config sub system rather then a config file.

Comment by Anil Vishnoi [ 31/Mar/15 ]

https://git.opendaylight.org/gerrit/17439

Comment by Flavio Fernandes [ 31/Mar/15 ]

(In reply to Anil Vishnoi from comment #1)
> The root cause of this problem is a race condition while setting up the
> openflow controller for bridge. When ovsdb node connects to the controller,
> it generate NODE event, that results in creating br-int and setting up the
> OF controller. Once bridge table is modified with br-int entry, it generate
> table update for Bridge table. Ovsdb Plugin receives this table update and
> it's default behavior is to set of controller for any new bridge, so when it
> gets the bridge table update, it again tries to set the openflow controller
> for the same bridge (br-int) and that causes the race condition, given that
> both the actions happens in two separate thread. Following are the two stack
> that try to set the OF controller for the bridge when ovsdb node connects to
> the controller first time.
>
> 1)
> java.lang.Exception: Stack trace
> at java.lang.Thread.dumpStack(Thread.java:1365)
> at
> org.opendaylight.ovsdb.plugin.impl.ConfigurationServiceImpl.
> setOFController(ConfigurationServiceImpl.java:407)
> at
> org.opendaylight.ovsdb.openstack.netvirt.impl.BridgeConfigurationManagerImpl.
> addBridge(BridgeConfigurationManagerImpl.java:677)
> at
> org.opendaylight.ovsdb.openstack.netvirt.impl.BridgeConfigurationManagerImpl.
> createIntegrationBridge(BridgeConfigurationManagerImpl.java:376)
> at
> org.opendaylight.ovsdb.openstack.netvirt.impl.BridgeConfigurationManagerImpl.
> prepareNode(BridgeConfigurationManagerImpl.java:182)
> at
> org.opendaylight.ovsdb.openstack.netvirt.SouthboundHandler.
> processNodeUpdate(SouthboundHandler.java:128)
> at
> org.opendaylight.ovsdb.openstack.netvirt.SouthboundHandler.
> processEvent(SouthboundHandler.java:345)
> at
> org.opendaylight.ovsdb.openstack.netvirt.impl.EventDispatcherImpl.
> dispatchEvent(EventDispatcherImpl.java:92)
> at
> org.opendaylight.ovsdb.openstack.netvirt.impl.EventDispatcherImpl.
> access$100(EventDispatcherImpl.java:28)
> at
> org.opendaylight.ovsdb.openstack.netvirt.impl.EventDispatcherImpl$1.
> run(EventDispatcherImpl.java:55)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
> 1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:
> 615)
> at java.lang.Thread.run(Thread.java:745)
>
> 2)
> java.lang.Exception: Stack trace
> at java.lang.Thread.dumpStack(Thread.java:1365)
> at
> org.opendaylight.ovsdb.plugin.impl.ConfigurationServiceImpl.
> setOFController(ConfigurationServiceImpl.java:407)
> at
> org.opendaylight.ovsdb.plugin.impl.InventoryServiceImpl$1.
> run(InventoryServiceImpl.java:184)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.
> access$201(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.
> run(ScheduledThreadPoolExecutor.java:292)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
> 1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:
> 615)
> at java.lang.Thread.run(Thread.java:745)
>
> Second stack is the one that set's the OF controller for any new bridge.
>
> Although setOFController check for the existing controller, but it checks
> the internal cache for that, and internal cache won't get updated till
> controller gets the update from ovsdb node for Controller table update. So
> this race condition happens in the duration between controller sent the "set
> controller for br-int" to ovsdb-node and it gets updated in internal cache.
>
> From Openstack integration perspective, i think we should not explicitly set
> controller for each bridge that we find on ovsdb node. I am referring to
> following code in InventoryServiceImpl.java
>
> private void handleOpenVSwitchSpecialCase(final Node node, final String
> databaseName, final String tableName, final UUID uuid) {
> if
> (OvsVswitchdSchemaConstants.shouldConfigureController(databaseName,
> tableName)) {
> Runnable updateControllerRunnable = new Runnable() {
> @Override
> public void run() {
> try

{ > if (ovsdbConfigurationService != null) > ovsdbConfigurationService.setOFController(node, uuid.toString()); > }

catch (InterruptedException | ExecutionException e)

{ > e.printStackTrace(); > }

> }
> };
> executor.execute(updateControllerRunnable);
> }
> }
>
> Above method is called from method processTableUpdates() in the same class
>
> if (update.getNew(uuid) != null) {
> boolean isNewRow = (tCache == null ||
> tCache.get(uuid.toString()) == null) ? true : false;
> db.updateRow(databaseName, tableName, uuid.toString(),
> update.getNew(uuid));
> if (isNewRow) {
> this.handleOpenVSwitchSpecialCase(n, databaseName,
> tableName, uuid);
> if (!ovsdbInventoryListeners.isEmpty()) {
> for (OvsdbInventoryListener listener :
> ovsdbInventoryListeners)

{ > listener.rowAdded(n, tableName, uuid.toString(), > update.getNew(uuid)); > }

> }
>
> Once i disable "setting OF controller for every new bridge" behavior, it
> fixes the issue.
>
> IMO assuming that compute node ovsdb instance will be clean (no bridges)
> before it connects to the controller is absolutely fine. After that ovsdb
> instance will only persist the configuration that is created by controller.
> This will give more controlled environment for controller to make sure that
> things work as expected.

Very good catch, Anil! I don't think we can assume there are no bridges in
ovs when we connect it to the manager. In some start-up scripts, openstack
explicitly creates br-int before connecting to ODL. Also, br-ex is only
created by openstack. Any ideas on how we could avoid the race w/out losing
the ability to set controller in cases when bridge is already created?

Generated at Wed Feb 07 20:35:37 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.