https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/OVSDB-144 ovsdb <p>In my two node openstack setup, when i try to connect my compute node ovsdb instance to controller, i see that controller sets two controller value for the integration bridge. I can easily recreate this issue, after 2-3 attempts.</p> <p>root@ComputeNode:~/openstack-setup# ovs-vsctl show<br/> 4575bb26-b73b-4e0a-a62a-9b3ff06e19af<br/> Manager "tcp:192.168.57.1:6640"<br/> is_connected: true<br/> Bridge br-int<br/> Controller "tcp:192.168.57.1:6633"<br/> is_connected: true<br/> Controller "tcp:192.168.57.1:6633"<br/> is_connected: true<br/> fail_mode: secure<br/> Port br-int<br/> Interface br-int<br/> ovs_version: "2.0.2"</p> <p>Operating System: All<br/> Platform: All</p> OVSDB-144

plugin set two controller's for integration bridge

Bug Resolved Done Anil Vishnoi Anil Vishnoi Tue, 31 Mar 2015 12:20:29 +0000 Fri, 26 Jun 2015 11:09:20 +0000 Fri, 26 Jun 2015 11:09:20 +0000 unspecified Plugin 0 3 <p>The root cause of this problem is a race condition while setting up the openflow controller for bridge. When ovsdb node connects to the controller, it generate NODE event, that results in creating br-int and setting up the OF controller. Once bridge table is modified with br-int entry, it generate table update for Bridge table. Ovsdb Plugin receives this table update and it's default behavior is to set of controller for any new bridge, so when it gets the bridge table update, it again tries to set the openflow controller for the same bridge (br-int) and that causes the race condition, given that both the actions happens in two separate thread. Following are the two stack that try to set the OF controller for the bridge when ovsdb node connects to the controller first time.</p> <p>1) <br/> java.lang.Exception: Stack trace<br/> at java.lang.Thread.dumpStack(Thread.java:1365)<br/> at org.opendaylight.ovsdb.plugin.impl.ConfigurationServiceImpl.setOFController(ConfigurationServiceImpl.java:407)<br/> at org.opendaylight.ovsdb.openstack.netvirt.impl.BridgeConfigurationManagerImpl.addBridge(BridgeConfigurationManagerImpl.java:677)<br/> at org.opendaylight.ovsdb.openstack.netvirt.impl.BridgeConfigurationManagerImpl.createIntegrationBridge(BridgeConfigurationManagerImpl.java:376)<br/> at org.opendaylight.ovsdb.openstack.netvirt.impl.BridgeConfigurationManagerImpl.prepareNode(BridgeConfigurationManagerImpl.java:182)<br/> at org.opendaylight.ovsdb.openstack.netvirt.SouthboundHandler.processNodeUpdate(SouthboundHandler.java:128)<br/> at org.opendaylight.ovsdb.openstack.netvirt.SouthboundHandler.processEvent(SouthboundHandler.java:345)<br/> at org.opendaylight.ovsdb.openstack.netvirt.impl.EventDispatcherImpl.dispatchEvent(EventDispatcherImpl.java:92)<br/> at org.opendaylight.ovsdb.openstack.netvirt.impl.EventDispatcherImpl.access$100(EventDispatcherImpl.java:28)<br/> at org.opendaylight.ovsdb.openstack.netvirt.impl.EventDispatcherImpl$1.run(EventDispatcherImpl.java:55)<br/> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)<br/> at java.util.concurrent.FutureTask.run(FutureTask.java:262)<br/> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)<br/> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)<br/> at java.lang.Thread.run(Thread.java:745)</p> <p>2)<br/> java.lang.Exception: Stack trace<br/> at java.lang.Thread.dumpStack(Thread.java:1365)<br/> at org.opendaylight.ovsdb.plugin.impl.ConfigurationServiceImpl.setOFController(ConfigurationServiceImpl.java:407)<br/> at org.opendaylight.ovsdb.plugin.impl.InventoryServiceImpl$1.run(InventoryServiceImpl.java:184)<br/> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)<br/> at java.util.concurrent.FutureTask.run(FutureTask.java:262)<br/> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)<br/> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)<br/> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)<br/> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)<br/> at java.lang.Thread.run(Thread.java:745)</p> <p>Second stack is the one that set's the OF controller for any new bridge.</p> <p>Although setOFController check for the existing controller, but it checks the internal cache for that, and internal cache won't get updated till controller gets the update from ovsdb node for Controller table update. So this race condition happens in the duration between controller sent the "set controller for br-int" to ovsdb-node and it gets updated in internal cache.</p> <p>From Openstack integration perspective, i think we should not explicitly set controller for each bridge that we find on ovsdb node. I am referring to following code in InventoryServiceImpl.java </p> <p> private void handleOpenVSwitchSpecialCase(final Node node, final String databaseName, final String tableName, final UUID uuid) {<br/> if (OvsVswitchdSchemaConstants.shouldConfigureController(databaseName, tableName)) {<br/> Runnable updateControllerRunnable = new Runnable() {<br/> @Override<br/> public void run() {<br/> try </p> { if (ovsdbConfigurationService != null) ovsdbConfigurationService.setOFController(node, uuid.toString()); } <p> catch (InterruptedException | ExecutionException e) </p> { e.printStackTrace(); } <p> }<br/> };<br/> executor.execute(updateControllerRunnable);<br/> }<br/> }</p> <p>Above method is called from method processTableUpdates() in the same class</p> <p> if (update.getNew(uuid) != null) {<br/> boolean isNewRow = (tCache == null || tCache.get(uuid.toString()) == null) ? true : false;<br/> db.updateRow(databaseName, tableName, uuid.toString(), update.getNew(uuid));<br/> if (isNewRow) {<br/> this.handleOpenVSwitchSpecialCase(n, databaseName, tableName, uuid);<br/> if (!ovsdbInventoryListeners.isEmpty()) {<br/> for (OvsdbInventoryListener listener : ovsdbInventoryListeners) </p> { listener.rowAdded(n, tableName, uuid.toString(), update.getNew(uuid)); } <p> }</p> <p>Once i disable "setting OF controller for every new bridge" behavior, it fixes the issue.</p> <p>IMO assuming that compute node ovsdb instance will be clean (no bridges) before it connects to the controller is absolutely fine. After that ovsdb instance will only persist the configuration that is created by controller. This will give more controlled environment for controller to make sure that things work as expected.</p> <p>If any external project wants to enable the behavior of setting OF Controller for every new bridge, they can override the value in controller config file to enable that. Although it's good if that can be exposed through config sub system rather then a config file.</p> <p><a href="https://git.opendaylight.org/gerrit/17439" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/17439</a></p> <p>(In reply to Anil Vishnoi from comment #1)<br/> &gt; The root cause of this problem is a race condition while setting up the<br/> &gt; openflow controller for bridge. When ovsdb node connects to the controller,<br/> &gt; it generate NODE event, that results in creating br-int and setting up the<br/> &gt; OF controller. Once bridge table is modified with br-int entry, it generate<br/> &gt; table update for Bridge table. Ovsdb Plugin receives this table update and<br/> &gt; it's default behavior is to set of controller for any new bridge, so when it<br/> &gt; gets the bridge table update, it again tries to set the openflow controller<br/> &gt; for the same bridge (br-int) and that causes the race condition, given that<br/> &gt; both the actions happens in two separate thread. Following are the two stack<br/> &gt; that try to set the OF controller for the bridge when ovsdb node connects to<br/> &gt; the controller first time.<br/> &gt; <br/> &gt; 1) <br/> &gt; java.lang.Exception: Stack trace<br/> &gt; at java.lang.Thread.dumpStack(Thread.java:1365)<br/> &gt; at<br/> &gt; org.opendaylight.ovsdb.plugin.impl.ConfigurationServiceImpl.<br/> &gt; setOFController(ConfigurationServiceImpl.java:407)<br/> &gt; at<br/> &gt; org.opendaylight.ovsdb.openstack.netvirt.impl.BridgeConfigurationManagerImpl.<br/> &gt; addBridge(BridgeConfigurationManagerImpl.java:677)<br/> &gt; at<br/> &gt; org.opendaylight.ovsdb.openstack.netvirt.impl.BridgeConfigurationManagerImpl.<br/> &gt; createIntegrationBridge(BridgeConfigurationManagerImpl.java:376)<br/> &gt; at<br/> &gt; org.opendaylight.ovsdb.openstack.netvirt.impl.BridgeConfigurationManagerImpl.<br/> &gt; prepareNode(BridgeConfigurationManagerImpl.java:182)<br/> &gt; at<br/> &gt; org.opendaylight.ovsdb.openstack.netvirt.SouthboundHandler.<br/> &gt; processNodeUpdate(SouthboundHandler.java:128)<br/> &gt; at<br/> &gt; org.opendaylight.ovsdb.openstack.netvirt.SouthboundHandler.<br/> &gt; processEvent(SouthboundHandler.java:345)<br/> &gt; at<br/> &gt; org.opendaylight.ovsdb.openstack.netvirt.impl.EventDispatcherImpl.<br/> &gt; dispatchEvent(EventDispatcherImpl.java:92)<br/> &gt; at<br/> &gt; org.opendaylight.ovsdb.openstack.netvirt.impl.EventDispatcherImpl.<br/> &gt; access$100(EventDispatcherImpl.java:28)<br/> &gt; at<br/> &gt; org.opendaylight.ovsdb.openstack.netvirt.impl.EventDispatcherImpl$1.<br/> &gt; run(EventDispatcherImpl.java:55)<br/> &gt; at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)<br/> &gt; at java.util.concurrent.FutureTask.run(FutureTask.java:262)<br/> &gt; at<br/> &gt; java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:<br/> &gt; 1145)<br/> &gt; at<br/> &gt; java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:<br/> &gt; 615)<br/> &gt; at java.lang.Thread.run(Thread.java:745)<br/> &gt; <br/> &gt; 2)<br/> &gt; java.lang.Exception: Stack trace<br/> &gt; at java.lang.Thread.dumpStack(Thread.java:1365)<br/> &gt; at<br/> &gt; org.opendaylight.ovsdb.plugin.impl.ConfigurationServiceImpl.<br/> &gt; setOFController(ConfigurationServiceImpl.java:407)<br/> &gt; at<br/> &gt; org.opendaylight.ovsdb.plugin.impl.InventoryServiceImpl$1.<br/> &gt; run(InventoryServiceImpl.java:184)<br/> &gt; at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)<br/> &gt; at java.util.concurrent.FutureTask.run(FutureTask.java:262)<br/> &gt; at<br/> &gt; java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.<br/> &gt; access$201(ScheduledThreadPoolExecutor.java:178)<br/> &gt; at<br/> &gt; java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.<br/> &gt; run(ScheduledThreadPoolExecutor.java:292)<br/> &gt; at<br/> &gt; java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:<br/> &gt; 1145)<br/> &gt; at<br/> &gt; java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:<br/> &gt; 615)<br/> &gt; at java.lang.Thread.run(Thread.java:745)<br/> &gt; <br/> &gt; Second stack is the one that set's the OF controller for any new bridge.<br/> &gt; <br/> &gt; Although setOFController check for the existing controller, but it checks<br/> &gt; the internal cache for that, and internal cache won't get updated till<br/> &gt; controller gets the update from ovsdb node for Controller table update. So<br/> &gt; this race condition happens in the duration between controller sent the "set<br/> &gt; controller for br-int" to ovsdb-node and it gets updated in internal cache.<br/> &gt; <br/> &gt; From Openstack integration perspective, i think we should not explicitly set<br/> &gt; controller for each bridge that we find on ovsdb node. I am referring to<br/> &gt; following code in InventoryServiceImpl.java <br/> &gt; <br/> &gt; private void handleOpenVSwitchSpecialCase(final Node node, final String<br/> &gt; databaseName, final String tableName, final UUID uuid) {<br/> &gt; if<br/> &gt; (OvsVswitchdSchemaConstants.shouldConfigureController(databaseName,<br/> &gt; tableName)) {<br/> &gt; Runnable updateControllerRunnable = new Runnable() {<br/> &gt; @Override<br/> &gt; public void run() {<br/> &gt; try </p> { &gt; if (ovsdbConfigurationService != null) &gt; ovsdbConfigurationService.setOFController(node, uuid.toString()); &gt; } <p> catch (InterruptedException | ExecutionException e) </p> { &gt; e.printStackTrace(); &gt; } <p>&gt; }<br/> &gt; };<br/> &gt; executor.execute(updateControllerRunnable);<br/> &gt; }<br/> &gt; }<br/> &gt; <br/> &gt; Above method is called from method processTableUpdates() in the same class<br/> &gt; <br/> &gt; if (update.getNew(uuid) != null) {<br/> &gt; boolean isNewRow = (tCache == null ||<br/> &gt; tCache.get(uuid.toString()) == null) ? true : false;<br/> &gt; db.updateRow(databaseName, tableName, uuid.toString(),<br/> &gt; update.getNew(uuid));<br/> &gt; if (isNewRow) {<br/> &gt; this.handleOpenVSwitchSpecialCase(n, databaseName,<br/> &gt; tableName, uuid);<br/> &gt; if (!ovsdbInventoryListeners.isEmpty()) {<br/> &gt; for (OvsdbInventoryListener listener :<br/> &gt; ovsdbInventoryListeners) </p> { &gt; listener.rowAdded(n, tableName, uuid.toString(), &gt; update.getNew(uuid)); &gt; } <p>&gt; }<br/> &gt; <br/> &gt; Once i disable "setting OF controller for every new bridge" behavior, it<br/> &gt; fixes the issue.<br/> &gt; <br/> &gt; IMO assuming that compute node ovsdb instance will be clean (no bridges)<br/> &gt; before it connects to the controller is absolutely fine. After that ovsdb<br/> &gt; instance will only persist the configuration that is created by controller.<br/> &gt; This will give more controlled environment for controller to make sure that<br/> &gt; things work as expected.</p> <p>Very good catch, Anil! I don't think we can assume there are no bridges in<br/> ovs when we connect it to the manager. In some start-up scripts, openstack<br/> explicitly creates br-int before connecting to ODL. Also, br-ex is only<br/> created by openstack. Any ideas on how we could avoid the race w/out losing<br/> the ability to set controller in cases when bridge is already created?</p> Component Version Development External issue ID 2925 External issue URL Issue Type Priority Rank 0|i020x3: