https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/OVSDB-439 ovsdb <p>Hello Anil,<br/>  <br/> Looks like we hit the same issue in our local testing. <br/> With ODL Carbon (+ Pike, OVS2.7), during one reboot scenario, we observed some race condition in ODL/OVSDB. <br/> Can you please let us know if this issue is a known-issue/addressed in OVSDB?<br/>  <br/> Steps to Reproduce (in a working setup with a controller and two compute nodes):<br/> 1. Restart the compute node and wait for the compute node to come up.<br/> 2. Launch an instance on the compute node<br/> 3. You can observe that the instance initially stays in "spawning" state and then transitions to "error" state.<br/> 4. Restart the openvswitch on the compute node<br/> 5. Launch a new instance and it would boot successfully.<br/>  <br/> Basically, when we issue the reboot on the compute node, ODL identifies that the node is idle and triggers the disconnection chain. <br/> But, while this is going on, when the Compute node comes up, we could see that there is a race condition between the cleanup events and the events related to the node reconciliation.<br/>  <br/> In this process, we could see that finally the Compute node is deleted from the operational store <span class="error">&#91;#&#93;</span> eventhough its connected to the controller. <br/> Since the node info is deleted from the datastore, the side effect is that port-binding fails and we are unable to spawn new VMs until we restart the OVS Switch on the Compute node.<br/> Following<span class="error">&#91;@&#93;</span> is a SNAP of the karaf logs which show this sequence.<br/>  <br/> Additional notes:<br/> In case, the compute node comes up with some delay (i.e., after the cleanup is properly done in ODL) this issue (i.e., step3 above) is not seen.<br/>  <br/> <span class="error">&#91;#&#93;</span> 2017-08-01 07:48:16,660 | INFO  | lt-dispatcher-49 | OvsdbConnectionManager           | 289 - org.opendaylight.ovsdb.southbound-impl - 1.4.1.Carbon-redhat-1 | Entity{type='ovsdb', id=/(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)network-topology/topology/topology[</p> {(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)topology-id=ovsdb:1} <p>]/node/node<span class="error">&#91;\{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)node-id=ovsdb://uuid/e9806896-8dc2-4f17-83ea-c1c957608915}&#93;</span>} has no owner, cleaning up the operational data store<br/> <span class="error">&#91;@&#93;</span> <a href="https://gist.github.com/sridhargaddam/3761ef080e11f2dd2429c8d7016ae6d0" class="external-link" target="_blank" rel="nofollow noopener">https://gist.github.com/sridhargaddam/3761ef080e11f2dd2429c8d7016ae6d0</a></p> OVSDB-439

Stale connection check is failing and end up removing node from data store.

Bug High Resolved Done suneelu varma Sam Hague csit:3node ds Fri, 15 Dec 2017 13:49:17 +0000 Mon, 9 Jul 2018 18:39:43 +0000 Mon, 9 Jul 2018 18:39:43 +0000 Carbon-SR3 Oxygen-SR3 Fluorine Southbound.Open_vSwitch 0 3 <p>This issue can be recreated with just OVSDB southbound plugin by using the following steps.</p> <p> </p> <p>(1) Connect the ovs (set-manager) to the controller.</p> <p>(2) Add following iptables rule, that will disconnect the communication to the controller </p> <p>sudo iptables -A OUTPUT -d &lt;CONTROLLER_IP&gt; -j DROP<br/> sudo iptables -A INPUT -s &lt;CONTROLLER_IP&gt; -j DROP</p> <p> </p> <p>(3) Sleep for 90 seconds</p> <p>(4) Remove the roles and enable the communication to the controller</p> <p>sudo iptables -D OUTPUT -d &lt;CONTROLLER_IP&gt; -j DROP<br/> sudo iptables -D INPUT -s &lt;CONTROLLER_IP&gt; -j DROP</p> <p> </p> <p>This should trigger the second connection from the switch to the controller, but the previous connection is still lingering in the controller. Current code has a bug that ends up disconnecting the second connection as well.</p> <p> </p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jluhrsen" class="user-hover" rel="jluhrsen">jluhrsen</a> this is a good CSIT test to check the connection flips between OVS and controller.</p> <p>stable/carbon : <a href="https://git.opendaylight.org/gerrit/66718" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/66718</a></p> <p>I can't reproduce this. I have a CSIT <a href="https://git.opendaylight.org/gerrit/c/66718/" class="external-link" target="_blank" rel="nofollow noopener">patch</a> in the works, but I want to make sure the CSIT<br/> can actually hit the bug. Locally, I'm doing this:</p> <ul> <li>set manager</li> <li>iptables block 6640 incoming</li> <li>wait until I see the connection is not there anymore (ovs-vsctl show AND netstat)</li> <li>unblock 6640</li> </ul> <p>at this point I see the connection is made again and seems to stay. I must be missing<br/> some step.</p> <p>BTW,I know ovs will continue to try and sometimes the ovs-vsctl show will say<br/> is_connected: True briefly, as I think it does that as soon as the initial TCP 3whs<br/> succeeds. But, even so if it's rejected by ovsdb-lib then we'll see ovs trying to<br/> reconnect again. Because of this, I'm trying to write the test to actually make sure<br/> there is no long term ESTABLISHED connection on port 6640 showing up using the<br/> same source tcp port number.</p> <p>Basically this issue will recreate only if there are parallel connection to the controller from the same switch. To put controller you will have to drop the traffic both ways (controller to switch and vice versa), that's the reason i was installing two rules </p> <p>sudo iptables -A OUTPUT -d &lt;CONTROLLER_IP&gt; -j DROP<br/> sudo iptables -A INPUT -s &lt;CONTROLLER_IP&gt; -j DROP</p> <p> </p> <p>You don't have to wait for connection to be gone, just wait for 60-90 seconds before removing the rules and it will get recreated consistently.</p> <p>this patch is still unmerged. what's the plan here?</p> <p>marking this done, as the patch went in back in December for carbon. I think we just didn't<br/> close this jira at that point.</p> Relates OVSDB-443 OVSDB-462 OVSDB-433 OVSDB-438 OVSDB-443 Development Epic Link NETVIRT-996 Rank 0|i0391r: