https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/OVSDB-288 ovsdb <p>scenario:</p> <p>controller initiates the connection to passive ovs instance (e.g. ptcp)</p> <p>instance is connected and configurations can be made</p> <p>restart the ovs instance (reboot entire system, stop/start ovsdb-server/ovs-vswitchd)</p> <p>result:</p> <p>ovs instance is no longer connected or existing in operational store. ovsdb<br/> plugin does not seem to retry connection.</p> <p>Operating System: All<br/> Platform: All</p> OVSDB-288

passive connection not reconnected if ovs service is restarted

Bug Resolved Done Anil Vishnoi Jamo Luhrsen Wed, 3 Feb 2016 21:53:38 +0000 Wed, 6 Apr 2016 21:49:58 +0000 Wed, 6 Apr 2016 21:49:58 +0000 unspecified Southbound.Open_vSwitch 0 3 <p>Although we can add the retry logic, but there is no deterministic approach about when and how long we should retry. There is a possibility that the switch is totally gone and it's not going to come back up at all. </p> <p>But i agree that we should have some retry logic for a reasonable time to atleast try and see if switch is back on, probably do 10 retry where first try will be 10 second after disconnection and then we increment the delay by 10 seconds for each next try e.g (1st try at = 10S, 2nd = 20S 3rd=30S 4th=40S.... 10th=100S ), that will give around 9 minutes window to user to reboot/reconfigure the system.</p> <p>(In reply to Anil Vishnoi from comment #1)<br/> &gt; Although we can add the retry logic, but there is no deterministic approach<br/> &gt; about when and how long we should retry. There is a possibility that the<br/> &gt; switch is totally gone and it's not going to come back up at all. <br/> &gt; <br/> &gt; But i agree that we should have some retry logic for a reasonable time to<br/> &gt; atleast try and see if switch is back on, probably do 10 retry where first<br/> &gt; try will be 10 second after disconnection and then we increment the delay by<br/> &gt; 10 seconds for each next try e.g (1st try at = 10S, 2nd = 20S 3rd=30S<br/> &gt; 4th=40S.... 10th=100S ), that will give around 9 minutes window to user to<br/> &gt; reboot/reconfigure the system.</p> <p>ofp has retry mechanisms right? why not use something similar. Not sure<br/> I think we should ever give up though. maybe max out at 60s and then keep<br/> going once every 60s. Thinking about scheduled outages, etc.</p> <p>(In reply to Jamo Luhrsen from comment #2)<br/> &gt; (In reply to Anil Vishnoi from comment #1)<br/> &gt; &gt; Although we can add the retry logic, but there is no deterministic approach<br/> &gt; &gt; about when and how long we should retry. There is a possibility that the<br/> &gt; &gt; switch is totally gone and it's not going to come back up at all. <br/> &gt; &gt; <br/> &gt; &gt; But i agree that we should have some retry logic for a reasonable time to<br/> &gt; &gt; atleast try and see if switch is back on, probably do 10 retry where first<br/> &gt; &gt; try will be 10 second after disconnection and then we increment the delay by<br/> &gt; &gt; 10 seconds for each next try e.g (1st try at = 10S, 2nd = 20S 3rd=30S<br/> &gt; &gt; 4th=40S.... 10th=100S ), that will give around 9 minutes window to user to<br/> &gt; &gt; reboot/reconfigure the system.<br/> &gt; <br/> &gt; ofp has retry mechanisms right? why not use something similar. Not sure<br/> &gt; I think we should ever give up though. maybe max out at 60s and then keep<br/> &gt; going once every 60s. Thinking about scheduled outages, etc.</p> <p>OFP has no retry logic, because OFP there is only active connections (switch to controller). Keep re-trying is a bad idea, there is possibility that servers/compute node will never join back, in that case you will keep retrying it forever. If this is a planned outage, application need to remove that configuration from the data store and add it back once server/ovsdb is back.</p> <p>Another thought, OVS and OVSDB by default retry connections. It is in the other direction rather than the request here, but maybe since that is supported we don't really need the functionality requested in the bug as a high priority.</p> <p>(In reply to Anil Vishnoi from comment #3)<br/> &gt; (In reply to Jamo Luhrsen from comment #2)<br/> &gt; &gt; (In reply to Anil Vishnoi from comment #1)<br/> &gt; &gt; &gt; Although we can add the retry logic, but there is no deterministic approach<br/> &gt; &gt; &gt; about when and how long we should retry. There is a possibility that the<br/> &gt; &gt; &gt; switch is totally gone and it's not going to come back up at all. <br/> &gt; &gt; &gt; <br/> &gt; &gt; &gt; But i agree that we should have some retry logic for a reasonable time to<br/> &gt; &gt; &gt; atleast try and see if switch is back on, probably do 10 retry where first<br/> &gt; &gt; &gt; try will be 10 second after disconnection and then we increment the delay by<br/> &gt; &gt; &gt; 10 seconds for each next try e.g (1st try at = 10S, 2nd = 20S 3rd=30S<br/> &gt; &gt; &gt; 4th=40S.... 10th=100S ), that will give around 9 minutes window to user to<br/> &gt; &gt; &gt; reboot/reconfigure the system.<br/> &gt; &gt; <br/> &gt; &gt; ofp has retry mechanisms right? why not use something similar. Not sure<br/> &gt; &gt; I think we should ever give up though. maybe max out at 60s and then keep<br/> &gt; &gt; going once every 60s. Thinking about scheduled outages, etc.<br/> &gt; <br/> &gt; OFP has no retry logic, because OFP there is only active connections (switch<br/> &gt; to controller). Keep re-trying is a bad idea, there is possibility that<br/> &gt; servers/compute node will never join back, in that case you will keep<br/> &gt; retrying it forever. If this is a planned outage, application need to remove<br/> &gt; that configuration from the data store and add it back once server/ovsdb is<br/> &gt; back.</p> <p>ok, you are right about retrying indefinitely.</p> <p>so this bug is being attributed to CSIT failures for not quite the right<br/> reason. Maybe this should be, as Sam says, lowered to a simple enhancement<br/> request. Some short term re-try mechanism in the case that our ovs node<br/> services go missing for a period.</p> <p>The more serious failure that CSIT is failing on goes like this<br/> (seems like a new bug to track, but please advise on that)</p> <p>A<br/> ovs in passive mode</p> <p>B<br/> initiate connection from controller</p> <p>C<br/> verify it exists in config and operational</p> <p>D<br/> ovs-ctl stop, then start on ovs node (simulates node going away for a brief period)</p> <p>E<br/> ovs configured back in passive mode</p> <p>F<br/> verify operational does NOT see node (because we don't retry)</p> <p>G<br/> verify still in config (we never deleted it)</p> <p>H<br/> delete from config (starting suggested steps to recover)</p> <p>I<br/> verify not in operational and config (as expected)</p> <p>J<br/> initiate connection from controller fails now.</p> <p>log messages of interest, I think, coming from step E:</p> <p>2016-02-11 01:03:28,759 | INFO | entLoopGroup-8-1 | OvsdbConnectionService | 153 - org.opendaylight.ovsdb.library - 1.2.1.SNAPSHOT | Connection closed ConnectionInfo <span class="error">&#91;Remote-address=209.132.179.50, Remote-port=6634, Local-address172.18.182.19, Local-port=36453, type=ACTIVE&#93;</span><br/> 2016-02-11 01:03:28,760 | INFO | entLoopGroup-8-1 | OvsdbConnectionManager | 159 - org.opendaylight.ovsdb.southbound-impl - 1.2.1.SNAPSHOT | Library disconnected ACTIVE from /209.132.179.50:6634 to /172.18.182.19:36453. Cleaning up the operational data store<br/> 2016-02-11 01:03:28,783 | INFO | lt-dispatcher-21 | OvsdbConnectionManager | 159 - org.opendaylight.ovsdb.southbound-impl - 1.2.1.SNAPSHOT | Entity{type='ovsdb', id=/(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)network-topology/topology/topology[</p> {(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)topology-id=ovsdb:1} <p>]/node/node[</p> {(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)node-id=ovsdb://209.132.179.50:6634} <p>]} has no owner, cleaning up the operational data store</p> <p>log messages of interest, I think, coming from step J:</p> <p>2016-02-11 01:03:32,432 | INFO | lt-dispatcher-21 | OvsdbConnectionManager | 159 - org.opendaylight.ovsdb.southbound-impl - 1.2.1.SNAPSHOT | Disconnecting from 209.132.179.50:6634<br/> 2016-02-11 01:03:32,621 | WARN | ult-dispatcher-4 | OvsdbDataChangeListener | 159 - org.opendaylight.ovsdb.southbound-impl - 1.2.1.SNAPSHOT | Connection to device ConnectionInfo{getRemoteIp=IpAddress [_ipv4Address=Ipv4Address <span class="error">&#91;_value=209.132.179.50&#93;</span>, _value=<span class="error">&#91;2, 0, 9, ., 1, 3, 2, ., 1, 7, 9, ., 5, 0&#93;</span>], getRemotePort=PortNumber <span class="error">&#91;_value=6634&#93;</span>, augmentations={}} already exists. Plugin does not allow multiple connections to same device, hence dropping the request OvsdbNodeAugmentation{getConnectionInfo=ConnectionInfo{getRemoteIp=IpAddress [_ipv4Address=Ipv4Address <span class="error">&#91;_value=209.132.179.50&#93;</span>, _value=<span class="error">&#91;2, 0, 9, ., 1, 3, 2, ., 1, 7, 9, ., 5, 0&#93;</span>], getRemotePort=PortNumber <span class="error">&#91;_value=6634&#93;</span>, augmentations={}}}</p> <p>(In reply to Sam Hague from comment #4)<br/> &gt; Another thought, OVS and OVSDB by default retry connections. It is in the<br/> &gt; other direction rather than the request here, but maybe since that is<br/> &gt; supported we don't really need the functionality requested in the bug as a<br/> &gt; high priority.</p> <p>Yes, but this is something that works for active connection only, and not for passive connection and for passive connection it's kind of enhancement work.</p> <p>(In reply to Anil Vishnoi from comment #7)<br/> &gt; (In reply to Sam Hague from comment #4)<br/> &gt; &gt; Another thought, OVS and OVSDB by default retry connections. It is in the<br/> &gt; &gt; other direction rather than the request here, but maybe since that is<br/> &gt; &gt; supported we don't really need the functionality requested in the bug as a<br/> &gt; &gt; high priority.<br/> &gt; <br/> &gt; Yes, but this is something that works for active connection only, and not<br/> &gt; for passive connection and for passive connection it's kind of enhancement<br/> &gt; work.</p> <p>Agreed, that is what I meant by "other direction" - it is the active connection rather than passive.</p> <p>stable/beryllium : <a href="https://git.opendaylight.org/gerrit/36028" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/36028</a></p> <p>There are two issues discussed in the issue </p> <p>(1) No retry for controller initiated connection if connected gets dropped.</p> <p>(2) Sometime when switch abruptly goes away (machine down etc), controller is not able to establish connection to switch and there is no operational data in the data store.</p> <p>Above patch fixes issue (1). This patch added a reconciliation mechanism, where if controller initiated connection get dropped (connection flapping, machine crash), it will immediately attempt to connect back and after that it will make 10 attempts to connect to switch with the incremental time interval (10,20,30,40....100). Overall it will wait for 9 minutes for switch to come back up. After that it will give up. If there are usecase where we need to wait for longer time, I am open for suggestion.</p> <p>We still recommend that if it's planned outage, user application should explicitly disconnect the switch. </p> <p>Issue (2) is happening because tcp connection is not reset/terminated properly from controller side, because switch went down without sending TCP_FIN packet. Fix for this issue is proposed in following patch <a href="https://git.opendaylight.org/gerrit/#/c/35436/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/35436/</a></p> <p>patch stable/beryllium : <a href="https://git.opendaylight.org/gerrit/#/c/36028/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/36028/</a></p> <p>wiki : <a href="https://wiki.opendaylight.org/view/OVSDB_Integration:OVSDB_SB_Reconciliation" class="external-link" target="_blank" rel="nofollow noopener">https://wiki.opendaylight.org/view/OVSDB_Integration:OVSDB_SB_Reconciliation</a></p> <p>As outlined in comment 6, this bug is fixed. I have verified in stable/beryllium distro built on 04/02/2016.</p> <p>moving to Resolved/Fixed state.</p> Development External issue ID 5221 External issue URL Issue Type ODL SR Target Milestone Rank 0|i021t3: