https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/OPNFLWPLUG-1049 OpenFlowPlugin <p>I'm running ODL fluorine stable release (opendaylight-0.9.0.tar.gz) and facing this problem:</p> <p>when a simulated switch (mininet) connects and disconnects very frequently, the openflowplugin enters in a loop and cannot recover the handshaking properly. This also affects the connection with some other switches.</p> <p>I can reproduce the same issue in a network of real switches (Extreme/Edgecore switches).</p> <p>When this problem happens, there is also a huge memory leak resulting in having the number of DeviceContextImpl instances growing indefinitely.</p> <p>Steps to reproduce:</p> <p>1) run opendaylight-0.9.0</p> <p>2) feature:install features-openflowplugin</p> <p>3) run mininet (sudo mn --topo linear,20 --switch ovsk,protocols=OpenFlow13 --mac --controller remote,port=6633,ip=127.0.0.1)</p> <p>4) simulate a switch disconnection by running the command "./changectrl.sh 10000 0.1" (script in attachments)</p> <p>5) wait 1-2 minutes, you should see odl trying indefinitely to regain the connection</p> <p>6) stop the script, the memory leak is now growing (you can check the number of instances of DeviceContextImpl by running "jcmd &lt;pid&gt; GC.class_histogram | grep -e "org.opendaylight.openflowplugin.impl.device.DeviceContextImpl$")</p> <p> </p> <p>In attachment also the karaf.log</p> OPNFLWPLUG-1049

Switch handshaking loops indefinitely

Bug High Resolved Done Somashekhar Javalagi Leonardo Milleri Thu, 15 Nov 2018 10:17:16 +0000 Mon, 27 Sep 2021 09:00:58 +0000 Thu, 14 Nov 2019 09:54:31 +0000 Nitrogen-SR1 Fluorine openflowplugin-impl 0 4 <p>FYI I downloaded latest fluorine controller from:<br/> <a href="https://nexus.opendaylight.org/content/repositories/opendaylight.snapshot/org/opendaylight/integration/karaf/0.9.2-SNAPSHOT" class="external-link" target="_blank" rel="nofollow noopener">https://nexus.opendaylight.org/content/repositories/opendaylight.snapshot/org/opendaylight/integration/karaf/0.9.2-SNAPSHOT</a></p> <p>and installed feature: odl-openflowplugin-flow-services-rest</p> <p>Then I started mininet and run attached script on a VM with OVS 2.8.1.</p> <p>After the above I am not able to reproduce, maybe the CPU plays a role here. I am using a 4 CPU VM for controller + 2 CPU VM for mininet.</p> <p>I've just tried with the latest fluorine controller (0.9.2-SNAPSHOT) and I can still reproduce the problem.</p> <p>The first time was ok, but then I tried two more times and in both cases the problem was reproduced.</p> <p>To be honest, it is better to wait till the last moment, when the changectrl scripts ends.</p> <p>I'm running mininet (Open vSwitch 2.9.0) in the same machine of the ODL controller (dell laptop precision 3530)</p> <p> </p> <p>OK, I changed the test setup to run in my laptop and now I see couple of issues:</p> <p>1) Switches different from the flapping one (s1) start to disconnect &amp; reconnect after the default inactivity probe (3 sec) kicks in. This aggravates the issue but it can be easily suppressed by setting OVS inactivity probe to some value bigger than controller default inactivity probe (15 sec). You can use following script after mininet is up to set inactivity probe:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> #!/bin/bash x=`sudo ovs-vsctl --columns=_uuid list Controller | awk <span class="code-quote">'{print $NF}'</span>` echo $x <span class="code-keyword">for</span> i in $x <span class="code-keyword">do</span> sudo ovs-vsctl set Controller $i inactivity_probe=20 done </pre> </div></div> <p>2) After applying the above and letting the flap run for a while I see that after stopping the flap, the switch s1 cannot connect anymore to controller. This is something we have to fix as for now the only workaround is to restart the controller.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=leonardo.milleri" class="user-hover" rel="leonardo.milleri">leonardo.milleri</a>, can you try with the inactivity probe configuration I posted and check if you see the same (switch s1 cannot connect anymore to controller) or any other faulty behavior?</p> <p>Thank you Luis.</p> <p>Today for some reasons I can't reproduce the problem with inactivity_probe=3.</p> <p>As far as concern inactivity_probe=20, this is also working perfectly for me, eventually all the switches are connected (including s1).</p> <p>Can you please provide some more details about how the inactivity probe can affect the openflow connection? This can help on reproducing the problem as well.</p> <p>I'll carry on doing some others tests and let you know</p> <p> </p> <p> </p> <p> </p> <p>Adding some more information: I've just seen the problem in a real network of edgecore switches running pica8 NOS, opendaylight version is nitrogen-SR1.</p> <p>The following log fragment is related to the issue:</p> <p>2018-11-28 05:10:11,251 | WARN | tLoopGroup-16-14 | ClusterSingletonServiceGroupImpl | 397 - org.opendaylight.mdsal.singleton-dom-impl - 2.3.1 | Service group openflow:2465031308174919169 stopping unregistered service org.opendaylight.openflowplugin.impl.lifecycle.ContextChainImpl@111fb98b<br/> 2018-11-28 05:10:11,262 | ERROR | pool-60-thread-1 | OutboundQueueProviderImpl | 436 - org.opendaylight.openflowplugin.impl - 0.5.1 | No queue present, failing request<br/> 2018-11-28 05:10:11,262 | WARN | pool-60-thread-1 | RpcContextImpl | 436 - org.opendaylight.openflowplugin.impl - 0.5.1 | Xid cannot be reserved for new RequestContext, node:openflow:2465031308174919169</p> <p>If you are running OpenFlow in a real network the first thing to do is the set the switches inactivity_probe or equivalent timer in switch (e.g. time the switch waits before sending echo request to controller when it does not receive any packet from controller) to more than 15 sec. The reason is when controller is busy it can miss to respond to the switch echo request and that causes switch disconnect and further problems when switch reconnect again.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=leonardo.milleri" class="user-hover" rel="leonardo.milleri">leonardo.milleri</a>, The script in the attached file has only connection command for the switch, should it be modified to trigger disconnect as well.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=Somashekhar" class="user-hover" rel="Somashekhar">Somashekhar</a>, if you open the script, the line that says:<br/> sudo ovs-vsctl set-controller s1 tcp:128.0.0.1:6633<br/> should normally disconnect the switch as 128.0.0.1 is not a known address.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=ecelgp" class="user-hover" rel="ecelgp">ecelgp</a>, do you know if anyone is working on this issue in ODL and how long would it take to fix it?</p> <p>Anything can I help with?</p> <p>In the meantime, I'll try to understand the implications of the workaround (inactivity_probe) and if it'll sort out any effects.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=leonardo.milleri" class="user-hover" rel="leonardo.milleri">leonardo.milleri</a> Can you please try testing the scenario with the gerrit review to check if issue still comes?</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=Somashekhar" class="user-hover" rel="Somashekhar">Somashekhar</a>, FYI I quickly tried your patch distribution:<br/> <a href="https://nexus.opendaylight.org/content/repositories/opendaylight.snapshot/org/opendaylight/integration/openflowplugin/karaf/0.9.2-SNAPSHOT/karaf-0.9.2-20181210.143812-12.zip" class="external-link" target="_blank" rel="nofollow noopener">https://nexus.opendaylight.org/content/repositories/opendaylight.snapshot/org/opendaylight/integration/openflowplugin/karaf/0.9.2-SNAPSHOT/karaf-0.9.2-20181210.143812-12.zip</a></p> <p>And I still see after running the flap script for a while the switch s1 (openflow:1) cannot connect anymore to controller.</p> <p>I tried the fix on top of the stable-fluorine branch and I could be able to reproduce the problem when running the script the third time (for the first 2 attempts I couldn't reproduce it).</p> <p>Instead, it seems the master branch is not having the same problem (run the script 5 times), or at least it is more robust. Are you aware of any commits in the master branch actually fixing/mitigating the issue?</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=leonardo.milleri" class="user-hover" rel="leonardo.milleri">leonardo.milleri</a> we are planning to introduce device connection hold time, which is the minimum amount of time switch has to wait until it gets connected again, to reduce load on controller.</p> <p>We will record the switch last connected time. If switch is connected again within the hold time, then the connection will not be accepted.</p> <p>Thank you, I'll import the attached changes and retest</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=ecelgp" class="user-hover" rel="ecelgp">ecelgp</a> If you get a chance can you please test the latest patch for this issue?</p> <p>Discussed dampening mechanism for 3-node cluster setup. Having a local dampening mechanism (connection dampening in the context of single node ) and global dampening mechiansm (connection dampening across the three node cluster) would be a great value add.</p> <p>Hi <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=ecelgp" class="user-hover" rel="ecelgp">ecelgp</a></p> <p>For this issue, we are expecting csit to pass to proceed further. I am seeing some of openflowplugin sodium csit jobs failing consistently only with reason ConnectionError: HTTPConnectionPool(host='10.30.170.90', port=8181). Are these seen before?</p> <p>I just ran csit on dummy test review and below are logs.</p> <p><a href="https://jenkins.opendaylight.org/releng/job/openflowplugin-patch-test-core-sodium/83/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/job/openflowplugin-patch-test-core-sodium/83/</a></p> <p> </p> <p>This is due to this regression:</p> <p><a href="https://jira.opendaylight.org/browse/CONTROLLER-1906" class="external-link" rel="nofollow">https://jira.opendaylight.org/browse/CONTROLLER-1906</a></p> <p>Merged in master branch, which is magnesium</p> Development Rank 0|i03knj: