https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/OPNFLWPLUG-630 OpenFlowPlugin <p>While performing scalability test against OpenFlowPlugin codebase, stable/lithium branch, -li plugin, with statistics collection enable, I encounter the following issue:</p> <ul class="alternate" type="square"> <li>OvS 2.3.x or previous version:<br/> Scalability seems fine, or at least goes up to 400+ switches.</li> </ul> <ul class="alternate" type="square"> <li>OvS 2.4.x or newer version:<br/> Scalability is capped around 45-50 switches. <br/> At the tipping point, switches close the connection then send hello message. As ODL is in bad shape at this time, ODL goes in a crazy loop where it tries to reestablish the connection for all switches, one by one, but failed. This was ending in OOM error.<br/> This crazy loop behaviour was recently fixed with Bug-4957; now, at the tipping point, ODL goes crazy for a bit then recovers and stabilizes, although all switches are disconnected and no more connection is possible.<br/> Here are some logs, and a Yourkit Java Profiler snapshot: <a href="https://www.dropbox.com/sh/1zz1x6i1bl5uor8/AACHJfML-RqvOk7vFI5U-haJa?dl=0" class="external-link" target="_blank" rel="nofollow noopener">https://www.dropbox.com/sh/1zz1x6i1bl5uor8/AACHJfML-RqvOk7vFI5U-haJa?dl=0</a></li> </ul> <p>I will setup tests in ODL infra to track progress, and to know better about scalability performance. Work started here: <a href="https://git.opendaylight.org/gerrit/#/c/35813/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/35813/</a></p> <p>Operating System: All<br/> Platform: All</p> OPNFLWPLUG-630

Li plugin: Scalability issues with OVS 2.4

Bug Resolved Done Alexis de Talhouët Alexis de Talhouët Fri, 4 Mar 2016 15:32:18 +0000 Mon, 27 Sep 2021 09:01:44 +0000 Thu, 30 Jun 2016 13:25:17 +0000 General 0 4 <p>Alexis please can you explain what exactly means "ODL goes crazy a bit", can you point me for the exact part of the log ? Thank you.</p> <p>Jozef</p> <p>Jozef, please read this mail I send a while now: <a href="https://lists.opendaylight.org/pipermail/openflowplugin-dev/2016-February/004631.html" class="external-link" target="_blank" rel="nofollow noopener">https://lists.opendaylight.org/pipermail/openflowplugin-dev/2016-February/004631.html</a></p> <p>I know the behavior has changed a bit since a lot of code came in to stabilize transaction/role/etc... </p> <p>Also it pretty easy to reproduce, I have create a patch in intergration test with all the Dockerfile, scripts I've been using to create the scale testbed.</p> <p>I shall reproduce it with current codebase by the end of the week if you want to.</p> <p>Thanks,<br/> Alexis</p> <p>This bug is actually very easy to reproduce with OVS 2.4:</p> <p>1) Download latest ODL Be from Nexus<br/> 2) Start He or Li plugin (I think issue is on both)<br/> 3) Start mininet tree,4 (this is not even a large topology)<br/> 4) CPU goes crazy but there is also memory leak</p> <p>&gt;OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000007b5c00000, 95944704, 0) failed; error='Cannot allocate memory' (errno=12)<br/> #</p> <ol> <li>There is insufficient memory for the Java Runtime Environment to continue.</li> <li>Native memory allocation (malloc) failed to allocate 95944704 bytes for committing reserved memory.</li> <li>An error report file with more information is saved as:</li> <li>/home/mininet/controller-master/distribution-karaf-0.4.1-SNAPSHOT/hs_err_pid18041.log</li> </ol> <p>So I analyzed what is going on:</p> <p>OvS has a inactivity probe timeout that by default is 5 second <span class="error">&#91;0&#93;</span>. If in the period of 2 * inactivity probe (10 second by default) the switch doesn't hear from the controller (ECHO_REQUEST, or whatever message in fact), the switch will send a disconnected request. <span class="error">&#91;1&#93;</span><br/> So under a certain load, ODL is not able to process all switches in this given timeout <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/star_yellow.png" height="16" width="16" align="absmiddle" alt="" border="0"/>, thus receive disconnected event followed by hello message from switches.</p> <p><img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/star_yellow.png" height="16" width="16" align="absmiddle" alt="" border="0"/> wondering if this has something to do with the HashedWheelTimer from the DeviceManagerImpl class: <a href="https://github.com/opendaylight/openflowplugin/blob/stable/lithium/openflowplugin-impl/src/main/java/org/opendaylight/openflowplugin/impl/device/DeviceManagerImpl.java#l81" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/opendaylight/openflowplugin/blob/stable/lithium/openflowplugin-impl/src/main/java/org/opendaylight/openflowplugin/impl/device/DeviceManagerImpl.java#l81</a></p> <p><span class="error">&#91;0&#93;</span><br/> Logic to initialize inactivity timer value for a specific bridge from ovsdb content.<br/> <a href="https://github.com/openvswitch/ovs/blob/master/vswitchd/bridge.c#L3415" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/openvswitch/ovs/blob/master/vswitchd/bridge.c#L3415</a></p> <p><span class="error">&#91;1&#93;</span><br/> Logic to handle no response from inactivity probe<br/> <a href="https://github.com/openvswitch/ovs/blob/master/lib/rconn.c#L559-L560" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/openvswitch/ovs/blob/master/lib/rconn.c#L559-L560</a></p> <p>Updating bug title to reflect issue only happens in Li plugin.</p> <p>We were talking wint Michal Rehak and there is really a huge amount of data and huge load on DS with table features. With this possibility to switch off table features was running locally on my mininet all smoothly with 127 switches. </p> <p>I would appreciate when someone will test it properly with this patch:</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/36506/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/36506/</a></p> <p>This is anyway a workaround, but we can provide it into SR1. </p> <p>If I am not mistaken, everyone can still use RPC for table feature request.</p> <p>Thanks</p> <p>Jozef</p> <p>Jozeph,</p> <p>I confirmed this increase the scalability of the OFP-lithium. In fact this is the workaround I did for my app.<br/> One question though, would it make sense to make all the features configurable through config subsystem? <br/> I sent a mail couple of weeks ago proposing to do that, but got few reply: <a href="https://lists.opendaylight.org/pipermail/openflowplugin-dev/2016-March/004769.html" class="external-link" target="_blank" rel="nofollow noopener">https://lists.opendaylight.org/pipermail/openflowplugin-dev/2016-March/004769.html</a></p> <p>Would you mind amending this patch so one can define what features he wants? This would be great! Else I can do it, if you agree it makes sense.</p> <p>Thanks,<br/> Alexis</p> <p>Hi Alexis, </p> <p>I don't think to have "all" features configurable it such a good idea, as you have more possibility something to set or change then you get more wrong settings and more problems and question about how to properly set. </p> <p>I would be wise to let configurable features to hold on minimum.</p> <p>Jozef</p> <p>Hi all,<br/> this is more like fix then workaround: <br/> <a href="https://git.opendaylight.org/gerrit/#/c/36559" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/36559</a></p> <p>Here table-features got moved outside table in order not to put additional heavy load on statistic updates.</p> <p>Trade off is that downstream apps would need to adapt their instance-identifiers to table features (I checked NIC and DIDM - both are reading table-features).</p> <p>Positive by this fix is that no exclusive flag in config is needed and projects like NIC/DIDM can still coexist beside other - "table-features free" projects.</p> <p>So please test this change. Thank you.</p> <p>I like this solution better than the config workaround. One of the reasons is we do not want to ask users to set special configuration to run perf/scale tests, perf/scale should be good out-of-the-box.</p> <p>BR/Luis</p> <p>Hi all,</p> <p>I've found out that the patch for the <a href="https://jira.opendaylight.org/browse/OPNFLWPLUG-630" title="Li plugin: Scalability issues with OVS 2.4" class="issue-link" data-issue-key="OPNFLWPLUG-630"><del>OPNFLWPLUG-630</del></a> has been already merged.<br/> <a href="https://git.opendaylight.org/gerrit/#/c/36559/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/36559/</a><br/> That patch changes the model.</p> <p>In the result, it has broken NIC project's build.<br/> The build on the master branch of the NIC project contentiously fails since May 13.<br/> <a href="https://jenkins.opendaylight.org/releng/view/nic/job/nic-integration-boron/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/nic/job/nic-integration-boron/</a></p> <p>I think we should notify downstream projects of any model changes before we merge the patch for model changes.</p> <p>Hideyuki we were talking about it a lot and second its a long time that nobody from NIC has noticed that NIC is broken <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/wink.png" height="16" width="16" align="absmiddle" alt="" border="0"/> </p> <p>Here is link </p> <p><a href="https://lists.opendaylight.org/pipermail/openflowplugin-dev/2016-May/005123.html" class="external-link" target="_blank" rel="nofollow noopener">https://lists.opendaylight.org/pipermail/openflowplugin-dev/2016-May/005123.html</a></p> <p>where we were talked about it, </p> <p>if we wait to long when anyone give us feedback we would still be waiting for someone. </p> <p>PEACE <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Jozef</p> <p>P.S.: Sorry to cause you problem, but I really though we all agreed to merge the changes onto master.</p> <p>(In reply to Jozef Bacigal from comment #12)<br/> &gt; Hideyuki we were talking about it a lot and second its a long time that<br/> &gt; nobody from NIC has noticed that NIC is broken <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/wink.png" height="16" width="16" align="absmiddle" alt="" border="0"/> <br/> &gt; <br/> &gt; Here is link <br/> &gt; <br/> &gt; <a href="https://lists.opendaylight.org/pipermail/openflowplugin-dev/2016-May/005123" class="external-link" target="_blank" rel="nofollow noopener">https://lists.opendaylight.org/pipermail/openflowplugin-dev/2016-May/005123</a>.<br/> &gt; html<br/> &gt; <br/> &gt; where we were talked about it, <br/> &gt; <br/> &gt; if we wait to long when anyone give us feedback we would still be waiting<br/> &gt; for someone. <br/> &gt; <br/> &gt; PEACE <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/><br/> &gt; <br/> &gt; Jozef<br/> &gt; <br/> &gt; P.S.: Sorry to cause you problem, but I really though we all agreed to merge<br/> &gt; the changes onto master.</p> <p>Yeah, it's quite weird that nobody in the NIC project was aware of that build failure for a long time <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>And, I'm sorry I was not aware you guys talked about it a lot before the merge.</p> <p>Thanks!</p> Development External issue ID 5464 External issue URL Issue Type ODL SR Target Milestone Rank 0|i032c7: