https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/OPNFLWPLUG-217 OpenFlowPlugin <p>Following step in the Wiki <a href="https://wiki.opendaylight.org/view/CrossProject:Integration_Group:Performance_Test" class="external-link" target="_blank" rel="nofollow noopener">https://wiki.opendaylight.org/view/CrossProject:Integration_Group:Performance_Test</a></p> <p>With "dropAllPacketsRPC on" there is a noticeable performance degradation. </p> <p>Ever 30 minutes cbench is ran (cbench -c $controllerip -p 6633 -m 10000 -l 10 -s 256 -M 100000 -w 1).</p> <p>(Thu Jul 17 07:53:43 PDT 2014)30 minutes after starting controller min/max/avg is 21434.08/22523.66/21841.65<br/> (Thu Jul 17 11:53:43 PDT 2014) 4:30hr after starting controller min/max/avg is 19243.97/20008.8/19744.69<br/> (Thu Jul 17 19:53:43 PDT 2014) 12:30hr after starting controller min/max/avg is 14169.36/14786.94/14444.8<br/> (Fri Jul 18 07:23:43 PDT 2014) 24:00hr after starting controller min/max/avg is 10534.49/10943.67/10767.37 </p> <p>See CSV file for detail report. <br/> and PNG file for the visual report.</p> <p>Operating System: Linux<br/> Platform: PC</p> OPNFLWPLUG-217

24hr performance degradation

Bug Resolved Won't Do Marcus G K Williams Christopher O'Shea Fri, 18 Jul 2014 16:23:25 +0000 Mon, 27 Sep 2021 09:01:15 +0000 Wed, 17 Aug 2016 09:27:23 +0000 General 0 15 <p>Attachment OpenFlow performance.png has been added with description: Visual plot of performance.</p> <p>Attachment results-18-6-2014.csv has been added with description: CSV break down of memory and CPU stats</p> <p>Here's another example of this observed performance degradation<span class="error">&#91;1&#93;</span>.</p> <p>Note that (perhaps obviously) restarting ODL between CBench runs mitigates this decrease<span class="error">&#91;2&#93;</span>.</p> <p><span class="error">&#91;1&#93;</span>: <a href="https://github.com/dfarrell07/cbench_regression/issues/13#issuecomment-47366340" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/dfarrell07/cbench_regression/issues/13#issuecomment-47366340</a><br/> <span class="error">&#91;2&#93;</span>: <a href="https://github.com/dfarrell07/cbench_regression/issues/13#issuecomment-47538456" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/dfarrell07/cbench_regression/issues/13#issuecomment-47538456</a></p> <p>Switch to using Karaf controller RC0 </p> <ol> <li> <ol> <li> <ol> <li> <ol> <li>Installing the needed bundles<br/> feature:install odl-flow-services<br/> feature:install odl-openflowplugin-drop-test</li> </ol> </li> </ol> </li> </ol> </li> </ol> <ol> <li> <ol> <li> <ol> <li>Turning on droptest<br/> drop-test:dropallpacketsrpc on</li> </ol> </li> </ol> </li> </ol> <p>First set of test run fine.<br/> wait 30 minutes start the next test run and the first are ok but the last few return 0.</p> <p>I had the controller running for 24hour, but wasn't running Cbench ever 30 minutes. </p> <p>Look like the first run after getting stabled was around 38k, didnn't change when i ran it again 24hr later. </p> <p>Will setting to run ever 30 minutes tonight and update ticket tomorrow.</p> <p>I'm still seeing perf degradation over time in Helium 0.2.1. Lots of details on this WCBench wiki page<span class="error">&#91;1&#93;</span>.</p> <p><span class="error">&#91;1&#93;</span>: <a href="https://github.com/dfarrell07/wcbench/wiki/Helium-0.2.1-Performance-Results#results-0" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/dfarrell07/wcbench/wiki/Helium-0.2.1-Performance-Results#results-0</a></p> <p>The perf results wiki I linked to previously has changed addresses. The updated URLs (including much more detailed results) are:</p> <p><a href="https://github.com/dfarrell07/wcbench/wiki/Unusual-Helium-0.2.1-Performance-Results" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/dfarrell07/wcbench/wiki/Unusual-Helium-0.2.1-Performance-Results</a><br/> <a href="https://github.com/dfarrell07/wcbench/wiki/Experiment:-Varying-NUM_SWITCHES" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/dfarrell07/wcbench/wiki/Experiment:-Varying-NUM_SWITCHES</a></p> <p>Reproducing this issue. I'll work on a fix after reproduction.</p> <p>Graph of <a href="https://jira.opendaylight.org/browse/OPNFLWPLUG-217" title="24hr performance degradation" class="issue-link" data-issue-key="OPNFLWPLUG-217"><del>OPNFLWPLUG-217</del></a> reproduction showing cbench stats and ram usage.</p> <p>Attachment OF_1_Bug1395.PNG has been added with description: Graph of CBench Results showing <a href="https://jira.opendaylight.org/browse/OPNFLWPLUG-217" title="24hr performance degradation" class="issue-link" data-issue-key="OPNFLWPLUG-217"><del>OPNFLWPLUG-217</del></a></p> <p>I've reproduced this many times. I've attached a graph of the results I'm seeing on Helium SR1, though I've also reproduce on latest integration nightly build.</p> <p>Currently java profiling ODL during bug reproduction in an attempt to narrow likely causes and focus on correct area of the code. Initial data based on profiling show ever increasing nioEventLoopGroup threads after each cbench run. More analysis is needed before this can be definitively named as the root cause. Source of nio thread generation is under investigation and currently being traced back to code.</p> <p>I've retested this with the Lithium release and the issue is still there.<br/> with 57 iterations, the average performance value went from 20k to under<br/> 5k. Attached is a spreadsheet with this info.</p> <p>Attachment 1395_Lithium_data.ods has been added with description: re-test with Lithium</p> <p>Submitted a patch for review <a href="https://git.opendaylight.org/gerrit/#/c/27867/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/27867/</a></p> <p>Attachment 1395.odc has been added with description: degradation chart with potential fix</p> <p>(In reply to Jamo Luhrsen from comment #13)<br/> &gt; Created attachment 656 <span class="error">&#91;details&#93;</span><br/> &gt; degradation chart with potential fix</p> <p>The attached chart is the data I obtained when testing with Sania's fix.</p> <p>&gt; The attached chart is the data I obtained when testing with Sania's fix.</p> <p>Am I reading this correctly - the summery is that perf is still degrading in a similar way to the initial reports?</p> <p>(In reply to Daniel Farrell from comment #15)<br/> &gt; &gt; The attached chart is the data I obtained when testing with Sania's fix.<br/> &gt; <br/> &gt; Am I reading this correctly - the summery is that perf is still degrading in<br/> &gt; a similar way to the initial reports?</p> <p>that's correct.</p> <p>I have tested Sania's patch and its working fine with no degradation.</p> <p>Attachment 1395_fix.csv has been added with description: 1395fix</p> <p>(In reply to Muhammad Umar Hameed from comment #17)<br/> &gt; Created attachment 672 <span class="error">&#91;details&#93;</span><br/> &gt; 1395fix<br/> &gt; <br/> &gt; I have tested Sania's patch and its working fine with no degradation.</p> <p>Your standard deviation is quite high, which could be masking a trend.</p> <p>Still, interesting result, thanks.</p> <p>Graph attached.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=%2Fwcbench" class="user-hover" rel="/wcbench">/wcbench</a>$ ./stats.py -s flows 12:12:53<br/> {'flows': </p> {'max': 62104, 'mean': 44968.913, 'min': 26658, 'relstddev': 15.511, 'stddev': 6975.297} <p>,<br/> 'sample_size': 500}</p> <p>Attachment Graph of 1395fix CSV file flows.png has been added with description: Graph of 1395fix CSV file created using WCBench</p> <p>Anyone else tested with the same patch?</p> <p>(In reply to Muhammad Umar Hameed from comment #17)<br/> &gt; Created attachment 672 <span class="error">&#91;details&#93;</span><br/> &gt; 1395fix<br/> &gt; <br/> &gt; I have tested Sania's patch and its working fine with no degradation.</p> <p>Jamo also verified the fix and the patch has now been merged.</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/27867/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/27867/</a></p> <p>We may be able to close the longest-standing known perf bug in ODL. <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Fixed by Sania's patch: <a href="https://git.opendaylight.org/gerrit/#/c/27867/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/27867/</a></p> <p>Should we reopen this bug since it was reverted for stable/lithium in the patch below?</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/30085" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/30085</a></p> <p>I just reopened it.</p> <p>We employed black-box type analysis to locate the source of this performance issue with a minimum understanding of ODL code. So the solution we provided doesn't work well and I can now see why. <br/> Anyway, I want to share our findings regarding the source of the problem.<br/> Wcbench connects a new set of switches after each run, disconnecting the previous set and this can be verified using Wireshark or even netstat. <br/> CPU profiling of ODL shows that a poll method being invoked from QueueKeeperHarvester gets called an increasing number of times after each run. According to my understanding, this method polls a collection of queuekeeper objects that contain queues with OF messages (data+control) coming from the switches. Now if we check the size of the said collection while Wcbnch is running, it keeps on increasing after every run and the increase is equal to number of switches. Number of calls to the poll method increase as the size of the collection increases linearly. This was verified by keeping the same set of switches connected to ODL for a long running test; this time the performance dint degrade. So the problem is that when a switch disconnects, queuekeeper object corresponding to that switch is not closed/destroyed/removed-from-the-collection. And since a new set of switches is connected after each run, the size of aforementioned collection keeps on increasing. What (I think) I did wrong, in the patch I submitted, was that I prematurely closed the queuekeeper. <br/> It would be great if someone with a better understanding of ODL code could comment on this.</p> <p>Is there a plan for how to incorporate the fact that we know the root cause of the bug into an actual fix? Also, could we please explain here why the fix was reverted?</p> <p>(In reply to Colin Dixon from comment #26)<br/> &gt; Is there a plan for how to incorporate the fact that we know the root cause<br/> &gt; of the bug into an actual fix? Also, could we please explain here why the<br/> &gt; fix was reverted?</p> <p>once this fix was given, a new bug was uncovered:<br/> <a href="https://bugs.opendaylight.org/show_bug.cgi?id=4649" class="external-link" target="_blank" rel="nofollow noopener">https://bugs.opendaylight.org/show_bug.cgi?id=4649</a></p> <p>it was decided to fix 4649 above by reverting the patch that fixed this 1395 <br/> bug.</p> <p>Closing this bug. Actually this bug was on performance degradation in He design. In Boron/Carbon we don't use it anymore. </p> <p>If any project or someone need to fix this problem for Beryllium SR4 Helium design should raise a new bug. But I doubt there is time for such thing, we got a lot of work with new releases <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> Development External issue ID 1395 External issue URL ODL SR Target Milestone Rank 0|i02zsf: