https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/OPNFLWPLUG-727 OpenFlowPlugin <p>Both plugins (He + Li) show the following:</p> <p>cbench -c 10.29.8.203 -t -m 12000 -M 10000 -s 8 -l 10</p> <p>cbench: controller benchmarking tool<br/> running in mode 'throughput'<br/> connecting to controller at 10.29.8.203:6633 <br/> faking 8 switches offset 1 :: 10 tests each; 12000 ms per test<br/> with 10000 unique source MACs per switch<br/> learning destination mac addresses before the test<br/> starting test with 0 ms delay after features_reply<br/> ignoring first 1 "warmup" and last 0 "cooldown" loops<br/> connection delay of 0ms per 1 switch(es)<br/> debugging info is off<br/> controller msgbuf_read() = -1: msgbuf_read: Connection reset by peer<br/> ... exiting</p> <p>My interpretation is that controller closes the switches connections and therefore we get following report:</p> <p>05:52:56.797 8 switches: flows/sec: 0 0 0 0 0 0 0 0 total = 0.000000 per ms <br/> 05:53:08.897 8 switches: flows/sec: 0 0 0 0 0 0 20000 0 total = 1.666627 per ms <br/> 05:53:20.998 8 switches: flows/sec: 0 0 0 0 0 0 0 0 total = 0.000000 per ms <br/> 05:53:33.098 8 switches: flows/sec: 0 0 0 0 0 0 0 0 total = 0.000000 per ms <br/> 05:53:45.222 8 switches: flows/sec: 0 0 0 0 0 776 0 0 total = 0.064542 per ms <br/> 05:53:57.323 8 switches: flows/sec: 0 0 0 0 0 0 0 0 total = 0.000000 per ms <br/> 05:54:09.423 8 switches: flows/sec: 0 0 0 0 0 0 0 0 total = 0.000000 per ms</p> <p>Operating System: All<br/> Platform: All</p> OPNFLWPLUG-727

cbench throughput test does not work in Boron

Bug Resolved Done Tomas Slusny Luis Gomez Fri, 8 Jul 2016 21:13:07 +0000 Mon, 27 Sep 2021 09:01:51 +0000 Mon, 21 Nov 2016 21:22:08 +0000 General 0 7 <p>Already solved.</p> <p>Reopening, issue shows again and it is tracked here:</p> <p><a href="https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-cbench-performance-only-boron/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-cbench-performance-only-boron/</a></p> <p>BR/Luis</p> <p>I easily reporduced locally:</p> <p>mininet@mininet-vm:~\&gt; cbench -c 192.168.0.1 -t -m 12000 -M 10000 -s 8 -l 10<br/> cbench: controller benchmarking tool<br/> running in mode 'throughput'<br/> connecting to controller at 192.168.0.1:6633 <br/> faking 8 switches offset 1 :: 10 tests each; 12000 ms per test<br/> with 10000 unique source MACs per switch<br/> learning destination mac addresses before the test<br/> starting test with 0 ms delay after features_reply<br/> ignoring first 1 "warmup" and last 0 "cooldown" loops<br/> connection delay of 0ms per 1 switch(es)<br/> debugging info is off<br/> 14:04:25.711 8 switches: flows/sec: 11006 0 0 16669 0 0 0 0 total = 2.306181 per ms <br/> 14:04:37.815 8 switches: flows/sec: 0 0 0 3331 0 0 0 11903 total = 1.269233 per ms <br/> controller msgbuf_read() = -1: msgbuf_read: Connection reset by peer<br/> ... exiting</p> <p>Not only the above but when the test stops the controller stays with 100% CPU with no switch connected so there is really an issue here.</p> <p>To install cbench on ubuntu just look at the cbench section of this script:</p> <p><a href="https://git.opendaylight.org/gerrit/gitweb?p=releng/builder.git;a=blob;f=packer/provision/mininet.sh" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/gitweb?p=releng/builder.git;a=blob;f=packer/provision/mininet.sh</a></p> <p>Rising to critical as people in the field will try cbench for sure.</p> <p>Hi Luis,</p> <p>Tested on master.</p> <p>We have found some blocked threads, because of logging.</p> <p>I have tried to turn logging off, but anyway the best I've got locally is this:</p> <p>cbench: controller benchmarking tool<br/> running in mode 'throughput'<br/> connecting to controller at localhost:6633 <br/> faking 8 switches offset 1 :: 9 tests each; 12000 ms per test<br/> with 10000 unique source MACs per switch<br/> learning destination mac addresses before the test<br/> starting test with 1000 ms delay after features_reply<br/> ignoring first 1 "warmup" and last 0 "cooldown" loops<br/> connection delay of 0ms per 1 switch(es)<br/> debugging info is off<br/> 13:43:16.118 8 switches: flows/sec: 0 19999 17883 19165 0 0 0 0 total = 4.751830 per ms <br/> 13:43:28.220 8 switches: flows/sec: 14254 0 0 0 0 3916 0 14849 total = 2.751527 per ms <br/> 13:43:40.320 8 switches: flows/sec: 0 0 0 0 7962 0 0 0 total = 0.663450 per ms <br/> 13:43:52.427 8 switches: flows/sec: 0 0 0 0 0 19999 20000 0 total = 3.331661 per ms <br/> 13:44:19.436 8 switches: flows/sec: 0 0 0 0 12038 0 0 0 total = 0.447354 per ms <br/> 13:44:31.544 8 switches: flows/sec: 0 0 0 0 0 0 0 0 total = 0.000000 per ms <br/> 13:44:43.645 8 switches: flows/sec: 20000 51877 56720 29757 0 60000 0 60000 total = 23.195637 per ms <br/> 13:44:55.746 8 switches: flows/sec: 78687 56798 71107 60000 0 55125 0 94532 total = 34.687287 per ms <br/> 13:45:08.235 8 switches: flows/sec: 45838 80000 53229 60000 0 74013 0 65008 total = 30.518821 per ms <br/> RESULT: 8 switches 8 tests min/max/avg/stdev = 0.00/34687.29/11949.47/13917.78 responses/s</p> <p>There could be some issue with that, but it will be probably not the only one.</p> <p>CPU is running also for me at 100% after cbench end. I am not sure about connected switches. How do you check it? </p> <p>That is what we have for now. </p> <p>Thanks,<br/> Miro</p> <p>With more testing I could fix the test by adding some start delay (-D 10000) and reducing the number of MACs (-M 100):</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/44773/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/44773/</a></p> <p>However even with these changes the controller stays with 100% CPU after the test which I think we should fix as this could be a security vulnerability.</p> <p>After connecting a profiler, this thread seems to be the issue:</p> <p>AbstractStackedOutboundQueue.java:333 org.opendaylight.openflowjava.protocol.impl.core.connection.StackedSegment.failAll(OutboundQueueException)</p> <p>Hi Luis, <br/> I think, there is still problem with excessive logging. I prepared the patch where log level is set to debug and also precondition isDebugEnabled is used. I ran cbench locally on this patch (+logging in karaf set to ERROR) and my CPU after test and karaf shutdown get to normal (2%) and didn't get stuck at 100%. Could you please test it in jenkins enviroment?</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/44868/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/44868/</a></p> <p>(In reply to Andrej Leitner from comment #8)<br/> &gt; Hi Luis, <br/> &gt; I think, there is still problem with excessive logging. I prepared the patch<br/> &gt; where log level is set to debug and also precondition isDebugEnabled is<br/> &gt; used. I ran cbench locally on this patch (+logging in karaf set to ERROR)<br/> &gt; and my CPU after test and karaf shutdown get to normal (2%) and didn't get<br/> &gt; stuck at 100%. Could you please test it in jenkins enviroment?<br/> &gt; <br/> &gt; <a href="https://git.opendaylight.org/gerrit/#/c/44868/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/44868/</a></p> <p>Logging the failures as error should be ok, since those are failures. any trace should be removed and lets have the debug logs checked before logging.I will also take a look at the places we log and come out with an improved logging</p> <p>Hi Shuva, <br/> I think that despite the fact that unsuccessful RPC result brings errors from device there is no reason for throwing error since it is not error/failure of karaf or OFP itself. It's "only" problem of device and we can go on. I think (can't find it on mailing-lists now) we should throw ERROR only if there is a real problem and we are not able to recover (e.g. restart needed). Hence log level for this shouldn't be more than info. I chose debug level because information about result is useful/targeted for admin and not common user (like everyone, there is still operational/DS where you see if flow got installed or not)</p> <p>Upgrading to blocker due to impact in ODL perf reports.</p> <p>(In reply to Luis Gomez from comment #11)<br/> &gt; Upgrading to blocker due to impact in ODL perf reports.</p> <p>Luis do we still see controller closing the switch connections ? or is it that the controller cpu stays at 100% even after the test ?</p> <p>I am currently testing the proposed patch, I will update shortly on it.</p> <p>This patch may improve some perf number but it does not help with:</p> <p>1) Test abort due to cbench switches disconnect issue: This is mostly addressed in this patch: <a href="https://git.opendaylight.org/gerrit/#/c/44773/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/44773/</a></p> <p>2) CPU very high after test finishes and switches are disconnected. This is to me the blocker now as controller seems to be inoperable after the test.</p> <p>BR/Luis</p> <p>(In reply to Luis Gomez from comment #14)<br/> &gt; This patch may improve some perf number but it does not help with:<br/> &gt; <br/> &gt; 1) Test abort due to cbench switches disconnect issue: This is mostly<br/> &gt; addressed in this patch: <a href="https://git.opendaylight.org/gerrit/#/c/44773/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/44773/</a><br/> &gt; <br/> &gt; 2) CPU very high after test finishes and switches are disconnected. This is<br/> &gt; to me the blocker now as controller seems to be inoperable after the test.<br/> &gt; <br/> &gt; BR/Luis</p> <p>Does the CPU usage come down after sometime or it remains same eternally ?</p> <p>For as long as my patient allows which is some minutes after the test the CPU is still 100%.</p> <p>FYI <a href="https://git.opendaylight.org/gerrit/#/c/44773/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/44773/</a> is already merged so test should go to green but CPU issue is still there and I will be probably extending the test to catch this scenario</p> <p>Is there an ETA for this bug and someone assigned to fix?</p> <p>FYI, I added check to ver</p> <p>FYI, I added extra check in cbench test to track this bug:</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/45046/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/45046/</a></p> <p>This job will fail until this issue gets fixed:</p> <p><a href="https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-cbench-performance-only-boron/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-cbench-performance-only-boron/</a></p> <p>BR/Luis</p> <p>To better assess the impact of this bug and fix, could someone from your team please help us identify the following:<br/> Severity: Could you elaborate on the severity of this bug? Is this a BLOCKER such that we cannot release Boron without it? Is there a workaround such that we can write a release note and fix in future Boron SR1?<br/> Testing: Could you also elaborate on the testing of this patch? How extensively has this patch been tested? Is it covered by any unit tests or system tests? <br/> Impact: Does this fix impact any dependent projects?</p> <p>Severity<br/> The bug is blocker because of performance regression -&gt;<br/> (In reply to Luis Gomez from comment #11)<br/> &gt; Upgrading to blocker due to impact in ODL perf reports.</p> <p>Testing<br/> The patch for this bug in ofplugin consists of decreasing log level to prevent overloading of netty thread with logging errors (seen in troughput test) and ConcurrentModificationException at higher load (ibid.). There was no need to unit test update, we ran patch-test for ofplugin more times to check impact.</p> <p>There is also an issue in ofjava described in <a href="https://jira.opendaylight.org/browse/OPNFLWJAVA-79" title="Infinite reschedule of flush" class="issue-link" data-issue-key="OPNFLWJAVA-79"><del>OPNFLWJAVA-79</del></a>. </p> <p>Impact<br/> In principle, the fix does not impact any dependent projects.</p> <p>merged in boron</p> <p>This is fixed now according to:</p> <p><a href="https://jenkins.opendaylight.org/releng/job/openflowplugin-csit-1node-cbench-performance-only-boron/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/job/openflowplugin-csit-1node-cbench-performance-only-boron/</a></p> <p>BR/Luis</p> <p>Reopening the bug: the CPU issue is fixed but after running throughput test couple of times (cbench -c 192.168.0.1 -t -m 12000 -M 100 -l 10 -s 16 -D 5000) I see memory issues:</p> <p>OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000cdc00000, 121634816, 0) failed; error='Cannot allocate memory' (errno=12)<br/> #</p> <ol> <li>There is insufficient memory for the Java Runtime Environment to continue.</li> <li>Native memory allocation (mmap) failed to map 121634816 bytes for committing reserved memory.</li> <li>An error report file with more information is saved as:</li> <li>/home/vagrant/controller-test/distribution-karaf-0.5.0-SNAPSHOT/hs_err_pid10272.log<br/> Killed</li> </ol> <p>After increasing the physical RAM size, I do not see issues running the test locally. Still some unstability in CI so downgrading to Major until we understand why Beryllium does not show this unstability.</p> <p>(In reply to Luis Gomez from comment #24)<br/> &gt; This is fixed now according to:<br/> &gt; <br/> &gt; <a href="https://jenkins.opendaylight.org/releng/job/openflowplugin-csit-1node-cbench-" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/job/openflowplugin-csit-1node-cbench-</a><br/> &gt; performance-only-boron/<br/> &gt; <br/> &gt; BR/Luis</p> <p>Based on Luis's comment, can I safely assume that this bug been verified as fixed in the latest Boron RC 3.1 Build?</p> <p>This bug needs to remain open with lower priority in case we get some questions after Boron release. The reason is I had to modify the test to sleep 10 secs between Cbench runs in order to stabilize it:</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/45294/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/45294/</a></p> <p>In Beryllium we did not have to do this.</p> <p>BR/Luis</p> <p>(In reply to Luis Gomez from comment #25)<br/> &gt; Reopening the bug: the CPU issue is fixed but after running throughput test<br/> &gt; couple of times (cbench -c 192.168.0.1 -t -m 12000 -M 100 -l 10 -s 16 -D<br/> &gt; 5000) I see memory issues:<br/> &gt; <br/> &gt; OpenJDK 64-Bit Server VM warning: INFO:<br/> &gt; os::commit_memory(0x00000000cdc00000, 121634816, 0) failed; error='Cannot<br/> &gt; allocate memory' (errno=12)<br/> &gt; #<br/> &gt; # There is insufficient memory for the Java Runtime Environment to continue.<br/> &gt; # Native memory allocation (mmap) failed to map 121634816 bytes for<br/> &gt; committing reserved memory.<br/> &gt; # An error report file with more information is saved as:<br/> &gt; #<br/> &gt; /home/vagrant/controller-test/distribution-karaf-0.5.0-SNAPSHOT/<br/> &gt; hs_err_pid10272.log<br/> &gt; Killed</p> <p><a href="https://jira.opendaylight.org/browse/OPNFLWJAVA-78" title="Failed entries not freed from queue, causing superfluous iterations" class="issue-link" data-issue-key="OPNFLWJAVA-78"><del>OPNFLWJAVA-78</del></a> in openflowjava could be related to memory issues. <a href="https://jira.opendaylight.org/browse/OPNFLWJAVA-79" title="Infinite reschedule of flush" class="issue-link" data-issue-key="OPNFLWJAVA-79"><del>OPNFLWJAVA-79</del></a> merged as dependency for our cbench blocker allows to fail unflushed segments, however failed entries in segment are not counted into completed (<a href="https://jira.opendaylight.org/browse/OPNFLWJAVA-78" title="Failed entries not freed from queue, causing superfluous iterations" class="issue-link" data-issue-key="OPNFLWJAVA-78"><del>OPNFLWJAVA-78</del></a>) and segment is not considered as completed thus not removed. The patch is already merged in carbon, tested by Luis as per comments: <a href="https://git.opendaylight.org/gerrit/#/c/45210/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/45210/</a> and got +2 from Michal. Not sure if it still possible to merge it for RC3.</p> <p>According to CBench unstability - this is bug with CBench.</p> <p>When we think that switch is IDLE, we send HELLO message to switch, and we are expecting that switch will also reply with HELLO message. But CBench was not sending this HELLO message and silently ignored all incoming HELLO messages.</p> <p>I added proper HELLO reply when HELLO is received to CBench sources. Here is my fork of CBench repo with this fix: <a href="https://github.com/deathbeam/oflops" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/deathbeam/oflops</a>.</p> <p>I also created pull request to official repository, but since it is pretty inactive, I doubt that it will be ever merged.</p> <p>After some more testing, we are actually sending ECHO and on HELLO on switch IDLE, and CBench is actually trying to send ECHO_REPLY, but I think for some reason we are not receiving this ECHO_REPLY in time. I will investigate this a bit more.</p> <p>OK there is a line on the sand of 11:59p UTC on sunday, if you find something and can get some patch by that fine, otherwise it will have to wait until SR1.</p> <p>we are getting 100% pass on jenkins from Sep 8</p> <p>Luis, could we close the bug as resolved?</p> <p>Yes, this is fixed now.</p> <p>latency rerun is failing occasionally</p> <p>According to jenkins: <a href="https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-cbench-performance-only-carbon/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-cbench-performance-only-carbon/</a> and <a href="https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-cbench-performance-only-boron/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-cbench-performance-only-boron/</a> cbench runs are now pretty stable, only with occasional fails (but there failed all 3 tests, so it is probably environment issue) so closing this, again.</p> <p>Right, this can be closed for now.</p> Blocks OPNFLWJAVA-79 Development External issue ID 6176 External issue URL Rank 0|i032xr: