https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/OPNFLWPLUG-322 OpenFlowPlugin <p>In some of my testing I've discovered an external message and code path that will cause ODL to fail. If left unfixed, I am concerned that this could be used for a Denial of Service attack</p> <p>Internally, we us 'monit' to ensure all core processes are up. The following line was added last month to our monit configuration:<br/> if failed host 127.0.0.1 with port 6653 type TCP for 3 times within 3 cycles then restart</p> <p>This line causes monit to connect to ODL (via the loop-back interface) and then disconnect. After this line was added, we discovered that our ODL installations would fail after a few days of running. </p> <p>I opened <a href="https://bugs.opendaylight.org/show_bug.cgi?id=2394" class="external-link" target="_blank" rel="nofollow noopener">https://bugs.opendaylight.org/show_bug.cgi?id=2394</a> to track a specific problem associated with this (OFHandshake threads building up).</p> <p>When an external entity opens an OpenFlow connection, but does not complete the handshake, I see log messages along the lines of:</p> <p>2014-11-24 15:17:42.200 UTC <span class="error">&#91;OFHandshake-7-0&#93;</span> WARN o.o.o.o.m.c.ErrorHandlerSimpleImpl - exception -&gt; FIRST HELLO sending failed because of connection issue., session -&gt; null<br/> org.opendaylight.openflowplugin.ConnectionException: FIRST HELLO sending failed because of connection issue.<br/> at org.opendaylight.openflowplugin.openflow.md.core.HandshakeManagerImpl.sendHelloMessage(HandshakeManagerImpl.java:297) ~<span class="error">&#91;bundlefile:na&#93;</span><br/> at org.opendaylight.openflowplugin.openflow.md.core.HandshakeManagerImpl.shake(HandshakeManagerImpl.java:95) ~<span class="error">&#91;bundlefile:na&#93;</span><br/> at org.opendaylight.openflowplugin.openflow.md.core.HandshakeStepWrapper.run(HandshakeStepWrapper.java:47) <span class="error">&#91;bundlefile:na&#93;</span><br/> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) <span class="error">&#91;na:1.7.0_71&#93;</span><br/> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) <span class="error">&#91;na:1.7.0_71&#93;</span><br/> at java.lang.Thread.run(Thread.java:745) <span class="error">&#91;na:1.7.0_71&#93;</span></p> <p>Resources are not freed and ODL eventually locks up.</p> <p>I've patched by code with my patch for 2394, but I am concerned that there are other code paths that I haven't investigated.</p> <p>Operating System: All<br/> Platform: All</p> OPNFLWPLUG-322

Need to close the ODL Denial of Service interface

Bug Resolved Done Michal Rehak Jim West Mon, 24 Nov 2014 15:48:26 +0000 Mon, 27 Sep 2021 09:01:23 +0000 Tue, 13 Oct 2015 21:15:17 +0000 General 0 8 <p>Jim,</p> <p>Do you have a patch for this? In case not then we will need to assign this bug to someone.</p> <p>Thanks,<br/> Abhijit</p> <p>Hi Abhijit</p> <p>I have a patch associated with 2394. With this patch, I don't see the problem any longer. I'm not sure that the patch for 2394 covers all the code patches (that's the reason I opened this bug)</p> <p>(In reply to Abhijit Kumbhare from comment #1)<br/> &gt; Jim,<br/> &gt; <br/> &gt; Do you have a patch for this? In case not then we will need to assign this<br/> &gt; bug to someone.<br/> &gt; <br/> &gt; Thanks,<br/> &gt; Abhijit</p> <p>Can we lower this from critical to major or normal?</p> <p>This needs retest - as handshake thread model was changed in Li codebase &amp; this is to do with close and cleanup after handshake failure.</p> <p>What's the reasoning for lowering the Importance? Is it because we think it's fixed?</p> <p>(In reply to Abhijit Kumbhare from comment #3)<br/> &gt; Can we lower this from critical to major or normal?</p> <p>(In reply to Jim West from comment #5)<br/> &gt; What's the reasoning for lowering the Importance? Is it because we think<br/> &gt; it's fixed?<br/> &gt; <br/> &gt; (In reply to Abhijit Kumbhare from comment #3)<br/> &gt; &gt; Can we lower this from critical to major or normal?</p> <p>Reason: you no longer see the issue with your patch - and the developers may need to concentrate on the other higher priority bugs at this moment (during the bug fix cycle). Any case - we need a retest of this (see my previous comment) - can you please retest?</p> <p>This is an easy issue to reproduce with the "nc" linux tool.</p> <p>steps:<br/> 1. while 1&gt;0; do nc -w 1 172.17.7.22 6633<br/> 2. get coffee</p> <p>with a profiler (e.g. jvisual) you can watch the thread count rise, or this linux CLI will work too:</p> <p>watch -n1 ps -o nlwp $ODL_PID </p> <p>I've recreated the thread count growth on the Helium release bits, as well as using this distro:<br/> distribution-karaf-0.3.0-20150506.171255-1469.zip</p> <p>around aprox 30k working threads, there may be some trouble with OOM or file descriptors. In every case, the controller proves to be useless.</p> <p>We should not reduce the priority of this.</p> <p>This issue is <b>NOT</b> seen in Helium SR3, but is seen in all Helium releases prior to it. I have manually checked this on Helium, Helium-SR1 and Helium-SR2.</p> <p>Yes, this issue was fixed in helium SR3, so this is expected. It means this is regression from the master branch.</p> <p>This patch should fix problem in Li codebase<br/> <a href="https://git.opendaylight.org/gerrit/#/c/20097/1" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/20097/1</a></p> <p>Please retest ans eventually mark resolved.</p> <p>This is resolved, and automation is there to show this. Search the <br/> flow-services robot suites for the test case with title "Bug Validation.2429"</p> <p>the test created for this bug start failing for lithium redesigned openflow plugin </p> <p>in stable/lithium <br/> <a href="https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-cds-flow-services-lithium-redesign-only-stable-lithium/47" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-cds-flow-services-lithium-redesign-only-stable-lithium/47</a></p> <p>and master<br/> <a href="https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-cds-flow-services-lithium-redesign-only-master/417" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-cds-flow-services-lithium-redesign-only-master/417</a></p> <p><a href="https://git.opendaylight.org/gerrit/#/c/23279/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/23279/</a></p> <p>I've fixed the test now:</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/23285/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/23285/</a></p> <p>Lithium (the release distro) does not fail this test case, and I was able to use the sandbox<br/> to verify the bug <b>was</b> there in a distro from 6/18 (before the fix for <a href="https://jira.opendaylight.org/browse/OPNFLWPLUG-507" title="OFHandshake thread leak leads to OOM" class="issue-link" data-issue-key="OPNFLWPLUG-507"><del>OPNFLWPLUG-507</del></a> was put in<br/> on 6/19)</p> <p>marking this RESOLVED/FIXED.</p> <p>I've fixed the test now:</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/23285/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/23285/</a></p> <p>Lithium (the release distro) does not fail this test case, and I was able to use the sandbox<br/> to verify the bug <b>was</b> there in a distro from 6/18 (before the fix for <a href="https://jira.opendaylight.org/browse/OPNFLWPLUG-507" title="OFHandshake thread leak leads to OOM" class="issue-link" data-issue-key="OPNFLWPLUG-507"><del>OPNFLWPLUG-507</del></a> was put in<br/> on 6/19)</p> <p>marking this RESOLVED/FIXED.</p> <p>This issue still seems to be there. It's not showing up in CI for some reason,<br/> but it did happen recently when the releng folks were trying to upgrade their<br/> CI robot VM to use centOS7. with the centOS6 vm this test is passing, so my <br/> guess is that something is broken in that test vm such that this bug is not being<br/> caught by CSIT.</p> <p>I've reproduced this with the robot suite locally. It has failed with both<br/> the Lithium release as well as the latest Beryllium distro from 9/22.</p> <p>Very good Jamo <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Michal,</p> <p>Can you look at this one before the next SR? I think next SR is in a month or 2.</p> <p>Abhijit</p> <p>Luis says fails in both He &amp; Li designs - is this related to OF Java Michal Polkorab?</p> <p>Hi all,<br/> I just tested stable/lithium HEAD locally and by both - </p> {He,Li} <p>-design I got no issues:</p> <p>for (( i=0; i&lt;1000; i++ )); do <br/> nc -w 1 localhost 6633 | hexdump -C &amp; <br/> sleep 0.1<br/> pkill -f 'nc -w'<br/> done</p> <p>netstat -pn | grep -E '66<span class="error">&#91;35&#93;</span>3' | nl</p> <p>Amount of threads used by controller was like 150. When running this cycle then the thread amount was always less then 160.<br/> Both designs behaved the same.<br/> With commented out the sleep line I got many opened tcp connections waiting for close. <br/> When sleep line was effective then connections got closed correctly.</p> <p>(In reply to michal rehak from comment #22)<br/> &gt; Hi all,<br/> &gt; I just tested stable/lithium HEAD locally and by both - </p> {He,Li} <p>-design I got<br/> &gt; no issues:<br/> &gt; <br/> &gt; for (( i=0; i&lt;1000; i++ )); do <br/> &gt; nc -w 1 localhost 6633 | hexdump -C &amp; <br/> &gt; sleep 0.1<br/> &gt; pkill -f 'nc -w'<br/> &gt; done<br/> &gt; <br/> &gt; <br/> &gt; netstat -pn | grep -E '66<span class="error">&#91;35&#93;</span>3' | nl<br/> &gt; <br/> &gt; Amount of threads used by controller was like 150. When running this cycle<br/> &gt; then the thread amount was always less then 160.<br/> &gt; Both designs behaved the same.<br/> &gt; With commented out the sleep line I got many opened tcp connections waiting<br/> &gt; for close. <br/> &gt; When sleep line was effective then connections got closed correctly.</p> <p>Yes, you are right. The test was working quickly enough to accidentally<br/> count the closing sockets as and mark it as a failure. I've pushed a fix<br/> in the test:</p> <p> <a href="https://git.opendaylight.org/gerrit/28356" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/28356</a> </p> <p>I verified manually that this new test code will fail on Helium SR2 where<br/> the bug was known to exist, but the test will pass with Lithium SR2.</p> <p>I also ran extra manual tests on Lithium SR2 to ensure I don't see this <br/> issue, so re-closing the bug now.</p> Development External issue ID 2429 External issue URL ODL SR Target Milestone Rank 0|i030fr: