https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/ODLPARENT-49 odlparent <p>JamO &amp; others (incl. dfarrell) report relatively frequently seeing EOFError in CSIT Robot Suites when ssh into Karaf (not OS level sshd).</p> <p>Operating System: Linux<br/> Platform: All</p> ODLPARENT-49

Karaf ssh EOFError (is it due to low entropy, due to Java's default use of blocking /dev/random instead of /dev/urandom?)

Bug Resolved Done Unassigned Michael Vorburger Fri, 23 Sep 2016 10:12:01 +0000 Wed, 24 Jan 2018 14:27:03 +0000 Wed, 18 Jan 2017 13:59:13 +0000 General 0 4 <p><a href="https://lists.opendaylight.org/pipermail/dev/2016-September/002704.html" class="external-link" target="_blank" rel="nofollow noopener">https://lists.opendaylight.org/pipermail/dev/2016-September/002704.html</a> : One theory we have is that perhaps this could be due to low entropy. One solution to this would be to make the JVM process of the ssh server in Karaf (which apparently we've configured to use boucycastle) use the non-blocking /dev/urandom instead of the blocking /dev/random.</p> <p><a href="https://brooklyn.apache.org/documentation/increase-entropy.html" class="external-link" target="_blank" rel="nofollow noopener">https://brooklyn.apache.org/documentation/increase-entropy.html</a> is another Java based system with similar issues, so maybe this is related.</p> <p><a href="https://lists.opendaylight.org/pipermail/dev/2016-September/002727.html" class="external-link" target="_blank" rel="nofollow noopener">https://lists.opendaylight.org/pipermail/dev/2016-September/002727.html</a> =&gt; <a href="https://git.opendaylight.org/gerrit/#/c/45749/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/45749/</a></p> <p><a href="https://lists.opendaylight.org/pipermail/dev/2016-September/002734.html" class="external-link" target="_blank" rel="nofollow noopener">https://lists.opendaylight.org/pipermail/dev/2016-September/002734.html</a> Ryan Goulding confirms that: "In our downstream internal CI, I had to<br/> make adjustments to seed from /dev/urandom instead as we were experiencing<br/> hanging tests (especially for netconf through sshd-core when mounting<br/> several devices)."</p> <p><a href="https://lists.opendaylight.org/pipermail/dev/2016-September/002785.html" class="external-link" target="_blank" rel="nofollow noopener">https://lists.opendaylight.org/pipermail/dev/2016-September/002785.html</a> Jamo Luhrsen reports entropy was 168 in /proc/sys/kernel/random/entropy_avail when this happened.</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/45760/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/45760/</a> is a change to use -Djava.security.egd=<a href="file:/dev/./urandom" class="external-link" target="_blank" rel="nofollow noopener">file:/dev/./urandom</a> on start up (based on <a href="http://stackoverflow.com/a/2325109/421602" class="external-link" target="_blank" rel="nofollow noopener">http://stackoverflow.com/a/2325109/421602</a>).</p> <p><a href="https://lists.opendaylight.org/pipermail/dev/2016-September/002786.html" class="external-link" target="_blank" rel="nofollow noopener">https://lists.opendaylight.org/pipermail/dev/2016-September/002786.html</a> disagrees that low entropy could be the root cause of this problem.</p> <p>This <a href="http://www.2uo.de/myths-about-urandom/" class="external-link" target="_blank" rel="nofollow noopener">http://www.2uo.de/myths-about-urandom/</a> is interesting.</p> <p>(In reply to Michael Vorburger from comment #3)<br/> &gt; <a href="https://lists.opendaylight.org/pipermail/dev/2016-September/002786.html" class="external-link" target="_blank" rel="nofollow noopener">https://lists.opendaylight.org/pipermail/dev/2016-September/002786.html</a><br/> &gt; disagrees that low entropy could be the root cause of this problem.</p> <p>But that might well be wrong, given that OpenSSH uses /dev/urandom anyway.</p> <p>FTR: <a href="https://bugs.openjdk.java.net/browse/JDK-4705093" class="external-link" target="_blank" rel="nofollow noopener">https://bugs.openjdk.java.net/browse/JDK-4705093</a> = <a href="http://bugs.java.com/view_bug.do?bug_id=4705093" class="external-link" target="_blank" rel="nofollow noopener">http://bugs.java.com/view_bug.do?bug_id=4705093</a></p> <p><a href="https://docs.oracle.com/cd/E13209_01/wlcp/wlss30/configwlss/jvmrand.html" class="external-link" target="_blank" rel="nofollow noopener">https://docs.oracle.com/cd/E13209_01/wlcp/wlss30/configwlss/jvmrand.html</a></p> <p>Once we have confirmation that with the merge of the <a href="https://git.opendaylight.org/gerrit/#/c/45760/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/45760/</a> (now cleaned up/refined, thanks Stephen Kitt!) this problem disappears, we should also:</p> <p>A. Open a bug + pull request (skitt) on upstream Karaf (4) to do the same as we are</p> <p>B. Open a bug on <a href="https://bugzilla.redhat.com" class="external-link" target="_blank" rel="nofollow noopener">https://bugzilla.redhat.com</a> to suggest that perhaps OpenJDK RPM packages could "Change the default for java.security in $JAVA_HOME/jre/lib/security/java.security from <a href="file:/dev/random" class="external-link" target="_blank" rel="nofollow noopener">file:/dev/random</a> to <a href="file:/dev/urandom" class="external-link" target="_blank" rel="nofollow noopener">file:/dev/urandom</a>", some time. "If deemed too risky a change for a Java 8 security patch, perhaps consider this for Java 9 packages? If this is controversial, perhaps it's time to re-raise this upstream on OpenJDK?"</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/50327/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/50327/</a> fixes SingleFeatureTest up re. this.</p> <p>Attached a jstack of the Karaf stuck due to this.</p> <p>Attachment karaf.jstack.txt has been added with description: jstack of Karaf stuck in netconf SSH server init due to low entropy</p> <p>This is still affecting CSIT. See comment <span class="error">&#91;0&#93;</span>.</p> <p>Either releng/builder manages to prepare machines with sufficinet entropy, or we should switch to testing Karaf with "-Djava.security.egd=<a href="file:/dev/./urandom" class="external-link" target="_blank" rel="nofollow noopener">file:/dev/./urandom</a>" option added.</p> <p><span class="error">&#91;0&#93;</span> <a href="https://git.opendaylight.org/gerrit/#/c/50362/3" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/50362/3</a></p> <p>More info from <a href="https://lists.opendaylight.org/pipermail/integration-dev/2017-January/008955.html" class="external-link" target="_blank" rel="nofollow noopener">https://lists.opendaylight.org/pipermail/integration-dev/2017-January/008955.html</a></p> <p>&gt; <span class="error">&#91;3&#93;</span> <a href="https://git.opendaylight.org/gerrit/#/c/45760" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/45760</a></p> <p>Oh, that was merged long time ago,<br/> I see karaf started with<br/> -Djava.security.egd=<a href="file:/dev/./urandom" class="external-link" target="_blank" rel="nofollow noopener">file:/dev/./urandom</a><br/> already, so something is not working right.<br/> Are we sure Karaf console ssh server takes this option into account?</p> <p>It the option worked, we would not need more entropy on ODL_SYSTEM.</p> <p>Vratko.</p> <p><a href="https://git.opendaylight.org/gerrit/50594" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/50594</a> claim to fix CSIT failures.</p> <p>If that is true, this error was not about entropy after all.<br/> Karaf SSH server was just slow for some reason, and Robot SSHLibrary with default timeout was not giving a helpful failure message.</p> <p>There is still some chance that the Karaf SSH server is slow because it blocks on /dev/random, but that would need closer examination to decide.</p> Development External issue ID 6790 External issue URL Issue Type ODL SR Target Milestone Rank 0|i022zr: