OpenDaylight JIRA https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 [INFRAUTILS-29] CSIT hit infrautils.metrics.ThreadsWatcher limit of 1000 threads https://jira.opendaylight.org/browse/INFRAUTILS-29 infrautils <p><em><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=vorburger" class="user-hover" rel="vorburger">vorburger</a> filing this issue, based on <a href="https://lists.opendaylight.org/pipermail/infrautils-dev/2018-February/000595.html" class="external-link" target="_blank" rel="nofollow noopener">https://lists.opendaylight.org/pipermail/infrautils-dev/2018-February/000595.html</a>, setting <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jluhrsen" class="user-hover" rel="jluhrsen">jluhrsen</a> as Reporter; initially filing in infrautils, but will move to suitable project once we figure where these many threads come from.</em></p> <p>Attached is a Thread Dump with &gt;1000 threads in ODL produced by the infrautils.metrics.ThreadsWatcher (new; see <a href="https://jira.opendaylight.org/browse/INFRAUTILS-22" title="Automatic threads over threshold detection and logging" class="issue-link" data-issue-key="INFRAUTILS-22"><del>INFRAUTILS-22</del></a>, also FYI <a href="https://jira.opendaylight.org/browse/INFRAUTILS-21" title="Automatic deadlock detection logging" class="issue-link" data-issue-key="INFRAUTILS-21"><del>INFRAUTILS-21</del></a>) which I extracted out of the 1.1 GB odl1_karaf.log.1 from <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/19/jamo-netvirt-csit-1node-openstack-pike-upstream-stateful-oxygen/4/odl_1/" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/19/jamo-netvirt-csit-1node-openstack-pike-upstream-stateful-oxygen/4/odl_1/</a> ... </p> <p>Why are there some many threads? What created them?</p> INFRAUTILS-29 CSIT hit infrautils.metrics.ThreadsWatcher limit of 1000 threads Bug Highest Resolved Done Michael Vorburger Jamo Luhrsen Wed, 28 Feb 2018 10:05:28 +0000 Fri, 2 Mar 2018 15:39:48 +0000 Fri, 2 Mar 2018 15:39:48 +0000 Oxygen Fluorine Oxygen Fluorine 0 5 <p>Whoa, hang on a sec, what are all these threads in the attached essentially <tt>jstack</tt> equivalent:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> "Karaf ssh console user karaf" id=... state=TIMED_WAITING</pre> </div></div> <p>let's count them:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>grep "Karaf ssh" infrautils.metrics.ThreadsWatcher.txt | wc -l 430</pre> </div></div> <p>There are 1000 - 430 = 570 other threads we could look more into as well, but this jumps as as the biggest immediate problem... That 430 is a lot of threads for 1 Karaf SSH session, no? <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/wink.png" height="16" width="16" align="absmiddle" alt="" border="0"/> <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jluhrsen" class="user-hover" rel="jluhrsen">jluhrsen</a> perhaps there isn't just 1 SSH session to Karaf, in CSIT... could that be? I don't suppose there could be some sort of issue more on the CSIT script side than in ODL / Karaf here? Like you guys aren't by any chance... I'm not sure, like sending hundreds of ODL shell commands to Karaf from the CSIT robot thing and doing that over a new ssh connection each time and for some reason not closing the previous SSH session - kind of thing? Even if you are not intentionally doing it like that, maybe some bug in.. Robot, or its SSH library?</p> <p>Every suite and test is logged when they start. ROBOT Message: Starting suite blah, Starting test foo... So you would see that at last 240 for that. Each test also has a mdsal dump for each test, so another 235 or so. That could be the issue. Those commands were supposed to use a single connection to ODL but maybe something is busted there.</p> <p>Any ideas how to debug this - meaning to tell if on the odl side the sessions are open? We have journalctl logs on the odl nodes and I see many ssh connections opened, but most are followed with a a close. Thee are only 30 so I think the karaf ssh sessions must not be hitting there.</p> <p><a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-ocata-upstream-stateful-oxygen/965/odl_1/journalctl.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-ocata-upstream-stateful-oxygen/965/odl_1/journalctl.log.gz</a></p> <p>&gt; you would see that at last 240 for that. Each test also (...), so another 235 or so</p> <p>that adds up to 400 something, roughly matching, so yeah that's probably it then...</p> <p>&gt; Those commands were supposed to use a single connection to ODL</p> <p>what exactly do those "commands" technically do? separate SSH or (probably) new SSH for each?</p> <p>&gt; Any ideas how to debug this - meaning to tell if on the odl side the sessions are open?</p> <p>dunno how to easily check for open ports inside the JVM, but <tt>jstack</tt> and see those threads?</p> <p>&gt; journalctl logs on the odl nodes and I see many ssh connections opened, but most are followed with a a close</p> <p>not sure what journalctl shows here - is that really the connection of the CSIT scripts into Karaf? Isn't that showing OS system ssh logins? I guess we could somehow check the TCP socket for the SSH sessions to Karaf is closed? But maybe the problem isn't at that level - I mean it COULD also be a bug not in CSIT ssh client connections, but in Karaf's built-in ssh server itself... like perhaps it's dumb enough to keep a loose open thread for each session even after the SSH client disconnects? That would clearly cause a thread leak, over time, and be a "Karaf" bug (in quotes because this more likely would be a problem in whatever library does the SSH server for Karaf, not even Karaf itself).</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jluhrsen" class="user-hover" rel="jluhrsen">jluhrsen</a> FYI I was looking a bit more into this with <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=shague" class="user-hover" rel="shague">shague</a> ... here are some new conclusions:</p> <p>1. according to <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=shague" class="user-hover" rel="shague">shague</a>, when you wrote on <a href="https://lists.opendaylight.org/pipermail/infrautils-dev/2018-February/000604.html" class="external-link" target="_blank" rel="nofollow noopener">https://lists.opendaylight.org/pipermail/infrautils-dev/2018-February/000604.html</a> that "<span class="error">&#91;b&#93;</span> had the same trouble and that was without odl-mdsal-trace" that really meant "without feature:instal odl-mdsal-trace" but still WITH <a href="https://git.opendaylight.org/gerrit/#/c/68447/6/csit/libraries/OpenStackOperations.robot" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/68447/6/csit/libraries/OpenStackOperations.robot</a>, right?</p> <p>2. <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=shague" class="user-hover" rel="shague">shague</a> <em>"ran sandbox with c/68447 removed and it <a href="https://logs.opendaylight.org/sandbox/vex-yul-odl-jenkins-2/netvirt-csit-1node-openstack-ocata-shague-upstream-stateful-oxygen/1/odl_1/" class="external-link" target="_blank" rel="nofollow noopener">looked good for logs</a>"</em>. That confirms that what caused these thread leaks to suddenly pop up, with absolutely no code having changed anywhere otherwise, is c/68447.</p> <p>3. my gut now tells me this is more likely a problem in CSIT robot scripts, than in Karaf's SSH server (first suspicion, above). Is it at all possible that, somehow, that <tt>Issue_Command_On_Karaf_Console</tt> thing somehow lets hundreds of SSH sessions hanging open? I know nothing about that stuff, but e.g. seeing that in <tt>OpenStackOperations.robot</tt> there is a <tt>SSHLibrary.Close All Connections</tt> in line 1043 after one FOR <tt>Issue_Command_On_Karaf_Console trace:transaction</tt> but in line 560 there isn't - could that be a problem?</p> <p>I sent an <a href="https://lists.opendaylight.org/pipermail/integration-dev/2018-February/011190.html" class="external-link" target="_blank" rel="nofollow noopener">email</a> with some more details and observations. This is not anything to do with a bug in CSIT or dependant library. We can get to the same state with a bash script doing "ssh" in a loop (properly opening/closing etc).</p> <p>Anyway, the gist of it is that running karaf shell commands is opening some threads and they remain open for 60s. If we do enough of these at a fast enough rate, we'll cross this 1k threshold.</p> <p>I'm not sure what state we are in (thread count wise) in our CSIT environment, but I suppose we have to assume we are somewhere near that threshold to start with. I don't think CSIT is doing more than a handful of these things per second (if that. more like 1-2 every 10-300 seconds).</p> <p>&gt; get to the same state with a bash script doing "ssh" in a loop (properly opening/closing etc).</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jluhrsen" class="user-hover" rel="jluhrsen">jluhrsen</a> if you have it handy and wouldn't mind attaching that here, it could be useful for repro. in the future.</p> <p>BTW Karaf has a built-in “threads” command which outputs the thread dump, or a formatted list of threads with the “–list” option; you can combine that with “| wc -l” to get a quick measure of the number of running threads.</p> <p>This is the patch  ready to be merged: <a href="https://git.opendaylight.org/gerrit/#/c/68937/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/68937/</a></p> <blockquote><p>&gt; get to the same state with a bash script doing "ssh" in a loop (properly opening/closing etc).</p> <p>Jamo Luhrsen if you have it handy and wouldn't mind attaching that here, it could be useful for repro. in the future.</p></blockquote> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-keyword">for</span> i in {1..5000}; <span class="code-keyword">do</span> sshpass -p karaf ssh -p 8101 karaf@localhost -o StrictHostKeyChecking=no feature:list -i ; done</pre> </div></div> <blockquote><p>BTW Karaf has a built-in “threads” command which outputs the thread dump, or a formatted list of threads with the “–list” option; you can combine that with “| wc -l” to get a quick measure of the number of running threads.</p></blockquote> <p>cool, this can be added to CSIT at some point if we think it's valuable. The fix to calm down the logging<br/> is merged, so I think we are safe to add something like this. Here's <a href="https://git.opendaylight.org/gerrit/#/c/68982/" class="external-link" target="_blank" rel="nofollow noopener">something </a> along those lines. </p> Development Rank 0|i03ax3: