https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/GENIUS-97 genius <p>Internal downstream testing reports OOM with latest stable/nitrogen builds.</p> <p>HPROF analysis by MAT points to something really badly wrong in IdPoolListener.</p> <p>see attached ZIP (ignore DJC, that's <a href="https://jira.opendaylight.org/browse/GENIUS-96" title="DataStoreJobCoordinator OOM" class="issue-link" data-issue-key="GENIUS-96"><del>GENIUS-96</del></a>) </p> GENIUS-97

IdPoolListener OOM

Bug Highest Verified Cannot Reproduce Kency Kurian Michael Vorburger Fri, 3 Nov 2017 22:06:00 +0000 Thu, 19 Apr 2018 14:39:50 +0000 Thu, 19 Apr 2018 14:39:50 +0000 Nitrogen-SR1 Nitrogen-SR1 0 7 <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=tpantelis" class="user-hover" rel="tpantelis">tpantelis</a> points out that:</p> <blockquote><p>Looks like an Executor queue is backed up - something's probably stuck - we need to get thread dumps.</p></blockquote> <p>Does anyone know any automatic way to get the JVM to do that, just like the -XX:+HeapDumpOnOutOfMemoryError makes it create the HRPOF? Or we ask folks to keep doing jstack up to until it boums - is that the best we can offer? (Perhaps I could come up with some nifty .. thingie in infrautils that does this - start logging WARN level if the thread count is above a (configurable) threshold, that would be quite useful in this kind of situation, wouldn't it?)</p> <p>It's probably not the thread count - it's a particular thread that gets stuck. Maybe the app code gets deadlocked or blocking on something (there's a lot of that). Or it's the cumulative effect of many queued tasks that are slow, ie the classic producer faster than the consumer. All of the listeners and JC use unbounded queues so if something gets stuck or runs slowly, it is inevitable a queue will back up and hit OOM. </p> <p>What the testers are doing? Scale testing? longevity? is it reproducible? Are they tracking the memory usage? If so, as it climbs then start taking thread dumps. </p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=Kency" class="user-hover" rel="Kency">Kency</a> will you be able to analyze this? I don't think it is IdPoolListener, probably too many other things have slowed down the system, and this is just an after effect</p> <p><a href="https://jira.opendaylight.org/browse/INFRAUTILS-23" title="Remove Executors.newSingleThreadExecutor() and replace users with SpecialExecutors.newBoundedSingleThreadExecutor()" class="issue-link" data-issue-key="INFRAUTILS-23">INFRAUTILS-23</a> proposes to switch to using bounded queues in listeners, and <a href="https://jira.opendaylight.org/browse/INFRAUTILS-24" title="JobCoordinator should use bounded Executors and bound its own job queue" class="issue-link" data-issue-key="INFRAUTILS-24"><del>INFRAUTILS-24</del></a> the same for JC.</p> <p><a href="https://jira.opendaylight.org/browse/INFRAUTILS-21" title="Automatic deadlock detection logging" class="issue-link" data-issue-key="INFRAUTILS-21"><del>INFRAUTILS-21</del></a> and maybe <a href="https://jira.opendaylight.org/browse/INFRAUTILS-22" title="Automatic threads over threshold detection and logging" class="issue-link" data-issue-key="INFRAUTILS-22"><del>INFRAUTILS-22</del></a> will do something re. automatic way to dump deadlocked threads (there's no readily available JVM option AFAIK).</p> <p>We need to fix the bug hit during testing in this JIRA issue; I'll get us a jstack.</p> <p>The task (in update method) being executed under IdPoolListener is a short-lived one which i don't think causes a thread starvation. <span class="error">&#91;1&#93;</span> is invoked only in a corner case where updating local id pool when reclamation happen from other local id pool due to id exhaustion in node's local and parent id pools. Does that happen in your tests ?</p> <p>It's better to look at jstack and jmap to see what is actually going on.</p> <p><span class="error">&#91;1&#93;</span> <a href="https://git.opendaylight.org/gerrit/gitweb?p=genius.git;a=blob;f=idmanager/idmanager-impl/src/main/java/org/opendaylight/genius/idmanager/IdPoolListener.java;h=adc77023c371853543745fe50ac4f649c7c5bab6;hb=refs/heads/master#l67" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/gitweb?p=genius.git;a=blob;f=idmanager/idmanager-impl/src/main/java/org/opendaylight/genius/idmanager/IdPoolListener.java;h=adc77023c371853543745fe50ac4f649c7c5bab6;hb=refs/heads/master#l67</a></p> <p>Is this issue related to <a href="https://jira.opendaylight.org/browse/NETVIRT-974" class="external-link" rel="nofollow">https://jira.opendaylight.org/browse/NETVIRT-974</a> ?</p> <p>Is this truly a blocker for Nitrogen-SR1? Thanks!</p> <p>&gt; Is this issue related to <a href="https://jira.opendaylight.org/browse/NETVIRT-974" class="external-link" rel="nofollow">https://jira.opendaylight.org/browse/NETVIRT-974</a> ?</p> <p>no, not at all. But it could turn out to have one and the same single cause as <a href="https://jira.opendaylight.org/browse/GENIUS-96" title="DataStoreJobCoordinator OOM" class="issue-link" data-issue-key="GENIUS-96"><del>GENIUS-96</del></a> though - we don't know yet.</p> <p>&gt; Is this truly a blocker for Nitrogen-SR1? Thanks!</p> <p>Yup.</p> <p>Attached jstack_info.log.xz has 67x jstack taken once per minute leading up to the OOM. Anyone spotting anything useful here?</p> <p>This can only be reproduced with latest stable/Nitrogen HEAD (which will be SR1), NOT with the first September 26 Nitro, so recently broke.</p> <p>Do we have ETA on resolution? Need input to assess how far we have to push out Nitrogen-SR1. Thanks!</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=klou" class="user-hover" rel="klou">klou</a> ETA is when it's Fixed. We can try to do earlier then when it's Done, but we would need a time machine.</p> <p> <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=klou" class="user-hover" rel="klou">klou</a> &amp; <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=vivekshiv" class="user-hover" rel="vivekshiv">vivekshiv</a> FYI I'm hoping to update and possibly close this one tomorrow, currentlywaiting for feedback from <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=ltomasbo%40redhat.com" class="user-hover" rel="ltomasbo@redhat.com">ltomasbo@redhat.com</a> ...</p> <p>Closing as CANNOT REPRO, because <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=ltomasbo" class="user-hover" rel="ltomasbo">ltomasbo</a> has clarified that he only hits an OOM with the 512 MB instead of 2 GB due to devstack, see <a href="https://jira.opendaylight.org/browse/GENIUS-96" title="DataStoreJobCoordinator OOM" class="issue-link" data-issue-key="GENIUS-96"><del>GENIUS-96</del></a>. </p> <p>What was particularly confusing me here was that the attached Leak Suspects MAT report I produced from his HPROF <b>does</b> show 2 GB, BUT this was only because: (quote) <em>"probably the previous one left the env (DB) unstable, with existing ports and flows... I then removed the env, recreated it with 0.7.1 (= SR1 to-be) with 2 GB on a completely fresh install. I'm re-testing it and is not seeing the OOM anymore."</em></p> Blocks INFRAUTILS-23 Development Priority Rank 0|i03793: