https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/OPNFLWPLUG-833 OpenFlowPlugin <p>OOM error seen about 4 hours after ODL bring up:</p> <p>2016-12-14 12:34:35,306 | WARN | entLoopGroup-6-1 | NioServerSocketChannel | 136 - io.netty.common - 4.0.37.Final | Failed to create a new channel from an accepted socket.<br/> java.lang.OutOfMemoryError: Java heap space</p> <p>2016-12-14 12:34:27,756 | ERROR | ntLoopGroup-7-28 | ExceptionHandler | 285 - org.opendaylight.ovsdb.library - 1.3.2.Boron-SR2 | Exception occurred while processing connection pipeline <br/> io.netty.handler.codec.DecoderException: java.lang.OutOfMemoryError: Java heap space</p> <p>karaf logs attached</p> <p>Operating System: All<br/> Platform: All</p> OPNFLWPLUG-833

OOM due to suspected memory leak in akka.dispatch.Dispatcher found by MAT in hprof

Bug Resolved Cannot Reproduce Unassigned Tim Rozet Wed, 14 Dec 2016 17:18:08 +0000 Mon, 27 Sep 2021 09:01:58 +0000 Wed, 21 Jun 2017 13:33:56 +0000 General 0 4 <p>Attachment karaf.zip has been added with description: karaf logs</p> <p>Hi Tim,</p> <p>do you have heapdump from this OOM? It will help us figure out where the leak is actually happening. Also do you have some detail about what operations were happening during these 4 hours of up time?</p> <p>Due to the size of the heap, I uploaded it to here:<br/> <a href="http://artifacts.opnfv.org/apex/random/java_pid19570.hprof.zip" class="external-link" target="_blank" rel="nofollow noopener">http://artifacts.opnfv.org/apex/random/java_pid19570.hprof.zip</a></p> <p>All that happened on this setup was an external network/subnet create + a basic healthcheck, which consists of creating a tenant network, and then bringing up a few instances and checking if they got dhcp ip (which they didnt). The deployment then sat idle until crash.</p> <p>Hi Tim,</p> <p>Quick question, are you running it in clustered setup?</p> <p>There are multiple OpenStack neutron nodes, but ODL is only running as a single instance and not clustered.</p> <p>Attachment java_pid19570_Leak_Suspects.zip has been added with description: MAT akka.dispatch.Dispatcher memory leak suspect HTML report</p> <p>Just had a look at this one using MAT, see report in attachment.</p> <p>This very most likely isn't actually OVSDB or io.netty.handler.codec.DecoderException related; that's just when it hit, so moving project and editing summary.</p> <p>I'll now email mdsal-dev &amp; controller-dev to get first reactions about this.</p> <p>Attachment Bug7370_Threads.zip has been added with description: MAT list of 604 threads with stack traces HTML report</p> <p><a href="https://lists.opendaylight.org/pipermail/mdsal-dev/2017-May/001218.html" class="external-link" target="_blank" rel="nofollow noopener">https://lists.opendaylight.org/pipermail/mdsal-dev/2017-May/001218.html</a></p> <p>Moving project again, from mdsal to openflowplugin, because <a href="https://lists.opendaylight.org/pipermail/mdsal-dev/2017-May/001219.html" class="external-link" target="_blank" rel="nofollow noopener">https://lists.opendaylight.org/pipermail/mdsal-dev/2017-May/001219.html</a> clarified that this means that a listener (in openflowplugin) instead of async dealing with events locked an Akka Dispatcher thread (and lead to an OOM there).</p> <p>I had had a closer look at the org.opendaylight.openflowplugin.impl.util.DeviceInitializationUtils.initializeNodeInformation on master as of 3 weeks ago, and found that the code there meanwhile considerably changed - the "blocking get on a Future" which Tom picked up on in the trace doesn't seem to be there anymore. <a href="https://lists.opendaylight.org/pipermail/mdsal-dev/2017-May/001220.html" class="external-link" target="_blank" rel="nofollow noopener">https://lists.opendaylight.org/pipermail/mdsal-dev/2017-May/001220.html</a> seems to say the same.</p> <p>I'm thus closing this particular OOM as "WORKSFORME" (in master; no interest in chasing this in stable maintenance). Of course, this does not mean that there could not be other OOM issues, elsewhere... we need people to reproduce them and provide more HPROF dump we can analyze (just like first did Tim here; thanks again).</p> <p>&gt; closer look at the org.opendaylight.openflowplugin.impl.util.DeviceInitializationUtils.initializeNodeInformation on master as of 3 weeks ago, and found that the code there meanwhile considerably changed - the "blocking get on a Future" which Tom picked up on in the trace doesn't seem to be there anymore. </p> <p>I've also had a (very quick) look at stable/boron branch, and it's the same there already; the code there meanwhile considerably changed - the "blocking get on a Future" which Tom picked up on in the trace doesn't seem to be there anymore.</p> Development External issue ID 7370 External issue URL Priority Rank 0|i033lb: