https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/OPNFLWPLUG-382 OpenFlowPlugin <p>1) Run karaf<br/> 2) Enable the below features<br/> feature:install odl-openflowplugin-flow-services<br/> feature:install odl-dlux-all<br/> feature:install odl-l2switch-all<br/> feature:install odl-l2switch-switch-ui<br/> feature:install odl-l2switch-switch-rest</p> <p>3) Using mininet connect 300 switches in linear topology<br/> 4) Controller crashed.<br/> 4) Switch connections move to CLOSE_WAIT state.<br/> 5) karaf console showed below exception<br/> java.io.IOException: Exception in opening zip file: /root/odl/distribution-karaf-0.3.0-SNAPSHOT/data/cache/org.eclipse.osgi/bundles/26/1/bundlefile<br/> at org.eclipse.osgi.framework.util.SecureAction.getZipFile(SecureAction.java:291)<br/> at org.eclipse.osgi.baseadaptor.bundlefile.ZipBundleFile.basicOpen(ZipBundleFile.java:87)<br/> at org.eclipse.osgi.baseadaptor.bundlefile.ZipBundleFile.getZipFile(ZipBundleFile.java:100)<br/> at org.eclipse.osgi.baseadaptor.bundlefile.ZipBundleFile.checkedOpen(ZipBundleFile.java:73)<br/> at org.eclipse.osgi.baseadaptor.bundlefile.ZipBundleFile.getEntry(ZipBundleFile.java:245)<br/> at org.eclipse.osgi.baseadaptor.loader.ClasspathManager.findClassImpl(ClasspathManager.java:542)<br/> at org.eclipse.osgi.baseadaptor.loader.ClasspathManager.findLocalClassImpl(ClasspathManager.java:492)<br/> at org.eclipse.osgi.baseadaptor.loader.ClasspathManager.findLocalClass(ClasspathManager.java:465)<br/> at org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.findLocalClass(DefaultClassLoader.java:216)<br/> at org.eclipse.osgi.internal.loader.BundleLoader.findLocalClass(BundleLoader.java:395)<br/> at org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(BundleLoader.java:464)<br/> at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:421)<br/> at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:412)<br/> at org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.loadClass(DefaultClassLoader.java:107)<br/> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)<br/> at jline.internal.Log.warn(Log.java:114)<br/> at jline.internal.TerminalLineSettings.getProperty(TerminalLineSettings.java:97)<br/> at jline.UnixTerminal.getWidth(UnixTerminal.java:75)<br/> at jline.console.ConsoleReader.drawBuffer(ConsoleReader.java:834)<br/> at jline.console.ConsoleReader.drawBuffer(ConsoleReader.java:853)<br/> at jline.console.ConsoleReader.putString(ConsoleReader.java:793)<br/> at jline.console.ConsoleReader.readLine(ConsoleReader.java:2533)<br/> at jline.console.ConsoleReader.readLine(ConsoleReader.java:2162)<br/> at org.apache.karaf.shell.console.impl.jline.ConsoleImpl.readAndParseCommand(ConsoleImpl.java:280)<br/> at org.apache.karaf.shell.console.impl.jline.ConsoleImpl.run(ConsoleImpl.java:207)<br/> at java.lang.Thread.run(Thread.java:744)<br/> at org.apache.karaf.shell.console.impl.jline.ConsoleFactoryService$3.doRun(ConsoleFactoryService.java:126)<br/> at org.apache.karaf.shell.console.impl.jline.ConsoleFactoryService$3$1.run(ConsoleFactoryService.java:117)<br/> at java.security.AccessController.doPrivileged(Native Method)<br/> at org.apache.karaf.jaas.modules.JaasHelper.doAs(JaasHelper.java:47)<br/> at org.apache.karaf.shell.console.impl.jline.ConsoleFactoryService$3.run(ConsoleFactoryService.java:115)<br/> Caused by: java.io.FileNotFoundException: /root/odl/distribution-karaf-0.3.0-SNAPSHOT/data/cache/org.eclipse.osgi/bundles/26/1/bundlefile (Too many open files)<br/> at java.util.zip.ZipFile.open(Native Method)<br/> at java.util.zip.ZipFile.&lt;init&gt;(ZipFile.java:215)<br/> at java.util.zip.ZipFile.&lt;init&gt;(ZipFile.java:145)<br/> at java.util.zip.ZipFile.&lt;init&gt;(ZipFile.java:159)<br/> at org.eclipse.osgi.framework.util.SecureAction.getZipFile(SecureAction.java:274)<br/> ... 30 more</p> <p>===============================================================<br/> Note:<br/> With Helium SR2 version the same test case passed<br/> Just connecting 300 switches without any links works fine.</p> <p>===============================================================</p> <p>Operating System: Linux<br/> Platform: Other</p> OPNFLWPLUG-382

[Lithium redesign] OOM errors, CPU 100% with 128 connected switches

Bug Resolved Done Unassigned SANDEEP GANGADHARAN Wed, 18 Mar 2015 23:47:48 +0000 Mon, 27 Sep 2021 09:01:27 +0000 Fri, 8 May 2015 23:42:53 +0000 General 0 8 <p>Attachment log.zip has been added with description: Attaching the logs</p> <p>Repeated this type of test with the lithium redesign (i.e., odl-openflowplugin-app-new-lldp-speaker feature).</p> <p>with a linear mininet topology using 128 switches, the controller will eventually (takes aprox 30m) give<br/> an OOM and all four CPUs are running at 100% making the system unusable. Eventually, the JVM is killed.</p> <p>not sure how relevant these messages are, but just a snippet from the karaf console:</p> <p>Exception in thread "Thread-3042" java.lang.OutOfMemoryError: GC overhead limit exceeded<br/> Exception in thread "CommitFutures-9" java.lang.OutOfMemoryError: GC overhead limit exceeded<br/> Exception in thread "Thread-3127" java.lang.OutOfMemoryError: GC overhead limit exceeded<br/> Exception in thread "Thread-3152" java.lang.OutOfMemoryError: GC overhead limit exceeded<br/> Exception in thread "Thread-3153" java.lang.OutOfMemoryError: GC overhead limit exceeded<br/> Uncaught error from thread <span class="error">&#91;odl-cluster-rpc-scheduler-1&#93;</span> shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[odl-cluster-rpcException in thread "fileinstall-/home/sdn/distribution-karaf-0.3.0-SNAPSHOT/deploy" java.lang.OutOfMemoryError: GC overhead limit exceeded</p> <p>Can you take a memory dump (-XX:+HeapDumpOnOutOfMemoryError for example), as this looks like an unbounded queue or a memory leak somewhere.</p> <p>My JVM was crippled long before I saw OutOfMemory so I did a dev:dump-create as fast as I could. Here's what I see in visualvm.</p> <p>Attachment dump.png has been added with description: heapdump</p> <p>Here's another dump, this time I made sure to run mininet with --switch ovsk,protocols=OpenFlow13</p> <p>Attachment dump2.png has been added with description: heapdump_of13</p> <p>Attachment openflow_multipartRequest_memoryLeak.jpg has been added with description: jvisual screenshot</p> <p>As Evan has already pointed out, and I have confirmed in my setup, it looks like the memory leak is around this:</p> <p>org.opendaylight.yang.gen.v1.urn.opendaylight.openflow.protocol.rev130731.multipart.reply.multipart.reply.body.multipart.reply.table._case.multipart.reply.table.TableStatsBuilder$TableStatsImpl</p> <p>The overall symptom is that memory is slowly consumed in my 100 switch topology until it hits the limit (3G in my case). multipart request/reply<br/> is a period event so if it's the activity leaking memory, then it would<br/> all jive.</p> <p>Added Hema if she has any knowledge of this table features code mentioned by Jamo in the previous comment as the source of the memory leak?</p> <p>does not appear to be resolved, at least as far as CI is concerned:<br/> <a href="https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-periodic-1node-cds-scalability-daily-only-master/9/robot/report/log.html#s1-s1-t1" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-periodic-1node-cds-scalability-daily-only-master/9/robot/report/log.html#s1-s1-t1</a></p> <p>this was using this specific distro:<br/> distribution-karaf-0.3.0-20150507.123410-1486.zip</p> <p>I left this open because of a CI scale test that was turning up an OOM Exception, but now it's been determined that the root cause in that scale test is from <a href="https://jira.opendaylight.org/browse/OPNFLWPLUG-322" title="Need to close the ODL Denial of Service interface" class="issue-link" data-issue-key="OPNFLWPLUG-322"><del>OPNFLWPLUG-322</del></a> and is only happening because the OVS version used<br/> in CI is version 2.0.2. The same test with newer OVS versions does not <br/> have trouble.</p> <p>scale test:<br/> <a href="https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-periodic-1node-cds-scalability-daily-only-master/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-periodic-1node-cds-scalability-daily-only-master/</a></p> Blocks OPNFLWPLUG-357 Development External issue ID 2869 External issue URL ODL SR Target Milestone Rank 0|i030t3: