OpenDaylight JIRA https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 [L2SWITCH-22] Out of memory error when l2 switch installed https://jira.opendaylight.org/browse/L2SWITCH-22 l2switch <p>Out of memory error is observed when l2switch is installed and flows are being added to the controller through the REST API. The out of memory condition seems to be related to the PS Survivor space (see attached jconsole snapshot). The exception message observed last was as follows (but it can vary):</p> <p>objc<span class="error">&#91;2718&#93;</span>: Class JavaLaunchHelper is implemented in both /Library/Java/JavaVirtualMachines/jdk1.7.0_65.jdk/Contents/Home/bin/java and /Library/Java/JavaVirtualMachines/jdk1.7.0_65.jdk/Contents/Home/jre/lib/libinstrument.dylib. One of the two will be used. Which one is undefined.<br/> Exception in thread "DOM-OPER-DCL-77" java.lang.OutOfMemoryError: unable to create new native thread<br/> at java.lang.Thread.start0(Native Method)<br/> at java.lang.Thread.start(Thread.java:714)<br/> at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)<br/> at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1017)<br/> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1163)<br/> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)<br/> at java.lang.Thread.run(Thread.java:745)</p> <p>To reproduce:<br/> 1. Get the RC2 image distribution-karaf-0.2.0-Helium-RC2-verify-v201409250627</p> <p>2. Install features as shown below (basically, openflow and l2switch)</p> <p>3. Run the attached python scripts to generate/ clear flows until the <br/> out of memory issue occurs:<br/> 3.1 &gt; python flow_config_perf.py --nflows=200 --nthreads=10 --ncycles=10 --no-delete --auth</p> <p> 3.2 Wait until 20000 flows show up in the operational data space. Use: <br/> &gt; python inventory_crawler.py --plevel=2 --auth</p> <p> 3.3 clear the flows from the controller:<br/> &gt; python config_cleanup.py --auth</p> <p> 3.4 Repeat until the out of memory condition occurs</p> <p>Installed features:</p> <p>tandard | 3.0.1 | x | standard-3.0.1 | Karaf standard feature <br/> config | 3.0.1 | x | standard-3.0.1 | Provide OSGi ConfigAdmin support <br/> package | 3.0.1 | x | standard-3.0.1 | Package commands and mbeans <br/> http | 3.0.1 | x | standard-3.0.1 | Implementation of the OSGI HTTP Service <br/> war | 3.0.1 | x | standard-3.0.1 | Turn Karaf as a full WebContainer <br/> kar | 3.0.1 | x | standard-3.0.1 | Provide KAR (KARaf archive) support <br/> ssh | 3.0.1 | x | standard-3.0.1 | Provide a SSHd server on Karaf <br/> management | 3.0.1 | x | standard-3.0.1 | Provide a JMX MBeanServer and a set of MBeans in K<br/> odl-netconf-api | 0.2.5-Helium-RC2-verify-v201409250627 | x | odl-netconf-0.2.5-Helium-RC2-verify-v201409250627 | OpenDaylight :: Netconf :: API <br/> odl-netconf-mapping-api | 0.2.5-Helium-RC2-verify-v201409250627 | x | odl-netconf-0.2.5-Helium-RC2-verify-v201409250627 | OpenDaylight :: Netconf :: Mapping API <br/> odl-netconf-util | 0.2.5-Helium-RC2-verify-v201409250627 | x | odl-netconf-0.2.5-Helium-RC2-verify-v201409250627 | <br/> odl-netconf-impl | 0.2.5-Helium-RC2-verify-v201409250627 | x | odl-netconf-0.2.5-Helium-RC2-verify-v201409250627 | OpenDaylight :: Netconf :: Impl <br/> odl-config-netconf-connector | 0.2.5-Helium-RC2-verify-v201409250627 | x | odl-netconf-0.2.5-Helium-RC2-verify-v201409250627 | OpenDaylight :: Netconf :: Connector <br/> odl-netconf-netty-util | 0.2.5-Helium-RC2-verify-v201409250627 | x | odl-netconf-0.2.5-Helium-RC2-verify-v201409250627 | OpenDaylight :: Netconf :: Netty Util <br/> odl-netconf-monitoring | 0.2.5-Helium-RC2-verify-v201409250627 | x | odl-netconf-0.2.5-Helium-RC2-verify-v201409250627 | OpenDaylight :: Netconf :: Monitoring <br/> odl-yangtools-models | 0.6.2-Helium-RC2-verify-v201409250627 | x | odl-yangtools-0.6.2-Helium-RC2-verify-v201409250627 | OpenDaylight :: Yangtools :: Models <br/> odl-yangtools-data-binding | 0.6.2-Helium-RC2-verify-v201409250627 | x | odl-yangtools-0.6.2-Helium-RC2-verify-v201409250627 | OpenDaylight :: Yangtools :: Data Binding <br/> odl-yangtools-binding | 0.6.2-Helium-RC2-verify-v201409250627 | x | odl-yangtools-0.6.2-Helium-RC2-verify-v201409250627 | OpenDaylight :: Yangtools :: Binding <br/> odl-yangtools-common | 0.6.2-Helium-RC2-verify-v201409250627 | x | odl-yangtools-0.6.2-Helium-RC2-verify-v201409250627 | OpenDaylight :: Yangtools :: Common <br/> odl-yangtools-binding-generator | 0.6.2-Helium-RC2-verify-v201409250627 | x | odl-yangtools-0.6.2-Helium-RC2-verify-v201409250627 | OpenDaylight :: Yangtools :: Binding Generator <br/> odl-config-persister | 0.2.5-Helium-RC2-verify-v201409250627 | x | odl-config-persister-0.2.5-Helium-RC2-verify-v201409250627 | OpenDaylight :: Config Persister <br/> odl-config-startup | 0.2.5-Helium-RC2-verify-v201409250627 | x | odl-config-persister-0.2.5-Helium-RC2-verify-v201409250627 | OpenDaylight :: Config Persister:: Config Startup <br/> odl-mdsal-common | 1.1-Helium-RC2-verify-v201409250627 | x | odl-config-0.2.5-Helium-RC2-verify-v201409250627 | OpenDaylight :: Config :: All <br/> odl-config-api | 0.2.5-Helium-RC2-verify-v201409250627 | x | odl-config-0.2.5-Helium-RC2-verify-v201409250627 | OpenDaylight :: Config :: API <br/> odl-config-netty-config-api | 0.2.5-Helium-RC2-verify-v201409250627 | x | odl-config-0.2.5-Helium-RC2-verify-v201409250627 | OpenDaylight :: Config :: Netty Config API <br/> odl-config-core | 0.2.5-Helium-RC2-verify-v201409250627 | x | odl-config-0.2.5-Helium-RC2-verify-v201409250627 | OpenDaylight :: Config :: Core <br/> odl-config-manager | 0.2.5-Helium-RC2-verify-v201409250627 | x | odl-config-0.2.5-Helium-RC2-verify-v201409250627 | OpenDaylight :: Config :: Manager <br/> odl-l2switch-switch | 0.1.0-Helium-RC2-verify-v201409250627 | x | l2switch-0.1.0-Helium-RC2-verify-v201409250627 | OpenDaylight :: L2Switch :: Switch <br/> odl-l2switch-hosttracker | 0.1.0-Helium-RC2-verify-v201409250627 | x | l2switch-0.1.0-Helium-RC2-verify-v201409250627 | OpenDaylight :: L2Switch :: HostTracker <br/> odl-l2switch-addresstracker | 0.1.0-Helium-RC2-verify-v201409250627 | x | l2switch-0.1.0-Helium-RC2-verify-v201409250627 | OpenDaylight :: L2Switch :: AddressTracker <br/> odl-l2switch-arphandler | 0.1.0-Helium-RC2-verify-v201409250627 | x | l2switch-0.1.0-Helium-RC2-verify-v201409250627 | OpenDaylight :: L2Switch :: ArpHandler <br/> odl-l2switch-loopremover | 0.1.0-Helium-RC2-verify-v201409250627 | x | l2switch-0.1.0-Helium-RC2-verify-v201409250627 | OpenDaylight :: L2Switch :: LoopRemover <br/> odl-l2switch-packethandler | 0.1.0-Helium-RC2-verify-v201409250627 | x | l2switch-0.1.0-Helium-RC2-verify-v201409250627 | OpenDaylight :: L2Switch :: PacketHandler <br/> odl-openflowjava-protocol | 0.5-Helium-RC2-verify-v201409250627 | x | odl-openflowjava-0.5-Helium-RC2-verify-v201409250627 | OpenDaylight :: Openflow Java :: Protocol <br/> odl-restconf | 1.1-Helium-RC2-verify-v201409250627 | x | odl-controller-1.1-Helium-RC2-verify-v201409250627 | OpenDaylight :: Restconf <br/> odl-restconf-noauth | 1.1-Helium-RC2-verify-v201409250627 | x | odl-controller-1.1-Helium-RC2-verify-v201409250627 | OpenDaylight :: Restconf <br/> odl-config-netty | 0.2.5-Helium-RC2-verify-v201409250627 | x | odl-config-persister-0.2.5-Helium-RC2-verify-v201409250627 | OpenDaylight :: Config-Netty <br/> odl-flow-model | 1.1-Helium-RC2-verify-v201409250627 | x | odl-flow-1.1-Helium-RC2-verify-v201409250627 | OpenDaylight :: Flow :: Model <br/> odl-flow-services | 1.1-Helium-RC2-verify-v201409250627 | x | odl-flow-1.1-Helium-RC2-verify-v201409250627 | OpenDaylight :: Flow :: Services <br/> odl-protocol-framework | 0.5.0-Helium-RC2-verify-v201409250627 | x | odl-protocol-framework-0.5.0-Helium-RC2-verify-v201409250627 | OpenDaylight :: Protocol Framework <br/> odl-aaa-authn | 0.1.0-Helium-RC2-verify-v201409250627 | x | odl-aaa-0.1.0-Helium-RC2-verify-v201409250627 | OpenDaylight :: AAA :: Authentication <br/> odl-openflowplugin-southbound | 0.0.3-Helium-RC2-verify-v201409250627 | x | openflowplugin-0.0.3-Helium-RC2-verify-v201409250627 | OpenDaylight :: Openflow Plugin :: SouthBound <br/> odl-openflowplugin-flow-services | 0.0.3-Helium-RC2-verify-v201409250627 | x | openflowplugin-0.0.3-Helium-RC2-verify-v201409250627 | OpenDaylight :: Openflow Plugin :: Flow Services <br/> odl-openflowplugin-drop-test | 0.0.3-Helium-RC2-verify-v201409250627 | x | openflowplugin-0.0.3-Helium-RC2-verify-v201409250627 | OpenDaylight :: Openflow Plugin :: Drop Test <br/> pax-jetty | 8.1.14.v20131031 | x | org.ops4j.pax.web-3.1.0 | Provide Jetty engine support <br/> pax-http | 3.1.0 | x | org.ops4j.pax.web-3.1.0 | Implementation of the OSGI HTTP Service <br/> pax-http-whiteboard | 3.1.0 | x | org.ops4j.pax.web-3.1.0 | Provide HTTP Whiteboard pattern support <br/> pax-war | 3.1.0 | x | org.ops4j.pax.web-3.1.0 | Provide support of a full WebContainer <br/> odl-mdsal-broker | 1.1-Helium-RC2-verify-v201409250627 | x | odl-mdsal-1.1-Helium-RC2-verify-v201409250627 | OpenDaylight :: MDSAL :: Broker</p> <p>Operating System: All<br/> Platform: All</p> L2SWITCH-22 Out of memory error when l2 switch installed Bug Resolved Done Amit Mandke Jan Medved Sat, 27 Sep 2014 05:45:16 +0000 Sun, 28 Sep 2014 22:21:20 +0000 Sun, 28 Sep 2014 22:21:20 +0000 unspecified General 0 5 <p>Attachment log.zip has been added with description: logs</p> <p>Attachment sfc-restconf-test.zip has been added with description: Python utilities to reproduce the bug</p> <p>Attachment bin.tar.gz has been added with description: Mininet flow check scripts</p> <p>L2Switch is allocating new ThreadPoolExecutor (with thread count which equals to number of CPU count) for each data event to process and submit only one new task.</p> <p>L2Switch could do processing in data change callback instead of allocating threadpool.</p> <p>In Jan case there is lot of data change events happening, which could lead to<br/> allocating lot of threads which are unused and only consumes memory.</p> <p>I tested it, and really L2Switch code base allocates new threadpool on each data change event. This threadpools are not shutdown, so they stay in background consuming<br/> resources.</p> <p>The count of allocated threads increases over time (64 nodes topolofy, no traffic) it is 5000 thread started per hour, count is not decreasing. All threads belong to unnamed threadpool.</p> <p>Fixed by: <a href="https://git.opendaylight.org/gerrit/#/c/11629/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/11629/</a></p> <p>Waiting for confirmation from Jan.</p> <p>I do not see the problem anymore.</p> Development External issue ID 2104 External issue URL Rank 0|i01z27: