[#L2SWITCH-22] Out of memory error when l2 switch installed

[L2SWITCH-22] Out of memory error when l2 switch installed Created: 27/Sep/14 Updated: 28/Sep/14 Resolved: 28/Sep/14
Status:	Resolved
Project:	l2switch
Component/s:	General
Affects Version/s:	unspecified
Fix Version/s:	None

Type:

Bug

Reporter:

Jan Medved

Assignee:

Amit Mandke

Resolution:

Done

Votes:

Labels:

None

Remaining Estimate:

Not Specified

Time Spent:

Not Specified

Original Estimate:

Not Specified

Environment:

Operating System: All
Platform: All

Attachments:

bin.tar.gz

log.zip

sfc-restconf-test.zip

External issue ID:

2104

Description

Out of memory error is observed when l2switch is installed and flows are being added to the controller through the REST API. The out of memory condition seems to be related to the PS Survivor space (see attached jconsole snapshot). The exception message observed last was as follows (but it can vary):

objc[2718]: Class JavaLaunchHelper is implemented in both /Library/Java/JavaVirtualMachines/jdk1.7.0_65.jdk/Contents/Home/bin/java and /Library/Java/JavaVirtualMachines/jdk1.7.0_65.jdk/Contents/Home/jre/lib/libinstrument.dylib. One of the two will be used. Which one is undefined.
Exception in thread "DOM-OPER-DCL-77" java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:714)
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1017)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1163)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

To reproduce:
1. Get the RC2 image distribution-karaf-0.2.0-Helium-RC2-verify-v201409250627

2. Install features as shown below (basically, openflow and l2switch)

3. Run the attached python scripts to generate/ clear flows until the
out of memory issue occurs:
3.1 > python flow_config_perf.py --nflows=200 --nthreads=10 --ncycles=10 --no-delete --auth

3.2 Wait until 20000 flows show up in the operational data space. Use:
> python inventory_crawler.py --plevel=2 --auth

3.3 clear the flows from the controller:
> python config_cleanup.py --auth

3.4 Repeat until the out of memory condition occurs

Installed features:

Comments

Comment by Jan Medved [ 27/Sep/14 ]

Attachment log.zip has been added with description: logs

Comment by Jan Medved [ 27/Sep/14 ]

Attachment sfc-restconf-test.zip has been added with description: Python utilities to reproduce the bug

Comment by Jan Medved [ 27/Sep/14 ]

Attachment bin.tar.gz has been added with description: Mininet flow check scripts

Comment by Tony Tkacik [ 27/Sep/14 ]

L2Switch is allocating new ThreadPoolExecutor (with thread count which equals to number of CPU count) for each data event to process and submit only one new task.

L2Switch could do processing in data change callback instead of allocating threadpool.

In Jan case there is lot of data change events happening, which could lead to
allocating lot of threads which are unused and only consumes memory.

Comment by Tony Tkacik [ 27/Sep/14 ]

I tested it, and really L2Switch code base allocates new threadpool on each data change event. This threadpools are not shutdown, so they stay in background consuming
resources.

The count of allocated threads increases over time (64 nodes topolofy, no traffic) it is 5000 thread started per hour, count is not decreasing. All threads belong to unnamed threadpool.

Comment by Colin Dixon [ 28/Sep/14 ]

Fixed by: https://git.opendaylight.org/gerrit/#/c/11629/

Waiting for confirmation from Jan.

Comment by Jan Medved [ 28/Sep/14 ]

I do not see the problem anymore.

Generated at Wed Feb 07 20:05:43 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.

[L2SWITCH-22] Out of memory error when l2 switch installed Created: 27/Sep/14 Updated: 28/Sep/14 Resolved: 28/Sep/14