[OPNFLWPLUG-539] memory leak with single switch connected having a lot of flows (50k)

Mon, 27 Sep 2021 09:01:38 +0000

This is occurring when using the karaf feature "odl-openflowplugin-flow-services-ui", but NOT coming when using the re-designed plugin "old-openflowplugin-flow-services-ui-li".

Using the Lithium release bits.
https://nexus.opendaylight.org/content/groups/public/org/opendaylight/integration/distribution-karaf/0.3.0-Lithium/distribution-karaf-0.3.0-Lithium.zip

Steps to recreate:

start karaf and install odl-openflowplugin-flow-services-ui

connect one OVS switch (ovs version 2.0.2 was used when creating this bug)

configure 50k flows via NB REST API.
I used our robot test [0] to do this, which in turn calls a python script [1], but
it's possible that any method to configure 50k flows will reproduce this. If you
want to try the robot method, edit the robot script to have "FATAL ERROR" after
line 44 which configures the flows. That forces robot to quit after installing the
flows. The pybot command line that I used is here [2]

wait.

after some time (maybe many hours) you will see messages like this on the karaf console:

java.lang.OutOfMemoryError: Java heap space
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.growArray(ForkJoinPool.java:1090)
Exception in thread "Timer-0"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Timer-0"
Exception in thread "qtp1473670433-83" Exception in thread "Karaf Console Local for user karaf" Exception in thread "qtp1473670433-80" at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1978)
Exception in thread "pool-38-thread-1" Exception in thread "qtp1473670433-1550" Exception in thread "qtp1473670433-85" Exception in thread "RMI TCP Connection(idle)" Exception in thread "sshd-SshServer[27195dd9]-timer-thread-1" Exception in thread "pool-34-thread-1" Exception in thread "RMI TCP Connection(idle)" Exception in thread "qtp1473670433-81" at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Exception in thread "RMI TCP Connection(idle)" java.lang.OutOfMemoryError: Java heap space
Exception in thread "Timer-1" Exception in thread "RMI TCP Connection(idle)" Exception in thread "RMI TCP Connection(idle)" java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space

I neglected to collect the karaf.log, but can do so if it's needed.

[0] https://github.com/opendaylight/integration/blob/master/test/csit/suites/openflowplugin/Performance_Flow_Config/070_Stats_Collection_100k_WithMininetRestart.robot
[1] https://github.com/opendaylight/integration/blob/master/test/csit/libraries/ScaleClient.py
[2] pybot -d /tmp/RobotResults/ -c critical -e exclude -v BUNDLEFOLDER:distribution-karaf-0.4.0-SNAPSHOT -v WORKSPACE:/tmp -v NEXUSURL_PREFIX:http://nexus01.dfw.opendaylight.org:8081/nexus -v CONTROLLER:<your controller ip> -v CONTROLLER_USER:fedora -v MININET:<your mininet vm ip> -v MININET1: -v MININET2: -v MININET_USER:ubuntu -v DEFAULT_LINUX_PROMPT:\$ -v USER_HOME:/home/jluhrsen -L TRACE Performance_Flow_Config/070_Stats_Collection_100k_WithMininetRestart.robot

OpenDaylight JIRA

[OPNFLWPLUG-539] memory leak with single switch connected having a lot of flows (50k)