[OPNFLWPLUG-281] Deleting of flows very slow with large number of flows in data store and controller connected to the network

Mon, 27 Sep 2021 09:01:20 +0000

When a large number of flows is inserted into the data store, individual flow delete operations become very slow (rate: 3-5 flows/sec). Flow commit statistics indicate average commit time of 200 ms, the longest commit time is about 17.5 seconds. No commits other than flow deletes are going to the data store. The CPU utilization is 10-13%.

This problem only occurs when the controller is connected to the mininet network. The size of the mininet network does not seem to matter, it occurs the same with 3 and 7 node network.

The reason this issue is raised against md-sal is that it seems to be a system issue (livelock?) that manifests itself in drastically prolonged IMDS commit times.

To reproduce:
1. Start the controller (karaf edition).Attach jconsole a

2. Connect mininet, e.g.:
> sudo mn --controller=remote,ip=192.168.162.1:6653 --topo tree,3 --switch ovsk,protocols=OpenFlow13

3. Use the attached script with the following parameters:
>python flow_config_perf.py --nflows 1000 --nthreads=5 --ncycles 10 --delay=60
This will insert 50k flows into the controller, wait for 60 seconds and then
start deleting the inserting flows.

4. While the controller is pausing for 60 second before starting the delete cycle,
clear the commit stats in jconsole and check with mininet that all flows have
been inserted into the network.

5. Observe that the delete goes very slowly. Check the commit stats (I observed
average commit rate of ~200 ms, which corresponds to 5 transactions per second.
I could see that flows were being deleted at a rate of about 5 flows per second)

OpenDaylight JIRA

[OPNFLWPLUG-281] Deleting of flows very slow with large number of flows in data store and controller connected to the network