[OPNFLWPLUG-683] Slow flow addition to operational datastore during 100k flow perf test using Beryllium-SR1 odl-openflowplugin-flow-services-ui

Mon, 27 Sep 2021 09:01:48 +0000

During testing performance of the OpenFlow Plugin using 100k flows we noticed flow addition takes 300 to 700 seconds to show up in operational datastore using Beryllium-SR1 odl-openflowplugin-flow-services-ui. This is a regression from Beryllium release where the numbers were 50 to 150 seconds.

Reproduction Steps:
1. Setup Controller and install odl-openflowplugin-flow-services-ui:
feature:install odl-openflowplugin-flow-services-ui

2. Start mininet and connect to controller:
sudo mn --controller=remote,ip=127.0.0.1,port=6653 --topo tree,4

3. Open a new shell and run the int/test repo tools/odl-mdsal-clustering-tests/clustering-performance-test/odl_tester.py perf script sending one flow per REST request:
pypy ./odl_tester.py --flows 100000 --threads 5 --timeout 10

The test first starts 5 threads, where each thread programs (adds) 20,000 flows using OpenDaylight authenticated REST API. The tool then waits 10 seconds and deletes all 100k flows.

4. Where mininet is started check flows:
./get-total-found.sh
Switch s1: 6706 flows
Switch s2: 6632 flows
Switch s3: 6604 flows
Switch s4: 6716 flows
Switch s5: 6632 flows
Switch s6: 6574 flows
Switch s7: 6663 flows
Switch s8: 6628 flows
Switch s9: 6809 flows
Switch s10: 6592 flows
Switch s11: 6826 flows
Switch s12: 6607 flows
Switch s13: 6657 flows
Switch s14: 6597 flows
Switch s15: 6757 flows

Total: 100000

5. Wait for test to finish and produce report:
---------------
pypy ./odl_tester.py --flows 100000 --threads 5 --timeout 10

Namespace(bulk_delete=False, flows=100000, fpr=1, host='127.0.0.1', no_delete=False, port='8181', threads=5, timeout=10)
BASELINE:
devices: 15
flows : 0
Added 100000 flows in 45.6335361004 seconds

{204: 100000}

Stats monitoring ...
Reported Flows: 13507/100000
Reported Flows: 13507/100000
Reported Flows: 13507/100000
Reported Flows: 13507/100000
Reported Flows: 13507/100000
Reported Flows: 13507/100000
Reported Flows: 13507/100000
Reported Flows: 13507/100000
Reported Flows: 13507/100000
Reported Flows: 13507/100000
Reported Flows: 13507/100000
Reported Flows: 13507/100000
Reported Flows: 13507/100000
Reported Flows: 13507/100000
Reported Flows: 13507/100000
Reported Flows: 13507/100000
Reported Flows: 13507/100000
Reported Flows: 13507/100000
...
Reported Flows: 68092/100000
Reported Flows: 68092/100000
Reported Flows: 68092/100000
Reported Flows: 68092/100000
Reported Flows: 68092/100000
Reported Flows: 68092/100000
Reported Flows: 68092/100000
Reported Flows: 85538/100000
Reported Flows: 85538/100000
Reported Flows: 85538/100000
Reported Flows: 85538/100000
Reported Flows: 85538/100000
Reported Flows: 85538/100000
Reported Flows: 85538/100000
Reported Flows: 85538/100000
Reported Flows: 85538/100000
Reported Flows: 85538/100000
Reported Flows: 85538/100000
Reported Flows: 85538/100000
Reported Flows: 85538/100000
Reported Flows: 85538/100000
Reported Flows: 85538/100000
Reported Flows: 100000/100000
... monitoring finished in +758 seconds

Flows to be removed: 100000
Removed 100000 flows in 37.0994989872 seconds

{200: 100000}

Stats monitoring ...
Reported Flows: 19/0
Reported Flows: 19/0
Reported Flows: 0/0
... monitoring finished in +2 seconds

addrate 2191.37083263
delrate 2695.45419022

---------------

Note 758 second flow monitoring time above.
This is a regression from Beryllium release where the numbers were 50 to 150 seconds for flow monitoring.

OpenDaylight JIRA

[OPNFLWPLUG-683] Slow flow addition to operational datastore during 100k flow perf test using Beryllium-SR1 odl-openflowplugin-flow-services-ui