Operating System: Linux
Platform: All

Attachment zero_flows_installed_when_connected_to_Leader_Node_boron_latest.mht has been added with description: Log file for zero flows in 15 switches connected to Leader Node

Is this not duplicate of https://bugs.opendaylight.org/show_bug.cgi?id=7884?

Hi Luis,
This is not duplicate of bug <7884>.
7884 : Some of the switches are not getting installed with flows . Zero flows will be present in few switches. This happens when connected to Follower and Leader Node as well.
Example: 4 out of 15 switches will have zero flows.

7915 : All switches will be having zero flows when connected to Leader Node only. This issue is consistently reproducing.

Regards,
Naveen.

OK, one question: can you reproduce this issue in some local environment or only in ODL jenkins?

Also, now I realize you are using vey high # flows (150K), does the issue reproduces with less number (e.g. 15K)?

Hi Luis,
Executed with less number of flows and given below log file. Issue is getting reproduced when connected to Leader Node.

Flows per switch : 10000
switches: 1
Total flows : 10000

log snippet:
============
Keyword 'MininetKeywords.Check Flows In Mininet' failed after retrying for 4 minutes 10 seconds. The last error was: 0.0 != 10000.0

Build : https://nexus.opendaylight.org/content/repositories/opendaylight.snapshot/org/opendaylight/integration/distribution-karaf/0.5.3-SNAPSHOT/distribution-karaf-0.5.3-20170312.234446-802.zip

Attached Log file for the same.

Regards,
Naveen.

Executed in latest build with single switch with 10k flows. Able to reproduce issue and attached log file.

Attachment 010_small_config_single_switch_10k_flows_SF142_march_13th_2017.mht has been added with description: Latest log with 10000 flows with single switch

Right, this also reproduces easily with 1 switch and 10K flows, I believe this bug is connected with https://bugs.opendaylight.org/show_bug.cgi?id=7884 as both share same scenario of switch reconnecting from no-leader to leader.

I think this could be a duplicate of https://bugs.opendaylight.org/show_bug.cgi?id=7884, therefore blocker too.

Reproduce is as simple as:

1) Start 1 controller
2) Start mininet: sudo mn --controller 'remote,ip=192.168.0.1,port=6633' --topo linear,1
3) Push 10K flows: python ./odl_tester.py --threads 5 --flows 10000 --fpr 100 --no-delete
4) Stop and start mininet
5) check flows in switch: mininet> dpctl dump-aggregate -O OpenFlow13

2 additional notes:

• The issue does not show when I enable FRS.
• The regression was introduced after Boron SR2 and is present on both Boron and master branches.

Hi Luis/Naveen,

I pushed the following patch that fixes this issue

https://git.opendaylight.org/gerrit/53383

Can you please take this patch try to recreate the issue again? I locally tried using the instruction that luis provided and it works for me.

carbon :https://git.opendaylight.org/gerrit/#/c/53521/

Reopen issue for carbon. Carbon fails the following tests for single controller (4 cpus):

10K flows in 1 switch:

1) Start 1 controller
2) Start mininet: sudo mn --controller 'remote,ip=192.168.0.1,port=6633' --topo linear,1
3) Push 10K flows: python ./odl_tester.py --threads 5 --flows 10000 --fpr 100 --no-delete
4) Stop and start mininet
5) check flows in switch: mininet> dpctl dump-aggregate -O OpenFlow13

20K flows in 8 switch:

1) Start 1 controller
2) Start mininet: sudo mn --controller 'remote,ip=192.168.0.1,port=6633' --topo linear,8
3) Push 20K flows: python ./odl_tester.py --threads 5 --flows 20000 --fpr 100 --no-delete
4) Stop and start mininet
5) check flows in switch: mininet> dpctl dump-aggregate -O OpenFlow13

I tested this on this patch: https://git.opendaylight.org/gerrit/#/c/53972/ and it is working. On master it is not, so this issue was probably caused by DeviceFlowRegistry performance regression.

Can you recheck this please Luis after my patches that fixed regression in 6917 was merged?

Yes, this is fixed now.