Operating System: All
Platform: All

Attachment flows_not_installed_connected_to_Leader_Node_SF142.mht has been added with description: Log_file for flows not installed into ovs switches when connected to leader node

Verified with minimal flows .
500 flows with 30 switches . Total = 15000 flows.
Four switches with zero flows and remaining switches each with 500 flows installed.

This actually reminds me: https://bugs.opendaylight.org/show_bug.cgi?id=7770 so maybe they are related. Also rising importance to critical as it really is.

Actually we can track this issue in this test case:

https://jenkins.opendaylight.org/releng/view/CSIT-3node/job/openflowplugin-csit-3node-periodic-bulkomatic-clustering-daily-only-carbon/199/robot/openflowplugin-bulkomatic-clustering-daily.txt/Cluster%20Reconcilliation%20Multi%20DPN/Verify%20Flows%20In%20Switch%20Connected%20To%20Leader/

It looks like the failure is often but not consistently happening.

This issue can be reproduced by just adding 1 flow in follower and then connect 1 switch to leader of inventory config. It is similar but not the same as OPNFLWPLUG-853. Also this issue is only in Carbon while 7770 is in both Carbon and Boron.

This issue seems very inconsistent, when i tried today in my local cluster setup I could not really reproduce

Hi Luis,
Reproduced the issue with less number of flows.

15 switches
Per switch : 1000 flows
Total : 15000 flows.

Given below log file :
https://jenkins.opendaylight.org/sandbox/job/openflowplugin-csit-3node-periodic-bulkomatic-clustering-daily-only-boron/9/robot/report/log.html

Some of the switches were populated with zero flows when connected to Leader Node.
Log snippet:
============

${flows} = [u'0', u'1000', u'1000', u'1000', u'1000', u'1000', u'1000', u'0', u'1000', u'1000', u'1000', u'1000', u'1000', u'1000', u'1000']
Start / End / Elapsed: 20170309 10:07:27.749 / 20170309 10:07:27.750 / 00:00:00.001
10:07:27.750 FAIL Keyword 'MininetKeywords.Check Flows In Mininet' failed after retrying for 1 minute 40 seconds. The last error was: 13000.0 != 15000.0

Right, I tried with 15 switches and 15K flows and it reproduces consistently. It is important to note that with small amount of switches (3 switches) and flows (1K flows) the issue does not show up which means this is kind of perf/scale related issue.

OK, after doing more test this issue also happens with single node so raising priority to blocker because this is an important regression. Also consider this bug could be the same as 7915.

Also you can use the odl_tester.py in int/test repo to generate the flows as alterative of bulkomatic:

python ./odl_tester.py --threads 5 --flows 10000 --fpr 100 --no-delete

Reproduce is as simple as:

1) Start 1 controller
2) Start mininet: sudo mn --controller 'remote,ip=192.168.0.1,port=6633' --topo linear,15
3) Push 10K flows: python ./odl_tester.py --threads 5 --flows 10000 --fpr 100 --no-delete
4) Stop and start mininet
5) check flows in switches: mininet> dpctl dump-aggregate -O OpenFlow13

2 additional notes:

• The issue does not show when I enable FRS as you mention.
• The regression was introduced after Boron SR2 and is present on both Boron and master branches.

Anil's fix for FRM was merged, so can this one be closed now?