[OPNFLWPLUG-621] [Clustering] Unable to provision flows with 30 switches, 1000 flows/switch. (Tried He design only) Created: 16/Feb/16  Updated: 27/Sep/21  Resolved: 29/Jul/16

Status: Resolved
Project: OpenFlowPlugin
Component/s: General
Affects Version/s: None
Fix Version/s: None

Type: Bug
Reporter: Saibal Roy Assignee: Unassigned
Resolution: Won't Do Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: All
Platform: All

Attachments: Zip Archive logs.zip    
External issue ID: 5364
Priority: Highest

 Description   

Flow provisioning failure in ODL cluster with 30 switches and 1000 flows per switch

Build used :
===================
Karaf distro from latest ODL Beryllium code

Flow provisioning failure in ODL cluster with 30 switches and 1000 flows per switch.

Test Type :
===================
Flow Provisioning failing on connecting 30 switches across 3 cluster nodes , 1000 flows per switch.Total 30,000 flows.

Test setup :
====================
Build used: Beryllium
OF-Plugin Used: Helium
OF-HA used: NO

Objective of test :
===================
verify if the flow provioning is stable when we scale switches per controller.

Test Steps :
============
1. Bring up healthy 3 node cluster say c1, c2 and c3.
2. Bring UP the mininet with following command and connect to controller c1(10.183.181.41),c2(10.183.181.42) and c3(10.183.181.43).c1 is the Leader of inventory-config-shard
Here each controller is connected with 10 switches.so overall 30 switches across the cluster.
3. Now i provision 30000 flows from c2(Follower).

Note:We are provisioning flows via Binding aware Api of Openflow Inventory Model.
4.Check if 30000 flows (1000 flows per switch) have been provisioned across 30 switches.

Commands
========
To connect 10 switches per controller, i use mininet custom command on each controller(c1,c2,c3)
sudo mn --custom /home/mininet/mininet/custom/mytopo.py --topo mytopo --controller remote,ip=10.183.181.41 --switch ovsk,protocols=OpenFlow13
sudo mn --custom /home/mininet/mininet/custom/mytopo.py --topo mytopo --controller remote,ip=10.183.181.42 --switch ovsk,protocols=OpenFlow13
sudo mn --custom /home/mininet/mininet/custom/mytopo.py --topo mytopo --controller remote,ip=10.183.181.43 --switch ovsk,protocols=OpenFlow13

The below command we use to check number of flows provisioned per controller.
dpctl dump-aggregate -O OpenFlow13

Test Results:
=============
1. I could not see 30000 flows(1000 flows per switch) and i get the below Exception (full stack could be seen in the logs)

i could see 30000 flows in the configDS but in the switch i could not see 1000 flows per switch..
Total 29953 flows got provisioned on the switch across the cluster..

root@mininet-vm:/home/mininet/integration/test/tools/odl-mdsal-clustering-tests/clustering-performance-test# ./inventory_crawler.py --auth --host 10.183.181.42 --datastore config
Crawling 'http://10.183.181.42:8181/restconf/config/opendaylight-inventory:nodes'

Totals:
Nodes: 30
Reported flows: 0
Found flows: 30000

root@mininet-vm:/home/mininet/integration/test/tools/odl-mdsal-clustering-tests/clustering-performance-test# ./inventory_crawler.py --auth --host 10.183.181.42 --datastore operational
Crawling 'http://10.183.181.42:8181/restconf/operational/opendaylight-inventory:nodes'

Totals:
Nodes: 30
Reported flows: 30163
Found flows: 29953

mininet> dpctl dump-aggregate -O OpenFlow13

      • s1 ------------------------------------------------------------------------
        OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=997
      • s2 ------------------------------------------------------------------------
        OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=997
      • s3 ------------------------------------------------------------------------
        OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=1000
      • s4 ------------------------------------------------------------------------
        OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=999
      • s5 ------------------------------------------------------------------------
        OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=997
      • s6 ------------------------------------------------------------------------
        OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=999
      • s7 ------------------------------------------------------------------------
        OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=998
      • s8 ------------------------------------------------------------------------
        OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=998
      • s9 ------------------------------------------------------------------------
        OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=998
      • s10 ------------------------------------------------------------------------
        OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=999

mininet> dpctl dump-aggregate -O OpenFlow13

      • s11 ------------------------------------------------------------------------
        OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=1000
      • s12 ------------------------------------------------------------------------
        OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=1000
      • s13 ------------------------------------------------------------------------
        OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=1000
      • s14 ------------------------------------------------------------------------
        OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=1000
      • s15 ------------------------------------------------------------------------
        OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=1000
      • s16 ------------------------------------------------------------------------
        OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=1000
      • s17 ------------------------------------------------------------------------
        OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=1000
      • s18 ------------------------------------------------------------------------
        OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=1000
      • s19 ------------------------------------------------------------------------
        OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=1000
      • s20 ------------------------------------------------------------------------
        OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=1000

mininet> dpctl dump-aggregate -O OpenFlow13

      • s21 ------------------------------------------------------------------------
        OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=1000
      • s22 ------------------------------------------------------------------------
        OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=1000
      • s23 ------------------------------------------------------------------------
        OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=996
      • s24 ------------------------------------------------------------------------
        OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=996
      • s25 ------------------------------------------------------------------------
        OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=996
      • s26 ------------------------------------------------------------------------
        OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=997
      • s27 ------------------------------------------------------------------------
        OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=996
      • s28 ------------------------------------------------------------------------
        OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=996
      • s29 ------------------------------------------------------------------------
        OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=997
      • s30 ------------------------------------------------------------------------
        OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=997

Attaching the logs for 1000 flows.

Note:Out of 10 times, only 1 time i could see 30K flows in ConfigDS,OperDS and in switch(1000 flows per switch).However rest 9 times there is a mismatch in flow count in switch,OperDS and ConfigDS.

Thanks & Regards,
Saibal Roy.

 Comments   
Comment by Saibal Roy [ 16/Feb/16 ]

Attachment logs.zip has been added with description: flow provisioning fails for 30 switch,1000 flows per switch.

Comment by Muthukumaran Kothandaraman [ 29/Jul/16 ]

Not seen in the latest Beryllium builds

Generated at Wed Feb 07 20:32:57 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.