[OPNFLWPLUG-621] [Clustering] Unable to provision flows with 30 switches, 1000 flows/switch. (Tried He design only)

Mon, 27 Sep 2021 09:01:43 +0000

Flow provisioning failure in ODL cluster with 30 switches and 1000 flows per switch

Build used :
===================
Karaf distro from latest ODL Beryllium code

Flow provisioning failure in ODL cluster with 30 switches and 1000 flows per switch.

Test Type :
===================
Flow Provisioning failing on connecting 30 switches across 3 cluster nodes , 1000 flows per switch.Total 30,000 flows.

Test setup :
====================
Build used: Beryllium
OF-Plugin Used: Helium
OF-HA used: NO

Objective of test :
===================
verify if the flow provioning is stable when we scale switches per controller.

Test Steps :
============
1. Bring up healthy 3 node cluster say c1, c2 and c3.
2. Bring UP the mininet with following command and connect to controller c1(10.183.181.41),c2(10.183.181.42) and c3(10.183.181.43).c1 is the Leader of inventory-config-shard
Here each controller is connected with 10 switches.so overall 30 switches across the cluster.
3. Now i provision 30000 flows from c2(Follower).

Note:We are provisioning flows via Binding aware Api of Openflow Inventory Model.
4.Check if 30000 flows (1000 flows per switch) have been provisioned across 30 switches.

Commands
========
To connect 10 switches per controller, i use mininet custom command on each controller(c1,c2,c3)
sudo mn --custom /home/mininet/mininet/custom/mytopo.py --topo mytopo --controller remote,ip=10.183.181.41 --switch ovsk,protocols=OpenFlow13
sudo mn --custom /home/mininet/mininet/custom/mytopo.py --topo mytopo --controller remote,ip=10.183.181.42 --switch ovsk,protocols=OpenFlow13
sudo mn --custom /home/mininet/mininet/custom/mytopo.py --topo mytopo --controller remote,ip=10.183.181.43 --switch ovsk,protocols=OpenFlow13

The below command we use to check number of flows provisioned per controller.
dpctl dump-aggregate -O OpenFlow13

Test Results:
=============
1. I could not see 30000 flows(1000 flows per switch) and i get the below Exception (full stack could be seen in the logs)

i could see 30000 flows in the configDS but in the switch i could not see 1000 flows per switch..
Total 29953 flows got provisioned on the switch across the cluster..

root@mininet-vm:/home/mininet/integration/test/tools/odl-mdsal-clustering-tests/clustering-performance-test# ./inventory_crawler.py --auth --host 10.183.181.42 --datastore config
Crawling 'http://10.183.181.42:8181/restconf/config/opendaylight-inventory:nodes'

Totals:
Nodes: 30
Reported flows: 0
Found flows: 30000

root@mininet-vm:/home/mininet/integration/test/tools/odl-mdsal-clustering-tests/clustering-performance-test# ./inventory_crawler.py --auth --host 10.183.181.42 --datastore operational
Crawling 'http://10.183.181.42:8181/restconf/operational/opendaylight-inventory:nodes'

Totals:
Nodes: 30
Reported flows: 30163
Found flows: 29953

mininet> dpctl dump-aggregate -O OpenFlow13

- - s1 ------------------------------------------------------------------------
    OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=997
  - s2 ------------------------------------------------------------------------
    OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=997
  - s3 ------------------------------------------------------------------------
    OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=1000
  - s4 ------------------------------------------------------------------------
    OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=999
  - s5 ------------------------------------------------------------------------
    OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=997
  - s6 ------------------------------------------------------------------------
    OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=999
  - s7 ------------------------------------------------------------------------
    OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=998
  - s8 ------------------------------------------------------------------------
    OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=998
  - s9 ------------------------------------------------------------------------
    OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=998
  - s10 ------------------------------------------------------------------------
    OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=999

mininet> dpctl dump-aggregate -O OpenFlow13

- - s11 ------------------------------------------------------------------------
    OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=1000
  - s12 ------------------------------------------------------------------------
    OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=1000
  - s13 ------------------------------------------------------------------------
    OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=1000
  - s14 ------------------------------------------------------------------------
    OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=1000
  - s15 ------------------------------------------------------------------------
    OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=1000
  - s16 ------------------------------------------------------------------------
    OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=1000
  - s17 ------------------------------------------------------------------------
    OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=1000
  - s18 ------------------------------------------------------------------------
    OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=1000
  - s19 ------------------------------------------------------------------------
    OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=1000
  - s20 ------------------------------------------------------------------------
    OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=1000

mininet> dpctl dump-aggregate -O OpenFlow13

- - s21 ------------------------------------------------------------------------
    OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=1000
  - s22 ------------------------------------------------------------------------
    OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=1000
  - s23 ------------------------------------------------------------------------
    OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=996
  - s24 ------------------------------------------------------------------------
    OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=996
  - s25 ------------------------------------------------------------------------
    OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=996
  - s26 ------------------------------------------------------------------------
    OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=997
  - s27 ------------------------------------------------------------------------
    OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=996
  - s28 ------------------------------------------------------------------------
    OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=996
  - s29 ------------------------------------------------------------------------
    OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=997
  - s30 ------------------------------------------------------------------------
    OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=997

Attaching the logs for 1000 flows.

Note:Out of 10 times, only 1 time i could see 30K flows in ConfigDS,OperDS and in switch(1000 flows per switch).However rest 9 times there is a mismatch in flow count in switch,OperDS and ConfigDS.

Thanks & Regards,
Saibal Roy.

OpenDaylight JIRA

[OPNFLWPLUG-621] [Clustering] Unable to provision flows with 30 switches, 1000 flows/switch. (Tried He design only)