[Clustering] Unable to provision flows with 30 switches, 1000 flows/switch. (Tried He design only)

Flow provisioning failure in ODL cluster with 30 switches and 1000 flows per switch

Build used :
===================
Karaf distro from latest ODL Beryllium code

Flow provisioning failure in ODL cluster with 30 switches and 1000 flows per switch.

Test Type :
===================
Flow Provisioning failing on connecting 30 switches across 3 cluster nodes , 1000 flows per switch.Total 30,000 flows.

Test setup :
====================
Build used: Beryllium
OF-Plugin Used: Helium
OF-HA used: NO

Objective of test :
===================
verify if the flow provioning is stable when we scale switches per controller.

Test Steps :
============
1. Bring up healthy 3 node cluster say c1, c2 and c3.
2. Bring UP the mininet with following command and connect to controller c1(10.183.181.41),c2(10.183.181.42) and c3(10.183.181.43).c1 is the Leader of inventory-config-shard
Here each controller is connected with 10 switches.so overall 30 switches across the cluster.
3. Now i provision 30000 flows from c2(Follower).

Note:We are provisioning flows via Binding aware Api of Openflow Inventory Model.
4.Check if 30000 flows (1000 flows per switch) have been provisioned across 30 switches.

Commands
========
To connect 10 switches per controller, i use mininet custom command on each controller(c1,c2,c3)
sudo mn --custom /home/mininet/mininet/custom/mytopo.py --topo mytopo --controller remote,ip=10.183.181.41 --switch ovsk,protocols=OpenFlow13
sudo mn --custom /home/mininet/mininet/custom/mytopo.py --topo mytopo --controller remote,ip=10.183.181.42 --switch ovsk,protocols=OpenFlow13
sudo mn --custom /home/mininet/mininet/custom/mytopo.py --topo mytopo --controller remote,ip=10.183.181.43 --switch ovsk,protocols=OpenFlow13

The below command we use to check number of flows provisioned per controller.
dpctl dump-aggregate -O OpenFlow13

Test Results:
=============
1. I could not see 30000 flows(1000 flows per switch) and i get the below Exception (full stack could be seen in the logs)

i could see 30000 flows in the configDS but in the switch i could not see 1000 flows per switch..
Total 29953 flows got provisioned on the switch across the cluster..

root@mininet-vm:/home/mininet/integration/test/tools/odl-mdsal-clustering-tests/clustering-performance-test# ./inventory_crawler.py --auth --host 10.183.181.42 --datastore config
Crawling 'http://10.183.181.42:8181/restconf/config/opendaylight-inventory:nodes'

Totals:
Nodes: 30
Reported flows: 0
Found flows: 30000

root@mininet-vm:/home/mininet/integration/test/tools/odl-mdsal-clustering-tests/clustering-performance-test# ./inventory_crawler.py --auth --host 10.183.181.42 --datastore operational
Crawling 'http://10.183.181.42:8181/restconf/operational/opendaylight-inventory:nodes'

Totals:
Nodes: 30
Reported flows: 30163
Found flows: 29953

mininet> dpctl dump-aggregate -O OpenFlow13

• • • s1 ------------------------------------------------------------------------
      OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=997
    • s2 ------------------------------------------------------------------------
      OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=997
    • s3 ------------------------------------------------------------------------
      OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=1000
    • s4 ------------------------------------------------------------------------
      OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=999
    • s5 ------------------------------------------------------------------------
      OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=997
    • s6 ------------------------------------------------------------------------
      OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=999
    • s7 ------------------------------------------------------------------------
      OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=998
    • s8 ------------------------------------------------------------------------
      OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=998
    • s9 ------------------------------------------------------------------------
      OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=998
    • s10 ------------------------------------------------------------------------
      OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=999

mininet> dpctl dump-aggregate -O OpenFlow13

• • • s11 ------------------------------------------------------------------------
      OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=1000
    • s12 ------------------------------------------------------------------------
      OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=1000
    • s13 ------------------------------------------------------------------------
      OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=1000
    • s14 ------------------------------------------------------------------------
      OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=1000
    • s15 ------------------------------------------------------------------------
      OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=1000
    • s16 ------------------------------------------------------------------------
      OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=1000
    • s17 ------------------------------------------------------------------------
      OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=1000
    • s18 ------------------------------------------------------------------------
      OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=1000
    • s19 ------------------------------------------------------------------------
      OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=1000
    • s20 ------------------------------------------------------------------------
      OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=1000

mininet> dpctl dump-aggregate -O OpenFlow13

• • • s21 ------------------------------------------------------------------------
      OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=1000
    • s22 ------------------------------------------------------------------------
      OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=1000
    • s23 ------------------------------------------------------------------------
      OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=996
    • s24 ------------------------------------------------------------------------
      OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=996
    • s25 ------------------------------------------------------------------------
      OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=996
    • s26 ------------------------------------------------------------------------
      OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=997
    • s27 ------------------------------------------------------------------------
      OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=996
    • s28 ------------------------------------------------------------------------
      OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=996
    • s29 ------------------------------------------------------------------------
      OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=997
    • s30 ------------------------------------------------------------------------
      OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=997

Attaching the logs for 1000 flows.

Note:Out of 10 times, only 1 time i could see 30K flows in ConfigDS,OperDS and in switch(1000 flows per switch).However rest 9 times there is a mismatch in flow count in switch,OperDS and ConfigDS.

Thanks & Regards,
Saibal Roy.