https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/OPNFLWPLUG-607 OpenFlowPlugin <p>Build used : <br/> ===================<br/> Karaf distro from latest ODL Beryllium code</p> <p>Unrecoverable flow provisioning failure in ODL cluster with 30 switches and 1000 flows per switch.</p> <p>Test Type : <br/> ===================<br/> Flow Provisioning failing on connecting 30 switches across 3 cluster nodes , 1000 flows per switch.Total 30,000 flows.</p> <p>Test setup :<br/> ====================<br/> Build used: Beryllium<br/> OF-Plugin Used: Helium<br/> OF-HA used: NO</p> <p>Objective of test : <br/> ===================<br/> verify if the flow provioning is stable when we scale switches per controller.</p> <p>Test Steps :<br/> ============<br/> 1. Bring up healthy 3 node cluster say c1, c2 and c3.<br/> 2. Bring UP the mininet with following command and connect to controller c1(10.183.181.41),c2(10.183.181.42) and c3(10.183.181.43).c2 is the Leader of inventory-config-shard<br/> Here each controller is connected with 10 switches.so overall 30 switches across the cluster.<br/> 3. Now i provision 30000 flows from c1(Follower).</p> <p>Note:We are provisioning flows via Binding aware Api of Openflow Inventory Model.<br/> 4.Check if 30000 flows (1000 flows per switch) have been provisioned across 30 switches.</p> <p>Commands<br/> ========<br/> To connect 10 switches per controller, i use mininet custom command on each controller(c1,c2,c3)<br/> sudo mn --custom /home/mininet/mininet/custom/mytopo.py --topo mytopo --controller remote,ip=10.183.181.41 --switch ovsk,protocols=OpenFlow13<br/> sudo mn --custom /home/mininet/mininet/custom/mytopo.py --topo mytopo --controller remote,ip=10.183.181.42 --switch ovsk,protocols=OpenFlow13<br/> sudo mn --custom /home/mininet/mininet/custom/mytopo.py --topo mytopo --controller remote,ip=10.183.181.43 --switch ovsk,protocols=OpenFlow13</p> <p>Note:mytopo.py is attached for quick reference.please modify mytopo.py accordingly</p> <p>The below command we use to check number of flows provisioned per controller.<br/> dpctl dump-aggregate -O OpenFlow13</p> <p>Test Results:<br/> =============<br/> 1. I could not see 30000 flows(1000 flows per switch) and i get the below Exception (full stack could be seen in the logs)<br/> Caused by: akka.pattern.AskTimeoutException: Ask timed out on [ActorSelection<span class="error">&#91;Anchor(akka.tcp://opendaylight-cluster-data@10.183.181.41:2550/), Path(/user/shardmanager-operational/member-1-shard-inventory-operational/shard-member-2-chn-157-txn-1#1338207586)&#93;</span>] after <span class="error">&#91;5000 ms&#93;</span></p> <p>2. After above condition, no further flow provisioning via config DS works and system requires complete cluster reboot to restore normalcy.</p> <p>Attaching all the karaf logs of 3 controller nodes. </p> <p>Thanks &amp; Regards,<br/> Saibal Roy.</p> <p>Operating System: All<br/> Platform: All</p> OPNFLWPLUG-607

[Clustering]: Unrecoverable cluster flow provisioning failure with 30 switches, 1000 flows/switch. (Tried He design only)

Bug Resolved Done Anil Vishnoi Saibal Roy Wed, 27 Jan 2016 11:11:23 +0000 Mon, 27 Sep 2021 09:01:42 +0000 Tue, 16 Feb 2016 11:40:37 +0000 General 0 5 <p>Attachment logs.zip has been added with description: Unrecoverable flow provisioning failure in ODL cluster with 30 switches and 1000 flows per switch</p> <p>Binding aware stub app follows typical ODL stereotype of <br/> Config DS update --&gt; FRM --&gt; OpenflowPlugin --&gt; Flow provisioning --&gt; Stats --&gt; Oper DS Update. </p> <p>Statistics was in enabled state as the evaluation is for "as is" functionality</p> <p>From the logs it appears that AskTimeoutException originates from extensive Oper DS updates from the plugin. Eventually, this can start having ripple effect on the Config DS updates too</p> <p>Hi,</p> <p>I connected 30 switches(10 switch per controller) and then i pushed 750 flows from follower c2.Note my leader is c1.<br/> I could see 30*750=22500 flows across the cluster.</p> <p>Now i increase the flow to 850 and i provision from follower c2 and i am not able to see all the flows in my switch across the cluster..It gives AskTimeoutException.</p> <p>Thanks &amp; Regards,<br/> Saibal Roy.</p> <p>Muthu will provide an update on this by Feb 3 or 4.</p> <p>Hi,</p> <p>In continuation of further troubleshooting following exercise was carried out. </p> <p>Objective:<br/> ==========<br/> To identify if AskTimeout exception surfaces only when both ConfigDS and OperDS are being updated concurrently in flow-provisioning. </p> <p>i did 2 flavors of testing with 10 switch per controller(Total 30 switches) and pushing 1000 flows per switch.Below are my observations.</p> <p>Test Case1:<br/> ========<br/> 1. c1,c2,c3 are the controller where c1 is the leader.<br/> 2. Connected 10 switches per controller(Total 30 switches)<br/> 3. I provision 1000 flows per switch(total 3*1000=3000 flows) from Follower c2<br/> 4. i could see that till 760 flows get provision per switch and then i get AskTimeOutException.</p> <p>Test Case2:<br/> ========<br/> 1. c1,c2,c3 are the controller where c1 is the leader.<br/> 2. I provision 1000 flows per switch(total 3*1000=3000 flows) from Follower c2<br/> 3. i could see 30000 flows in config DS.<br/> 4. Now i Connect 10 switches per controller(Total 30 switches)<br/> 5. i could see 1000 flows per switch(total 30000 flows).</p> <p>Observation :<br/> =============<br/> It appears that standard flow-provisioning (which updates Config DS as well as Oper DS concurrently) results in AskTimeout exception. Whereas in case of <br/> Reconciliation, Config DS update (30K flows) and Oper DS updates (updates of flows based on Statistics) are staggered across , we perhaps do not see this issue</p> <p>In the 1st TestCase when we are pushing 30K flows, it actually does 2x transaction where as in the 2nd TestCase, first 30K flows are pushed to config DS and then switch are connected where again another 30K transaction is pushed.Is this the reason for failure of TestCase1??</p> <p>Thanks &amp; Regards,<br/> Saibal Roy.</p> <p>Hi,<br/> Note:We are using the latest build and still the BUG exists as Test Case:1 fails to provision 30K flows across the cluster. </p> <p>Thanks &amp; Regards,<br/> Saibal Roy</p> <p>Hi Saibal,</p> <p>I tried the same test with with the latest stable/beryllium and things looks good to me. </p> <p><a href="https://git.opendaylight.org/gerrit/#/c/34115/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/34115/</a> this patch get rid of routed rpc use and use the local rpc registration. Although i didn't install equal number of flows on each switch but distribution is approximately even. This is what i did</p> <p>1) Started 3-node cluster (used feature :odl-openflowplugin-flow-service-rest)<br/> 2) Connect 10 switch to each controller<br/> 3) Installed 50K flow on all the connected switch using flow_config_blaster.py script from integration/test repo.<br/> sudo ./flow_config_blaster.py --host 10.0.0.3 --threads 5 --flows 10000 --auth --no-delete --fpr 200 --nodes 30</p> <p>4) Flow installed done through the inventory-config shard follower.<br/> 5) I can see that all 50K flows get installed successfully, following is the distribution of the flow on each switch</p> <ol> <li>./get-total-found.sh<br/> Switch s1: 1600 flows<br/> Switch s2: 800 flows<br/> Switch s3: 2200 flows<br/> Switch s4: 2200 flows<br/> Switch s5: 1200 flows<br/> Switch s6: 1800 flows<br/> Switch s7: 1400 flows<br/> Switch s8: 1600 flows<br/> Switch s9: 1800 flows<br/> Switch s10: 600 flows<br/> Switch s11: 2200 flows<br/> Switch s12: 1400 flows<br/> Switch s13: 1200 flows<br/> Switch s14: 2200 flows<br/> Switch s15: 2200 flows<br/> Switch s16: 1400 flows<br/> Switch s17: 2000 flows<br/> Switch s18: 1000 flows<br/> Switch s19: 1800 flows<br/> Switch s20: 2400 flows<br/> Switch s21: 2000 flows<br/> Switch s22: 2800 flows<br/> Switch s23: 1800 flows<br/> Switch s24: 1400 flows<br/> Switch s25: 1200 flows<br/> Switch s26: 2000 flows<br/> Switch s27: 1200 flows<br/> Switch s28: 1800 flows<br/> Switch s29: 1000 flows<br/> Switch s30: 1800 flows</li> </ol> <p>Total: 50000</p> <p>Can you please test it again in your environment and update the bug.</p> <p>Attachment logs.zip has been added with description: logs</p> <p>Hi Anil,<br/> In continuation of further testing following exercise was carried out.</p> <p>1. c1,c2,c3 are the controller where c2 is the leader.<br/> 2. Connected 10 switches per controller(Total 30 switches)<br/> 3. I provision 1000 flows per switch(total 30*1000=30000 flows) from Follower c1</p> <p>i could see 30000 flows in the configDS but in the switch i could not see 1000 flows per switch..<br/> Total 29953 flows got provisioned on the switch across the cluster..</p> <p>root@mininet-vm:/home/mininet/integration/test/tools/odl-mdsal-clustering-tests/clustering-performance-test# ./inventory_crawler.py --auth --host 10.183.181.42 --datastore config<br/> Crawling 'http://10.183.181.42:8181/restconf/config/opendaylight-inventory:nodes'</p> <p>Totals:<br/> Nodes: 30<br/> Reported flows: 0<br/> Found flows: 30000</p> <p>root@mininet-vm:/home/mininet/integration/test/tools/odl-mdsal-clustering-tests/clustering-performance-test# ./inventory_crawler.py --auth --host 10.183.181.42 --datastore operational<br/> Crawling 'http://10.183.181.42:8181/restconf/operational/opendaylight-inventory:nodes'</p> <p>Totals:<br/> Nodes: 30<br/> Reported flows: 30163<br/> Found flows: 29953</p> <p>mininet&gt; dpctl dump-aggregate -O OpenFlow13</p> <ul> <li> <ul> <li> <ul> <li>s1 ------------------------------------------------------------------------<br/> OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=997</li> <li>s2 ------------------------------------------------------------------------<br/> OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=997</li> <li>s3 ------------------------------------------------------------------------<br/> OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=1000</li> <li>s4 ------------------------------------------------------------------------<br/> OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=999</li> <li>s5 ------------------------------------------------------------------------<br/> OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=997</li> <li>s6 ------------------------------------------------------------------------<br/> OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=999</li> <li>s7 ------------------------------------------------------------------------<br/> OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=998</li> <li>s8 ------------------------------------------------------------------------<br/> OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=998</li> <li>s9 ------------------------------------------------------------------------<br/> OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=998</li> <li>s10 ------------------------------------------------------------------------<br/> OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=999</li> </ul> </li> </ul> </li> </ul> <p>mininet&gt; dpctl dump-aggregate -O OpenFlow13</p> <ul> <li> <ul> <li> <ul> <li>s11 ------------------------------------------------------------------------<br/> OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=1000</li> <li>s12 ------------------------------------------------------------------------<br/> OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=1000</li> <li>s13 ------------------------------------------------------------------------<br/> OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=1000</li> <li>s14 ------------------------------------------------------------------------<br/> OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=1000</li> <li>s15 ------------------------------------------------------------------------<br/> OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=1000</li> <li>s16 ------------------------------------------------------------------------<br/> OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=1000</li> <li>s17 ------------------------------------------------------------------------<br/> OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=1000</li> <li>s18 ------------------------------------------------------------------------<br/> OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=1000</li> <li>s19 ------------------------------------------------------------------------<br/> OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=1000</li> <li>s20 ------------------------------------------------------------------------<br/> OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=1000</li> </ul> </li> </ul> </li> </ul> <p>mininet&gt; dpctl dump-aggregate -O OpenFlow13</p> <ul> <li> <ul> <li> <ul> <li>s21 ------------------------------------------------------------------------<br/> OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=1000</li> <li>s22 ------------------------------------------------------------------------<br/> OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=1000</li> <li>s23 ------------------------------------------------------------------------<br/> OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=996</li> <li>s24 ------------------------------------------------------------------------<br/> OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=996</li> <li>s25 ------------------------------------------------------------------------<br/> OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=996</li> <li>s26 ------------------------------------------------------------------------<br/> OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=997</li> <li>s27 ------------------------------------------------------------------------<br/> OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=996</li> <li>s28 ------------------------------------------------------------------------<br/> OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=996</li> <li>s29 ------------------------------------------------------------------------<br/> OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=997</li> <li>s30 ------------------------------------------------------------------------<br/> OFPST_AGGREGATE reply (OF1.3) (xid=0x2): packet_count=0 byte_count=0 flow_count=997</li> </ul> </li> </ul> </li> </ul> <p>The above states that the BUG exist as we are not able to see 1000 flows per switch.</p> <p>There has been a definite improvement in the flow provisioning behaviour as we are able to push ~29.5K flows in the switch.<br/> Thanks for your help!!</p> <p>Attaching the logs.</p> <p>Thanks &amp; Regards,<br/> Saibal Roy.</p> <p>Hi Saibal,</p> <p>How long do you wait before dumping all the flows in the switches? Also what's the size of your VM that's running controller ? Just want to make sure that controller has enough computing power. Can you give some more details about how your application is writing flow ?</p> <p>I tested right now with 50K flows and i see that things are working fine, all the flows are getting installed properly.</p> <p>Hi Anil,<br/> Below are the details.</p> <p>1. How long do you wait before dumping all the flows in the switches? </p> <p>Once i connect the switches and i check all the switches are UP on the respective controller, i wait for a minute or so..then i push 30K flows.</p> <p>Once i push 30k flows from the Follower,i check the configDS how many flows got provisioned followed by checking in the switch and then in the OperDS.</p> <p>Once all the flows are provisioned in the configDS(30K) , I keep on checking for next 30-35 minutes so that all the flows gets provisioned in the OperDS as well and i check in the switch as well.</p> <p>2.Also what's the size of your VM that's running controller ? </p> <p>Each VM is of 20GB size.<br/> We are using 6VCPUs for each controller.<br/> Heap Size alloted is 8GB.</p> <p>3.Can you give some more details about how your application is writing flow ?</p> <p>The application used for writing flows pushes the flows directly to the config DS instead of pushing through RESTconf. <br/> One thread per dpn is used and the flows are pushed sequentially in each thread.</p> <p>Flows are written to Config DS using bindingaware stub application. It uses standard databroker and WriteTransaction with put calls.</p> <p>Hi,<br/> With further testing with detailed logs implemented, we didn't see the issue of cluster Datastore going into Unrecoverable state . </p> <p>But the issue of Flow provisioning still exists.Hence we are closing this BUG and raising another BUG.<br/> Below of the details of the new BUG.</p> <p><a href="https://bugs.opendaylight.org/show_bug.cgi?id=5364" class="external-link" target="_blank" rel="nofollow noopener">https://bugs.opendaylight.org/show_bug.cgi?id=5364</a></p> <p>Thanks &amp; Regards,<br/> Saibal Roy.</p> Development External issue ID 5114 External issue URL Priority Rank 0|i03273: