https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/OPNFLWPLUG-504 OpenFlowPlugin <p>used odl: distribution-karaf-0.3.0-Lithium-RC1-v201506160017.tar.gz</p> <p>installed feature: odl-openflowplugin-flowservices-ui-li</p> <p>when installed 80k flows, they were not collected, but after mininet restart all flow were collected well</p> <p>then after some time 4 switches were lost from inventory</p> <p>(1434487760.13342, 63, 74567, 74120)<br/> (1434487792.004231, 63, 79567, 79000)<br/> (1434487821.56991, 63, 79567, 79000) HERE ONLY 79k of 80k COLLECTED<br/> (1434487850.850809, 63, 79567, 79000)<br/> (1434487881.234324, 63, 79567, 79000) <br/> Could not retrieve inventory, response not in JSON format<br/> (1434487896.83477, 0, 0, 0)<br/> Could not retrieve inventory, response not in JSON format MININET RESTART<br/> (1434487926.857697, 0, 0, 0)<br/> Could not retrieve inventory, response not in JSON format<br/> (1434487956.889432, 0, 0, 0)<br/> (1434487991.114253, 30, 34490, 32880)<br/> (1434488028.17617, 56, 68324, 67820) 7 SWITCHES LOST FROM INVENTORY<br/> (1434488059.341353, 63, 80567, 80000)<br/> (1434488089.260265, 63, 80567, 80000)<br/> ...<br/> (1434488384.822826, 63, 80567, 80000)<br/> (1434488419.865389, 59, 75451, 74920)<br/> (1434488450.665859, 63, 80567, 80000)<br/> (1434488475.408658, 63, 80567, 80000)<br/> (1434488510.034752, 63, 80567, 80000)</p> <p>this can be just another variant of the same problem, which was reported here<br/> <a href="https://bugs.opendaylight.org/show_bug.cgi?id=3735" class="external-link" target="_blank" rel="nofollow noopener">https://bugs.opendaylight.org/show_bug.cgi?id=3735</a>, but stats collectio in Lithium RC1_June15 looked more stable than in this LIthium_RC1_June16 build</p> <p>Operating System: Linux<br/> Platform: PC</p> OPNFLWPLUG-504

flows statistics unstable when 80k flow configured

Bug Resolved Won't Do Michal Rehak Peter Gubka Tue, 16 Jun 2015 21:15:05 +0000 Mon, 27 Sep 2021 09:01:35 +0000 Fri, 18 Dec 2015 13:34:25 +0000 General 0 8 <p>Attachment karaf.log has been added with description: karaf log</p> <p>Hi Peter<br/> We tried to run the python script “flow_stats_stability_monitor.py” having 1 switch and installing 80K flows. We were able to successfully install the flows. I have few queries reg Bug ID : 3762</p> <p>• Is this problem always re-creatable with 63 switches. We hit memory issues on our VMs, so couldn’t try with 63 switches. <br/> • You had mentioned that after restarting mininet, flows were not collected. Is this issue seen only during first time? <br/> • (1434487760.13342, 63, 74567, 74120) – Could you pl. provide background about what these fields mean, for our better understanding.<br/> • From the o/p (1434487881.234324, 63, 79567, 79000), my understanding is that the third argument holds the statistics of number of flows added and the last one reflects the actual flows added. With this i only infer that only 79K flows are added and the statistics reflects additional 567 flows. When the mininet is restarted, (1434488510.034752, 63, 80567, 80000) 80 K flows have been added. However, statistics shows additional 567. So, i believe that there are 2 issues - (1) 1K flows not being added (2) Statistics shows additional 567 flows. Let me know if my understanding is right. </p> <p>Thanks and Regards<br/> Kavitha</p> <p>Also, i would like to know about how <a href="https://jira.opendaylight.org/browse/OPNFLWPLUG-505" title="stats for 80k fows not collected properly after mininet restart" class="issue-link" data-issue-key="OPNFLWPLUG-505"><del>OPNFLWPLUG-505</del></a> is different from this bug.</p> <p>Thanks and regards<br/> Kavitha</p> <p>I found that <a href="https://jira.opendaylight.org/browse/OPNFLWPLUG-494" title="flow collection did not collect all flows before mininet restart" class="issue-link" data-issue-key="OPNFLWPLUG-494"><del>OPNFLWPLUG-494</del></a> looks similar to this bug.</p> <p>I used latest image for my testing. I was able to successfully add 80K flows using the script flow_stats_stability_monitor.py and statistics were in tact. I'll wait for Peter to get back to understand the exact problem recreation steps. </p> <p>Excerpt from the o/p...</p> <ul> <li> <ul> <li> <ul> <li>Total flows added: 80000<br/> HTTP<span class="error">&#91;OK&#93;</span> results: 4000</li> </ul> </li> </ul> </li> </ul> <p>(1439379949.190588, 63, 76640, 76000)<br/> (1439379978.095428, 63, 80000, 80000)<br/> (1439380014.07273, 63, 80000, 80000)<br/> (1439380049.057831, 63, 80000, 80000)<br/> (1439380073.981449, 63, 80000, 80000)<br/> (1439380099.447188, 63, 80000, 80000)<br/> (1439380130.397766, 63, 80000, 80000)<br/> (1439380166.070169, 63, 80000, 80000)<br/> (1439380188.765338, 63, 80000, 80000)<br/> (1439380222.663056, 63, 80000, 80000)<br/> (1439380254.579992, 63, 80000, 80000)<br/> (1439380278.786276, 63, 80000, 80000)<br/> (1439380312.380398, 63, 80000, 80000)<br/> (1439380345.184222, 63, 80000, 80000)<br/> (1439380370.950659, 63, 80000, 80000)<br/> (1439380406.442469, 63, 80000, 80000)<br/> (1439380430.161879, 63, 80000, 80000)<br/> (1439380466.098296, 63, 80000, 80000)<br/> (1439380491.369774, 63, 80000, 80000)<br/> (1439380523.377546, 63, 80000, 80000)<br/> (1439380552.993085, 63, 80000, 80000)<br/> (1439380585.513106, 63, 80000, 80000)<br/> (1439380608.89825, 63, 80000, 80000)<br/> (1439380642.429276, 63, 80000, 80000)<br/> (1439380676.319508, 63, 80000, 80000)<br/> (1439380699.485387, 63, 80000, 80000)</p> <p>The tupple (1434487760.13342, 63, 74567, 74120) contains <br/> (timestamp, <br/> nr of switches present in operational inventory, <br/> flows present in table statistics of operationa invenroy datastore, <br/> nr of flow details present in operational inventory)</p> <p>Since this bug was reported there are jobs like <br/> <a href="https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-periodic-scale-stats-collection-daily-lithium-redesign-only-stable-lithium/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-periodic-scale-stats-collection-daily-lithium-redesign-only-stable-lithium/</a></p> <p>which automate the same workflow as it was done manually before. They use 100k flows and 63 switches. They are robotized. In the job builds #102 and #103 you can see that 100k was not collected with failure(build 103:<br/> Keyword 'Check Flows Inventory' failed after retrying for 7 minutes 30 seconds. The last error was: 100000.0 != 98421.0</p> <p>Also you can see that flow stats collection does not fail all the time. That it why i say "unstable" stats collection. All the builds which have all 11 tests passed were ok.</p> <p>I tried running the script more than 10 times. However, i was not able to hit the problem. Looks like its a timing issue. I'll go thro' the code and see if there are any possibility for loophole. </p> <p>Peter, meanwhile can you pl. let me know if i can get access to your test set-up where i could recreate this problem.</p> <p>Any thoughts Peter about reproducing this?<br/> Also added Anil to the cc. Any thoughts Anil?</p> <p>I dont know how to reproduce this "on demand". I have been thinking about this since i reported this behavior. But having a look at yellow dots at <a href="https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-periodic-scale-stats-collection-daily-lithium-redesign-only-stable-lithium/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-periodic-scale-stats-collection-daily-lithium-redesign-only-stable-lithium/</a> we see that it is happening often.</p> <p>Abhijit, this is related to statistics manager in alternate design, so i think michal would be the best person to comment on this.</p> <p>Anil</p> <p>Michal,</p> <p>Can you comment?</p> <p>Thanks,<br/> Abhijit</p> <p>Hi Peter ,</p> <p>Are you running in a single mininet all the switches? </p> <p>Sometimes i have seen disconnects and flows not getting properly installed in a single mininet vm.</p> <p>Can you retry by distributing the switches amongst more than 1 vm?</p> <p>(In reply to Kamal Rameshan from comment #11)<br/> &gt; Hi Peter ,<br/> &gt; <br/> &gt; Are you running in a single mininet all the switches? <br/> &gt; <br/> &gt; Sometimes i have seen disconnects and flows not getting properly installed<br/> &gt; in a single mininet vm.<br/> &gt; <br/> &gt; Can you retry by distributing the switches amongst more than 1 vm?</p> <p>Hello.<br/> At the moment all these tests run on single VM in integration jenkins. <br/> Please note, that this issue it is not reproducible on demand and preparing local env with switches distributed over more VMs will take time too. And as i have tasks with higher priorities these days, not really sure when i can get back to this issue.<br/> Peter</p> <p>Made to "normal" as stalled due to no easy way to reproduce the bug.</p> <p>This bug is covered by the suite <br/> <a href="https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-periodic-scale-stats-collection-daily-lithium-redesign-only-beryllium/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-periodic-scale-stats-collection-daily-lithium-redesign-only-beryllium/</a><br/> and<br/> <a href="https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-periodic-scale-stats-collection-daily-lithium-redesign-only-stable-lithium/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-periodic-scale-stats-collection-daily-lithium-redesign-only-stable-lithium/</a></p> <p>The particular test is "Stable State Monitoring". This test has failed in the past only when the previous test failed, because 100k flows were not present in the ds/operational.</p> <p>This bug would be valid only when this particular test fail as the only one in this suite. And this has not happened for several months.</p> <p>This bug was probably fixed as a side effect of stats manager improvement in the past. So closing ....</p> Duplicate OPNFLWPLUG-494 Development External issue ID 3762 External issue URL Issue Type ODL SR Target Milestone Priority Rank 0|i031k7: