https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/OPNFLWPLUG-918 OpenFlowPlugin <p>This issue is happening in Carbon/Nitrogen (not in Boron), it is tracked here:</p> <p>To reproduce:</p> <p>1) Start mininet linear 32:<br/> sudo mn --controller 'remote,ip=192.168.0.1,port=6633' --topo linear,32</p> <p>2) Push 100K flows (script available in int/test repo):<br/> python odl_tester.py --threads 5 --flows 100000 --no-delete --fpr 200</p> <p>Observe after 100K flows are added and controller is stable CPU is still very high.</p> <p>3) Remove flows from inventory:<br/> DELETE http://<tt>controller</tt>:8181/restconf/config/opendaylight-inventory:nodes/</p> <p>Observe after 100K flows are removed and controller is stable, there are still flows in operational that never get removed.</p> <p> </p> <p>CSIT job for this test : <a href="https://jenkins.opendaylight.org/releng/job/openflowplugin-csit-1node-periodic-scale-stats-collection-daily-only-carbon/" class="external-link" target="_blank" rel="nofollow noopener">ttps://jenkins.opendaylight.org/releng/job/openflowplugin-csit-1node-periodic-scale-stats-collection-daily-only-carbon/</a></p> <p>Operating System: All<br/> Platform: All</p> OPNFLWPLUG-918

Regression: Controller fails to delete 100K flows from switches

Bug Medium Confirmed Unresolved Anil Vishnoi Luis Gomez Mon, 3 Jul 2017 05:04:14 +0000 Mon, 27 Sep 2021 09:28:22 +0000 Nitrogen Oxygen Fluorine General 0 6 <p>Forgot to add the link:</p> <p><a href="https://jenkins.opendaylight.org/releng/job/openflowplugin-csit-1node-periodic-scale-stats-collection-daily-only-carbon/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/job/openflowplugin-csit-1node-periodic-scale-stats-collection-daily-only-carbon/</a></p> <p>So it looks like on nitrogen this is already working, and on carbon it will be working after this cherry-pick will be merged: <a href="https://git.opendaylight.org/gerrit/#/c/60196/1" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/60196/1</a></p> <p>Oh sorry, wrong 100k flows bug, this was meant for 6755 one.</p> <p>This issue only happens in Carbon now and it is FRM related because flows are not removed from switch when we delete them in the inventory.</p> <p>This is a regression for Carbon (throughout Carbon) - works in Boron and in Nitrogen.</p> <p>This is sporadically failing on Nitrogen too. Anyway, here is patch that will resolve this issue: <a href="https://git.opendaylight.org/gerrit/#/c/63101/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/63101/</a></p> <p>Yeah, also if you look at the karaf logs when it fails, there are few switch disconnects after DS is cleared in case of nitrogen and a bunch of disconnects in case of carbon. This means the controller gets so busy with the DS clear operation that it misses the ECHO requests from switches and these start to disconnect and reconnect aggravating the problem. If we do not fix this for nitrogen, i will open a blocker for carbon as the issue is much more apparent in carbon.</p> <p>BTW, to support my observation I started couple of tests with switch ECHO message disabled, and both passed in carbon and nitrogen:</p> <p><a href="https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-periodic-gate-scale-stats-collection-daily-only-carbon/154/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-periodic-gate-scale-stats-collection-daily-only-carbon/154/</a></p> <p><a href="https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-periodic-gate-scale-stats-collection-daily-only-nitrogen/281/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-periodic-gate-scale-stats-collection-daily-only-nitrogen/281/</a></p> <p>BR/Luis</p> <p>This was caused because of heavy load on Netty thread, what was caused probably by single-layer serialization. Explanation: with multi-layer, we are doing conversion on different thread and then serializing simple OFJ data structure. In single-layer, we are doing everything on Netty thread. Overall, it is faster by around 1/3 based on YourKit observations what I did during 100k flow test locally, but all load is on Netty thread, so it disconnects. My patch that I mentioned before solves this by pre-serializing data coming via single-layer and then sending raw bytes to netty thread, what causes almost no load on Netty thread.</p> <p>Right, you patch works in oxygen:</p> <p><a href="https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-periodic-gate-scale-stats-collection-daily-only-oxygen/13/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-periodic-gate-scale-stats-collection-daily-only-oxygen/13/</a></p> <p>Do you want to cherry-pick to other branches now to test? or later when your patch is merged in master?</p> <p>After it will be merged probably.</p> <p>Created cherry-pick for stable/nitrogen: <a href="https://git.opendaylight.org/gerrit/#/c/63491/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/63491/</a></p> <p>Cherry-pick to stable/carbon will come after that.</p> <p>stable/carbon: <a href="https://git.opendaylight.org/gerrit/#/c/63496/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/63496/</a></p> <p>Alright, updated the patch for stable/carbon, again I forgot to actually load the configuration flag to enable the preserialization.</p> <p>Tomas, does this patch also cover what was in <a href="https://git.opendaylight.org/gerrit/#/c/62792/?" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/62792/?</a> that patch also had bug-8787 and was abandoned.</p> <p>That abandoned patch was also supposed to fix <a href="https://bugs.opendaylight.org/show_bug.cgi?id=7826" class="external-link" target="_blank" rel="nofollow noopener">https://bugs.opendaylight.org/show_bug.cgi?id=7826</a>, but it didn't so also wondering if 7826 is fixed in this patch now?</p> <p>No, the flow-related patch tried to improve performance of flow deletion, but it did not (and also caused more issues) so I abandoned it as current method for flow deletion what we are using seems more performant. Also, that patch was never supposed to solve <a href="https://jira.opendaylight.org/browse/OPNFLWPLUG-858" title="Data validation failed for path /(nodes/node/node/../group-id" class="issue-link" data-issue-key="OPNFLWPLUG-858"><del>OPNFLWPLUG-858</del></a> i just told Sunil about it because he was looking at device registries and I did some modifications to registries and deletion of marks there.</p> <p>This current patch that solves this issue do not modifies the registry at all, so it will not help the <a href="https://jira.opendaylight.org/browse/OPNFLWPLUG-858" title="Data validation failed for path /(nodes/node/node/../group-id" class="issue-link" data-issue-key="OPNFLWPLUG-858"><del>OPNFLWPLUG-858</del></a> after it will be merged.</p> <p>Last patch seems to work so we can close this bug after it is merged.</p> <p>The patches was not working for extensions, so updated them to include real calls to serialization registry. Unfortunately that required changes in OFJ, and for stable/carbon it is stil separate project, so here is additional patch for OFJ for stable/carbon: <a href="https://git.opendaylight.org/gerrit/#/c/63843/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/63843/</a></p> <p>We have this issue in all branches because no patch has been merged yet.</p> <p>Moving it to Arun.</p> <p>Anil reviewing these set of patches:</p> <p><a href="https://git.opendaylight.org/gerrit/#/q/topic:bug/8787+status:open" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/q/topic:bug/8787+status:open</a></p> <p>From last test results:</p> <ul> <li>Carbon controller is not able to delete flows from switches: this is where we see switches disconnecting.</li> <li>Nitrogen and Oxygen controller delete flows from switches but stats are not correct (show flows in operational inventory).</li> </ul> <p>This job started to work after adjusting 2 things:</p> <ul class="alternate" type="square"> <li>poll stats interval: 3 -&gt; 10 sec.</li> <li>disable switch echo messages.</li> </ul> <p>I think we still have issue of echo messages not responded when controller is busy, so I will leave this bug open for now reducing severity to Major.</p> <p>Ideally this test should with without the above mentioned workaround. This bug is kept open to address this issue without the above workaround.</p> <p>Given that we have a workaround for this issue, reducing the priority.</p> Duplicate OPNFLWPLUG-890 Development External issue ID 8787 External issue URL Rank 0|i03447: