OpenDaylight JIRA https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 [OPNFLWPLUG-875] Switch scalability regression due to missing table miss flows https://jira.opendaylight.org/browse/OPNFLWPLUG-875 OpenFlowPlugin <p>This is detected here for both carbon and boron:</p> <p><a href="https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-periodic-sw-scalability-daily-only-carbon/plot/Switch%20Scalability/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-periodic-sw-scalability-daily-only-carbon/plot/Switch%20Scalability/</a></p> <p><a href="https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-periodic-sw-scalability-daily-only-boron/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-periodic-sw-scalability-daily-only-boron/</a></p> <p>Basically the scalability test fails because missing table miss flows required for topology discovery.</p> <p>Operating System: All<br/> Platform: All</p> OPNFLWPLUG-875 Switch scalability regression due to missing table miss flows Bug Resolved Done Luis Gomez Luis Gomez Wed, 29 Mar 2017 23:38:20 +0000 Mon, 27 Sep 2021 09:02:01 +0000 Mon, 16 Oct 2017 18:56:00 +0000 General 0 4 <p>Isn't this duplicate of 7770?</p> <p>Very possible, we can fix 7770 and then recheck this one.</p> <p>Note also that 7770 is cluster releated while this one happens with single instance.</p> <p>Hi Luis,<br/> Could you please let us know, from which build number you have seen this issue.</p> <p>Now the scalability TCs are passing in both boron and carbon.</p> <p>Regards,<br/> Arun</p> <p>Yes they are passing but see the regression from 500 to 200:</p> <p><a href="https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-periodic-sw-scalability-daily-only-boron/plot/Switch%20Scalability/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-periodic-sw-scalability-daily-only-boron/plot/Switch%20Scalability/</a></p> <p><a href="https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-periodic-sw-scalability-daily-only-carbon/plot/Switch%20Scalability/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-periodic-sw-scalability-daily-only-carbon/plot/Switch%20Scalability/</a></p> <p>The regresion happend long back by the beginning of the year. BTW If I remove the topology test, the test goes back to 500 switches because issue is related to missing table miss flows.</p> <p>BR/Luis</p> <p>Thanks Luis.</p> <p>Since its new to me, could you please help me in the steps to disable the topology tests and run the regression.</p> <p>Regards,<br/> Arun</p> <p>To reproduce the issue you can just generate multiple iterations of mininet linear topology with 100, 200, 300 nodes, etc... After a while, you will observe not all links are properly discovered. Nodes are good though.</p> <p>I always thought the issue was in table miss application but actually the issue could be more this ERROR present in all the karaf logs <span class="error">&#91;1&#93;</span> for this test:</p> <p>2017-05-10 07:13:04,862 | ERROR | pool-27-thread-1 | OutboundQueueProviderImpl | 193 - org.opendaylight.openflowplugin.impl - 0.4.0.Carbon | No queue present, failing request</p> <p><span class="error">&#91;1&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/openflowplugin-csit-1node-periodic-sw-scalability-daily-only-carbon/259/archives/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/openflowplugin-csit-1node-periodic-sw-scalability-daily-only-carbon/259/archives/odl1_karaf.log.gz</a></p> <p>Note this issue can be connected to: <a href="https://bugs.opendaylight.org/show_bug.cgi?id=8401" class="external-link" target="_blank" rel="nofollow noopener">https://bugs.opendaylight.org/show_bug.cgi?id=8401</a></p> <p>Note that karaf log for scale/perf test is set to ERROR only. I can also enable full debug if required.</p> <p>Luis,<br/> Could you please enable the debug logs and run the test again, so we can confirm if its same issue which was reported in </p> <p><a href="https://bugs.opendaylight.org/show_bug.cgi?id=8401" class="external-link" target="_blank" rel="nofollow noopener">https://bugs.opendaylight.org/show_bug.cgi?id=8401</a></p> <p>Regards,<br/> Arun</p> <p>Log is too big to attach, you can grab from here:</p> <p><a href="https://logs.opendaylight.org/releng/jenkins092/openflowplugin-csit-1node-periodic-sw-scalability-daily-only-carbon/272/archives/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/openflowplugin-csit-1node-periodic-sw-scalability-daily-only-carbon/272/archives/odl1_karaf.log.gz</a></p> <p>After looking at it, I do not see it is the same as <a href="https://jira.opendaylight.org/browse/OPNFLWPLUG-887" title="After karaf restart OF writes fail due to &quot;No queue present, failing request&quot; in OutboundQueueProviderImpl" class="issue-link" data-issue-key="OPNFLWPLUG-887"><del>OPNFLWPLUG-887</del></a>.</p> <p>BR/Luis</p> <p>Flowing exception seen several times in the karaf log</p> <p>2017-05-14 18:54:15,488 | WARN | entLoopGroup-7-5 | OFFrameDecoder | 182 - org.opendaylight.openflowjava.openflow-protocol-impl - 0.9.0.SNAPSHOT | Unexpected exception from downstream.<br/> io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: Connection reset by peer<br/> at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown Source)<br/> 2017-05-14 18:54:15,490 | WARN | entLoopGroup-7-5 | OFFrameDecoder | 182 - org.opendaylight.openflowjava.openflow-protocol-impl - 0.9.0.SNAPSHOT | Closing connection.<br/> 2017-05-14 18:54:15,491 | INFO | entLoopGroup-7-5 | SystemNotificationsListenerImpl | 193 - org.opendaylight.openflowplugin.impl - 0.4.0.SNAPSHOT | ConnectionEvent: Connection closed by device, Device:/10.29.12.106:46524, NodeId:openflow:78</p> <p>2017-05-14 19:06:26,955 | WARN | entLoopGroup-7-2 | OFFrameDecoder | 182 - org.opendaylight.openflowjava.openflow-protocol-impl - 0.9.0.SNAPSHOT | Unexpected exception from downstream.<br/> io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: Connection reset by peer<br/> at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown Source)<br/> 2017-05-14 19:06:26,956 | WARN | entLoopGroup-7-2 | OFFrameDecoder | 182 - org.opendaylight.openflowjava.openflow-protocol-impl - 0.9.0.SNAPSHOT | Closing connection.<br/> 2017-05-14 19:06:26,956 | INFO | entLoopGroup-7-2 | SystemNotificationsListenerImpl | 193 - org.opendaylight.openflowplugin.impl - 0.4.0.SNAPSHOT | ConnectionEvent: Connection closed by device, Device:/10.29.12.106:47065, NodeId:openflow:271</p> <p>I'm seeing the below error as well...<br/> =============================================</p> <p>2017-05-14 19:06:24,160 | ERROR | pool-31-thread-1 | OutboundQueueProviderImpl | 193 - org.opendaylight.openflowplugin.impl - 0.4.0.SNAPSHOT | No queue present, failing request<br/> 2017-05-14 19:06:24,160 | WARN | pool-31-thread-1 | RpcContextImpl | 193 - org.opendaylight.openflowplugin.impl - 0.4.0.SNAPSHOT | Xid cannot be reserved for new RequestContext, node:openflow:166<br/> 2017-05-14 19:06:24,161 | ERROR | pool-31-thread-1 | OutboundQueueProviderImpl | 193 - org.opendaylight.openflowplugin.impl - 0.4.0.SNAPSHOT | No queue present, failing request<br/> 2017-05-14 19:06:24,161 | WARN | pool-31-thread-1 | RpcContextImpl | 193 - org.opendaylight.openflowplugin.impl - 0.4.0.SNAPSHOT | Xid cannot be reserved for new RequestContext, node:openflow:29<br/> 2017-05-14 19:06:24,162 | ERROR | pool-31-thread-1 | OutboundQueueProviderImpl | 193 - org.opendaylight.openflowplugin.impl - 0.4.0.SNAPSHOT | No queue present, failing request<br/> 2017-05-14 19:06:24,165 | WARN | pool-31-thread-1 | RpcContextImpl | 193 - org.opendaylight.openflowplugin.impl - 0.4.0.SNAPSHOT | Xid cannot be reserved for new RequestContext, node:openflow:180<br/> 2017-05-14 19:06:24,165 | ERROR | pool-31-thread-1 | OutboundQueueProviderImpl | 193 - org.opendaylight.openflowplugin.impl - 0.4.0.SNAPSHOT | No queue present, failing request</p> <p>OK, I just reproduced the issue in my laptop using 300 switches. I can see all switches come good but some do not have the table miss flow. There is nothing relevant in karaf.log when this happens. Since table miss flow is a test application, I will modify the robot test to skip this app and use NB pushed table miss flows instead.</p> <p>I believe following patch should fix the "No queue present" issue reported in above comment.</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/56927/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/56927/</a></p> <p>Good point, let me try with latest carbon to see if I see any better.</p> <p>And it works in carbon, I do not see table miss flows missing with latest code <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Anil, is it possible to cherry-pick the change to boron?</p> <p>I was too fast to say it works, it seems it is better now, but still same logging:</p> <p><a href="https://logs.opendaylight.org/sandbox/jenkins091/openflowplugin-csit-1node-periodic-sw-scalability-daily-only-carbon/2/archives/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/openflowplugin-csit-1node-periodic-sw-scalability-daily-only-carbon/2/archives/odl1_karaf.log.gz</a></p> <p>So after reproducing this locally, I can see the main issue is still missing table miss flow after ~200 switches. Nothing in the logs give any hint of what the problem is.</p> <p>To reproduce, just generate linear,300 or 400 topology and check flows in mininet (dpctl dump-aggregate -O OpenFlow13), there should be 1 table miss flow per switch, instead some switch miss the flow and therefore topology verification fails in this test.</p> <p>So the real issue seems to be flow installation after switches connect (i.e. flow reconciliation). No matter if I pre-program flows in DS or use table miss flow feature, I always miss flows when many switches connect in short time. The only workaround is to install the table miss flows after all switches are fully connected to ODL. I created this patch to prove so:</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/57660" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/57660</a></p> <p>Hello Luis,</p> <p>I tried to reproduce the issue with steps provided by you, i tried gradually increasing the number of switches till 250 before my laptop ran out of resource at 300. <br/> I have cherry picked this patch on top of master branch: <br/> <a href="https://git.opendaylight.org/gerrit/#/c/57814/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/57814/</a><br/> I observed that table-miss-flow are programmed on all the switches connected. </p> <p>Please try with the patch and let us know on your findings. </p> <p>Thanks.</p> <p>So that patch mentioned by Sunil was merged on both carbon and nitrogen and it should fix also this issue, so can you confirm this Luis?</p> <p>When I test in my setup, 300 switches works good but when I go to 400, I still see table miss flows missing.</p> <p>Also this test shows table miss flows are not even stable:</p> <p><a href="https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-flow-services-only-carbon/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-flow-services-only-carbon/</a></p> <p>Luis</p> <p>Not a blocker for Nitrogen. However can be checked after <a href="https://bugs.opendaylight.org/show_bug.cgi?id=9089" class="external-link" target="_blank" rel="nofollow noopener">https://bugs.opendaylight.org/show_bug.cgi?id=9089</a> is merged.</p> <p>So <a href="https://jira.opendaylight.org/browse/OPNFLWPLUG-939" title="Regression in Switch scalability test (carbon)" class="issue-link" data-issue-key="OPNFLWPLUG-939"><del>OPNFLWPLUG-939</del></a> was merged, can you recheck this one Luis?</p> <p>OK, I just tested in my laptop and this seems to be fixed now.</p> Development External issue ID 8103 External issue URL ODL SR Target Milestone Rank 0|i033un: