https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/OPNFLWPLUG-758 OpenFlowPlugin <p>As this test shows:</p> <p><a href="https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-3node-clustering-only-boron/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-3node-clustering-only-boron/</a></p> <p>List of candidates vary from 1-2-3 when switch connects to 1 or 2 members only.</p> <p>BR/Luis</p> <p>Operating System: All<br/> Platform: All</p> OPNFLWPLUG-758

Li plugin: Controller rejects switch connections

Bug Resolved Done Jozef Bacigal Luis Gomez Fri, 26 Aug 2016 20:46:02 +0000 Mon, 27 Sep 2021 09:01:53 +0000 Thu, 8 Sep 2016 07:29:46 +0000 General 0 8 <p>I changed the title of the bug because I think the entity owner unstability is generated by a quick switch disconnect + connect transition. Looking at this log:</p> <p>2016-08-30 08:47:41,330 | INFO | entLoopGroup-5-1 | SystemNotificationsListenerImpl | 207 - org.opendaylight.openflowplugin.impl - 0.3.0.Boron-RC1 | ConnectionEvent: Connection closed by device, Device:/192.168.0.20:56480, NodeId:null<br/> 2016-08-30 08:47:41,447 | INFO | entLoopGroup-5-2 | ConnectionAdapterImpl | 197 - org.opendaylight.openflowjava.openflow-protocol-impl - 0.8.0.Boron-RC1 | Hello received / branch<br/> 2016-08-30 08:47:41,449 | WARN | entLoopGroup-5-2 | DeviceManagerImpl | 207 - org.opendaylight.openflowplugin.impl - 0.3.0.Boron-RC1 | Node org.opendaylight.openflowplugin.impl.connection.ConnectionContextImpl$DeviceInfoImpl@a774dfb8 already connected disconnecting device. Rejecting connection<br/> 2016-08-30 08:47:41,449 | WARN | entLoopGroup-5-2 | DeviceManagerImpl | 207 - org.opendaylight.openflowplugin.impl - 0.3.0.Boron-RC1 | Node openflow:1 context state not in TERMINATION state.<br/> 2016-08-30 08:47:41,449 | INFO | entLoopGroup-5-2 | ConnectionContextImpl | 207 - org.opendaylight.openflowplugin.impl - 0.3.0.Boron-RC1 | Unregister outbound queue successful.</p> <p>A hello packet is received before the device is fully removed from the controller. The hello packet stops the device unregister and at the same time switch connection gets rejected because the device is still register. This generates an unstable switch connection.</p> <p>This same issue is seen often during the cluster test and the result is always an unstable controller with entity-owner, inventory and topology entries when the switch in reality does not exist anymore.</p> <p>2016-08-31 00:21:20,553 | INFO | entLoopGroup-7-4 | ConnectionAdapterImpl | 270 - org.opendaylight.openflowjava.openflow-protocol-impl - 0.8.0.Boron-RC1 | Hello received / branch<br/> 2016-08-31 00:21:20,559 | WARN | entLoopGroup-7-4 | DeviceManagerImpl | 280 - org.opendaylight.openflowplugin.impl - 0.3.0.Boron-RC1 | Node org.opendaylight.openflowplugin.impl.connection.ConnectionContextImpl$DeviceInfoImpl@1f8d8b33 already connected disconnecting device. Rejecting connection<br/> 2016-08-31 00:21:20,559 | INFO | entLoopGroup-7-4 | ConnectionContextImpl | 280 - org.opendaylight.openflowplugin.impl - 0.3.0.Boron-RC1 | Unregister outbound queue successful.<br/> 2016-08-31 00:21:22,554 | INFO | entLoopGroup-7-1 | ConnectionAdapterImpl | 270 - org.opendaylight.openflowjava.openflow-protocol-impl - 0.8.0.Boron-RC1 | Hello received / branch<br/> 2016-08-31 00:21:22,558 | WARN | entLoopGroup-7-1 | DeviceManagerImpl | 280 - org.opendaylight.openflowplugin.impl - 0.3.0.Boron-RC1 | Node org.opendaylight.openflowplugin.impl.connection.ConnectionContextImpl$DeviceInfoImpl@1f8d8b33 already connected disconnecting device. Rejecting connection<br/> 2016-08-31 00:21:22,559 | INFO | entLoopGroup-7-1 | ConnectionContextImpl | 280 - org.opendaylight.openflowplugin.impl - 0.3.0.Boron-RC1 | Unregister outbound queue successful.<br/> 2016-08-31 00:21:22,722 | WARN | pool-31-thread-1 | StatisticsManagerImpl | 280 - org.opendaylight.openflowplugin.impl - 0.3.0.Boron-RC1 | Statistics gathering for single node was not successful: Device connection doesn't exist anymore. Primary connection status : RIP</p> <p>Raising to blocker, we cannot afford to have unstable OpenFlow cluster in Boron.</p> <p>The easiest way to reproduce this issue is to stop and start mininet with no delay for few times. After a while you will see stale entries in entity-owner, topology and inventory.</p> <p>The ERRORs described in this bug show very often in 2 scenarios:</p> <p>1) OpenFlow Cluster test:</p> <p><a href="https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-3node-clustering-only-boron/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-3node-clustering-only-boron/</a></p> <p>That is why the test suite is not stable.</p> <p>2) Switch scalability test single node:</p> <p><a href="https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-periodic-scalability-daily-only-boron/plot/Inventory%20Scalability/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-periodic-scalability-daily-only-boron/plot/Inventory%20Scalability/</a></p> <p>That is why we do not see good switch scalability.</p> <p>In general this issue is more reproducible:</p> <ul class="alternate" type="square"> <li>In cluster environment</li> <li>When flapping switch connections</li> <li>In switch scalibility scenarios</li> </ul> <p>Is there an ETA for this bug and someone assigned to fix?</p> <p>Do we think this bug is related to (or caused by) BUG-6540?</p> <p>No, this bugs occurs without cluster member isolation and even in single instance test. The bug you point seems to be related to <a href="https://bugs.opendaylight.org/show_bug.cgi?id=6177" class="external-link" target="_blank" rel="nofollow noopener">https://bugs.opendaylight.org/show_bug.cgi?id=6177</a> which is not blocker (major bug).</p> <p>I see also this ERROR very often prior to device reject:</p> <p>2016-09-03 20:03:06,703 | WARN | pool-32-thread-1 | RoleContextImpl | 183 - org.opendaylight.openflowplugin.impl - 0.3.0.SNAPSHOT | New role BECOMESLAVE was not propagated to device openflow:160 during 10 sec<br/> 2016-09-03 20:03:06,703 | ERROR | pool-32-thread-1 | SalRoleServiceImpl | 183 - org.opendaylight.openflowplugin.impl - 0.3.0.SNAPSHOT | SetRoleService set Role BECOMESLAVE for Node: KeyedInstanceIdentifier</p> {targetType=interface org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node, path=[org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.Nodes, org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node[key=NodeKey [_id=Uri [_value=openflow:160]]]]} <p> fail . Reason java.util.concurrent.CancellationException: Task was cancelled.<br/> 2016-09-03 20:03:06,703 | WARN | pool-32-thread-1 | RoleManagerImpl | 183 - org.opendaylight.openflowplugin.impl - 0.3.0.SNAPSHOT | Was not able to set role SLAVE to device on node openflow:160 </p> <p>The above ERROR is kind of often in single node switch scalability and longevity tests.</p> <p>In cluster test, there is an easy way to reproduce the above:</p> <p>1) start mininet 1 switch (s1) pointing to any of the instances (e.g. 192.168.0.101)<br/> 2) Run the following script in the mininet system to bounce connection quickly:</p> <p>#!/bin/bash<br/> sudo ovs-vsctl del-controller s1<br/> sleep 0.2<br/> sudo ovs-vsctl set-controller s1 "tcp:192.168.0.101"</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/44664/6" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/44664/6</a></p> <p>To better assess the impact of this bug and fix, could someone from your team please help us identify the following:<br/> Severity: Could you elaborate on the severity of this bug? Is this a BLOCKER such that we cannot release Boron without it? Is there a workaround such that we can write a release note and fix in future Boron SR1?<br/> Testing: Could you also elaborate on the testing of this patch? How extensively has this patch been tested? Is it covered by any unit tests or system tests? <br/> Impact: Does this fix impact any dependent projects?</p> <p>After this merge I see more stable cluster test but still some sporadic issue in switch scalability test:</p> <p><a href="https://logs.opendaylight.org/releng/jenkins092/openflowplugin-csit-1node-periodic-scalability-daily-only-boron/242/archives/karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/openflowplugin-csit-1node-periodic-scalability-daily-only-boron/242/archives/karaf.log.gz</a></p> <p>So far we cannot close this issue but maybe reduce importance if cluster behaves more stable (wee need few more runs to confirm).</p> <p>BR/Luis</p> <p>OK, after a day of test I downgraded the importance from Blocker to Major. The reason is cluster test seems stable now:</p> <p><a href="https://jenkins.opendaylight.org/releng/view/CSIT-3node/job/openflowplugin-csit-3node-clustering-only-boron/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/CSIT-3node/job/openflowplugin-csit-3node-clustering-only-boron/</a></p> <p>The switch scalability test, even when showing the issue sometimes, it has stabilize the max number:</p> <p><a href="https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-periodic-scalability-daily-only-boron/plot/Inventory%20Scalability/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-periodic-scalability-daily-only-boron/plot/Inventory%20Scalability/</a></p> <p>BR/Luis</p> <p>So, can we note the maximum number of switch connections somewhere? I guess that would be part of the performance report, which we'll hopefully have time to update...</p> <p>Has this bug been verified as fixed in the latest Boron RC 3.1 Build?</p> <p>Closing this as solved and creating new one for the switch scalability test.</p> <p>Jozef</p> Development External issue ID 6554 External issue URL Rank 0|i0334n: