OpenDaylight JIRA https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 [OPNFLWPLUG-920] Cluster member gets in bad state if 2 instances disconnect just after the 3rd is recovered https://jira.opendaylight.org/browse/OPNFLWPLUG-920 OpenFlowPlugin <p>This happens in stable/carbon. To reproduce:</p> <p>1) Connect single switch to cluster<br/> sudo ovs-vsctl set-controller s1 "tcp:192.168.0.101:6633" "tcp:192.168.0.102:6633" "tcp:192.168.0.103:6633"</p> <p>2) Wait until all connections get stablished, master and slave after ~20 secs.</p> <p>3) Break connection Master to switch:<br/> sudo iptables -A INPUT -s 192.168.0.103 -p tcp --sport 6633 -j DROP</p> <p>4) Wait until new master is elected and then restore the connection:<br/> sudo iptables -D INPUT -s 192.168.0.103 -p tcp --sport 6633 -j DROP</p> <p>5) Now that connection is waiting for slave break new master and remaining connection:<br/> sudo iptables -A INPUT -s 192.168.0.101 -p tcp --sport 6633 -j DROP<br/> sudo iptables -A INPUT -s 192.168.0.102 -p tcp --sport 6633 -j DROP</p> <p>6) Following is observed in the remaining node (102), after that only solution is to reboot the node:</p> <p>2017-07-06 01:06:06,352 | INFO | ofppool-0 | LifecycleServiceImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.1.SNAPSHOT | Starting clustering services for node openflow:1<br/> 2017-07-06 01:06:06,352 | INFO | ofppool-0 | DeviceContextImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.1.SNAPSHOT | Starting device context cluster services for node openflow:1<br/> 2017-07-06 01:06:10,319 | INFO | entLoopGroup-7-2 | SystemNotificationsListenerImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.1.SNAPSHOT | ConnectionEvent: Connection closed by device, Device:/192.168.0.24:49423, NodeId:openflow:1<br/> 2017-07-06 01:06:10,319 | INFO | entLoopGroup-7-2 | ContextChainHolderImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.1.SNAPSHOT | Device openflow:1 disconnected.<br/> 2017-07-06 01:06:10,319 | INFO | entLoopGroup-7-2 | LifecycleServiceImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.1.SNAPSHOT | Closing clustering services registration for node openflow:1<br/> 2017-07-06 01:06:10,323 | WARN | entLoopGroup-7-2 | OF13DeviceInitializer | 220 - org.opendaylight.openflowplugin.impl - 0.4.1.SNAPSHOT | Failed to write node openflow:1 to DS because we failed to gather device info.<br/> 2017-07-06 01:06:10,323 | ERROR | entLoopGroup-7-2 | OutboundQueueProviderImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.1.SNAPSHOT | No queue present, failing request<br/> 2017-07-06 01:06:10,324 | WARN | entLoopGroup-7-2 | OF13DeviceInitializer | 220 - org.opendaylight.openflowplugin.impl - 0.4.1.SNAPSHOT | Failed to retrieve static node OFPMPMETERFEATURES info: Outbound queue wasn't able to reserve XID.<br/> 2017-07-06 01:06:10,324 | ERROR | entLoopGroup-7-2 | OutboundQueueProviderImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.1.SNAPSHOT | No queue present, failing request<br/> 2017-07-06 01:06:10,324 | WARN | entLoopGroup-7-2 | OF13DeviceInitializer | 220 - org.opendaylight.openflowplugin.impl - 0.4.1.SNAPSHOT | Failed to retrieve static node OFPMPGROUPFEATURES info: Outbound queue wasn't able to reserve XID.<br/> 2017-07-06 01:06:10,325 | ERROR | entLoopGroup-7-2 | OutboundQueueProviderImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.1.SNAPSHOT | No queue present, failing request<br/> 2017-07-06 01:06:10,325 | WARN | entLoopGroup-7-2 | OF13DeviceInitializer | 220 - org.opendaylight.openflowplugin.impl - 0.4.1.SNAPSHOT | Failed to retrieve static node OFPMPPORTDESC info: Outbound queue wasn't able to reserve XID.<br/> 2017-07-06 01:06:10,325 | INFO | entLoopGroup-7-2 | OF13DeviceInitializer | 220 - org.opendaylight.openflowplugin.impl - 0.4.1.SNAPSHOT | Static node openflow:1 successfully finished collecting<br/> 2017-07-06 01:06:10,325 | INFO | entLoopGroup-7-2 | StatisticsContextImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.1.SNAPSHOT | Stopping running statistics gathering for node openflow:1<br/> 2017-07-06 01:06:10,326 | INFO | ofppool-0 | SalRoleServiceImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.1.SNAPSHOT | SetRole called with input:SetRoleInput [_controllerRole=BECOMEMASTER, _node=NodeRef [_value=KeyedInstanceIdentifier</p> {targetType=interface org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node, path=[org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.Nodes, org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node[key=NodeKey [_id=Uri [_value=openflow:1]]]]} <p>], augmentation=[]]<br/> 2017-07-06 01:06:10,326 | INFO | ofppool-0 | SalRoleServiceImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.1.SNAPSHOT | Device NodeRef [_value=KeyedInstanceIdentifier</p> {targetType=interface org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node, path=[org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.Nodes, org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node[key=NodeKey [_id=Uri [_value=openflow:1]]]]} <p>] has been disconnected<br/> 2017-07-06 01:06:10,326 | ERROR | ofppool-0 | SalRoleServiceImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.1.SNAPSHOT | SetRoleService set Role BECOMEMASTER for Node: KeyedInstanceIdentifier</p> {targetType=interface org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node, path=[org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.Nodes, org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node[key=NodeKey [_id=Uri [_value=openflow:1]]]]} <p> fail . Reason java.lang.Exception: Device connection doesn't exist anymore. Primary connection status : RIP<br/> 2017-07-06 01:06:10,326 | WARN | ofppool-0 | ContextChainHolderImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.1.SNAPSHOT | Not able to set MASTER role on device openflow:1, reason: Was not able to set MASTER role on device<br/> 2017-07-06 01:06:10,332 | INFO | ult-dispatcher-5 | LifecycleServiceImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.1.SNAPSHOT | Closing clustering services for node openflow:1<br/> 2017-07-06 01:06:10,334 | INFO | ofppool-0 | StatisticsContextImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.1.SNAPSHOT | Starting statistics context cluster services for node openflow:1<br/> 2017-07-06 01:06:10,335 | WARN | ofppool-0 | ContextChainHolderImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.1.SNAPSHOT | Not able to set MASTER role on device openflow:1, reason: Initial gathering statistics unsuccessful.<br/> 2017-07-06 01:06:10,335 | INFO | ofppool-0 | RpcContextImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.1.SNAPSHOT | Starting rpc context cluster services for node openflow:1<br/> 2017-07-06 01:06:10,336 | INFO | lt-dispatcher-29 | ContextChainHolderImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.1.SNAPSHOT | Removing device Uri <span class="error">&#91;_value=openflow:1&#93;</span> from operational DS<br/> 2017-07-06 01:06:11,380 | ERROR | n-dispatcher-590 | OutboundQueueProviderImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.1.SNAPSHOT | No queue present, failing request<br/> 2017-07-06 01:06:11,381 | WARN | n-dispatcher-590 | RpcContextImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.1.SNAPSHOT | Xid cannot be reserved for new RequestContext, node:openflow:1<br/> 2017-07-06 01:06:11,380 | ERROR | n-dispatcher-558 | OutboundQueueProviderImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.1.SNAPSHOT | No queue present, failing request<br/> 2017-07-06 01:06:11,381 | WARN | n-dispatcher-558 | RpcContextImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.1.SNAPSHOT | Xid cannot be reserved for new RequestContext, node:openflow:1<br/> 2017-07-06 01:06:11,382 | ERROR | n-dispatcher-589 | OutboundQueueProviderImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.1.SNAPSHOT | No queue present, failing request<br/> 2017-07-06 01:06:11,384 | WARN | n-dispatcher-589 | RpcContextImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.1.SNAPSHOT | Xid cannot be reserved for new RequestContext, node:openflow:1<br/> 2017-07-06 01:06:11,384 | ERROR | n-dispatcher-589 | OutboundQueueProviderImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.1.SNAPSHOT | No queue present, failing request<br/> 2017-07-06 01:06:11,384 | WARN | n-dispatcher-589 | RpcContextImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.1.SNAPSHOT | Xid cannot be reserved for new RequestContext, node:openflow:1<br/> 2017-07-06 01:06:12,377 | ERROR | ult-dispatcher-3 | OutboundQueueProviderImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.1.SNAPSHOT | No queue present, failing request<br/> 2017-07-06 01:06:12,378 | WARN | ult-dispatcher-3 | RpcContextImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.1.SNAPSHOT | Xid cannot be reserved for new RequestContext, node:openflow:1<br/> 2017-07-06 01:06:14,901 | ERROR | pool-26-thread-1 | OutboundQueueProviderImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.1.SNAPSHOT | No queue present, failing request<br/> 2017-07-06 01:06:14,902 | WARN | pool-26-thread-1 | RpcContextImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.1.SNAPSHOT | Xid cannot be reserved for new RequestContext, node:openflow:1<br/> 2017-07-06 01:06:14,902 | ERROR | pool-26-thread-1 | OutboundQueueProviderImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.1.SNAPSHOT | No queue present, failing request<br/> 2017-07-06 01:06:14,902 | WARN | pool-26-thread-1 | RpcContextImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.1.SNAPSHOT | Xid cannot be reserved for new RequestContext, node:openflow:1</p> <p>Operating System: All<br/> Platform: All</p> OPNFLWPLUG-920 Cluster member gets in bad state if 2 instances disconnect just after the 3rd is recovered Bug Resolved Done Luis Gomez Luis Gomez Thu, 6 Jul 2017 01:21:20 +0000 Mon, 27 Sep 2021 09:02:04 +0000 Wed, 20 Sep 2017 19:56:39 +0000 General 0 4 <p>Can you try it again on this patch Luis? <a href="https://git.opendaylight.org/gerrit/#/c/60075/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/60075/</a></p> <p>I was not able to reproduce the error on that patch.</p> <p>Here is same patch for stable/carbon: <a href="https://git.opendaylight.org/gerrit/#/c/60144/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/60144/</a></p> <p>Lowering priority to critical as this is kind of corner case.</p> <p>(In reply to Luis Gomez from comment #3)<br/> &gt; Lowering priority to critical as this is kind of corner case.</p> <p>Do we need a respin for this patch then? if not we should remove it from blocking bugs on the tracking spreadsheet.</p> <p>Can we confirm this is OK? (We think this should be OK now)</p> <p>Yes, I confirm this is fixed.</p> Development External issue ID 8805 External issue URL ODL SR Target Milestone Rank 0|i0344n: