https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/OPNFLWPLUG-970 OpenFlowPlugin <p>My environment has three physical switches connecting to the opendaylight controller of carbon-sr2.</p> <p>After keeping making two of the switches connected and disconnected frequently for a while, the opendaylight controller couldn't set up the connections with switches any longer. Please see the attachment for karaf log.</p> <p> </p> <p>One fact is that everything started to go wrong when the following log message showed up:</p> <p>2018-01-08 14:21:35,601 | WARN | entLoopGroup-7-8 | ContextChainHolderImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.2.Carbon | Device openflow:35776560030 is already in termination state, closing all incoming connections.</p> <p> </p> <p>Then all connections will be closed after that like this:</p> <p>2018-01-08 14:21:35,601 | INFO | entLoopGroup-7-8 | ContextChainHolderImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.2.Carbon | Device openflow:35776560030 connected.<br/> 2018-01-08 14:21:35,601 | WARN | entLoopGroup-7-8 | ContextChainHolderImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.2.Carbon | Device openflow:35776560030 is already in termination state, closing all incoming connections.</p> <p> </p> <p>Here is part of the log (filter by openflow:35776560030) when it went wrong:</p> <p>2018-01-08 14:19:24,615 | WARN | lt-dispatcher-19 | ContextChainImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.2.Carbon | Not able to start clustering services for node openflow:35776560030<br/> 2018-01-08 14:19:24,615 | WARN | ofppool-2 | ContextChainHolderImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.2.Carbon | Not able to set MASTER role on device openflow:35776560030, reason: Failed to initialize device openflow:35776560030 in 9s: java.util.concurrent.TimeoutException: Timeout waiting for task.<br/> 2018-01-08 14:19:24,615 | WARN | ofppool-2 | ContextChainHolderImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.2.Carbon | This mastering is mandatory, destroying context chain and closing connection for device openflow:35776560030.<br/> 2018-01-08 14:19:24,616 | INFO | ofppool-2 | ContextChainImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.2.Carbon | Closed clustering services registration for node openflow:35776560030<br/> 2018-01-08 14:19:24,621 | INFO | ult-dispatcher-3 | ContextChainImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.2.Carbon | Closing clustering services for node openflow:35776560030<br/> 2018-01-08 14:19:24,621 | INFO | ult-dispatcher-3 | StatisticsContextImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.2.Carbon | Stopping statistics context cluster services for node openflow:35776560030<br/> 2018-01-08 14:19:24,621 | INFO | ult-dispatcher-3 | StatisticsContextImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.2.Carbon | Stopping running statistics gathering for node openflow:35776560030<br/> 2018-01-08 14:19:24,621 | INFO | ult-dispatcher-3 | RpcContextImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.2.Carbon | Stopping rpc context cluster services for node openflow:35776560030<br/> 2018-01-08 14:19:24,621 | INFO | ult-dispatcher-3 | DeviceContextImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.2.Carbon | Stopping device context cluster services for node openflow:35776560030<br/> 2018-01-08 14:19:24,622 | INFO | ult-dispatcher-3 | ContextChainImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.2.Carbon | Closed clustering services for node openflow:35776560030<br/> 2018-01-08 14:19:24,625 | DEBUG | lt-dispatcher-20 | ContextChainHolderImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.2.Carbon | Entity openflow:35776560030 has no owner<br/> 2018-01-08 14:19:24,625 | INFO | lt-dispatcher-20 | ContextChainHolderImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.2.Carbon | Removing device Uri <span class="error">&#91;_value=openflow:35776560030&#93;</span> from operational DS<br/> 2018-01-08 14:19:24,626 | DEBUG | CommitFutures-1 | DeviceManagerImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.2.Carbon | Delete Node KeyedInstanceIdentifier{targetType=interface org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node, path=[org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.Nodes, org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node[key=NodeKey [_id=Uri <span class="error">&#91;_value=openflow:35776560030&#93;</span>]]]} was successful<br/> 2018-01-08 14:19:32,642 | INFO | ofppool-2 | SystemNotificationsListenerImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.2.Carbon | ConnectionEvent:Closing connection as device is idle. Echo sent at Mon Jan 08 14:19:30 CST 2018. Device:/192.168.1.103:29990, NodeId:openflow:35776560030<br/> 2018-01-08 14:19:32,642 | DEBUG | ofppool-2 | ConnectionContextImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.2.Carbon | Actively closing connection: device=openflow:35776560030 | auxiliaryId=0 | connectionState=RIP<br/> 2018-01-08 14:19:32,642 | DEBUG | ofppool-2 | ConnectionContextImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.2.Carbon | Trying unregister outbound queue handler registration for node openflow:35776560030<br/> 2018-01-08 14:19:32,642 | DEBUG | ofppool-2 | ConnectionContextImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.2.Carbon | Trying closing handshake context for node openflow:35776560030<br/> 2018-01-08 14:19:32,642 | DEBUG | ofppool-2 | ConnectionContextImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.2.Carbon | Propagating connection closed event: /192.168.1.103:29990, datapathId:35776560030.<br/> 2018-01-08 14:19:32,643 | INFO | ofppool-2 | ContextChainHolderImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.2.Carbon | Device openflow:35776560030 disconnected.<br/> 2018-01-08 14:19:32,644 | DEBUG | ofppool-2 | ContextChainImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.2.Carbon | ContextChain for node openflow:35776560030 is already in TERMINATION state.<br/> 2018-01-08 14:21:35,601 | DEBUG | entLoopGroup-7-8 | HandshakeListenerImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.2.Carbon | succeeded by getting sweep barrier after post-handshake for device openflow:35776560030<br/> 2018-01-08 14:21:35,601 | INFO | entLoopGroup-7-8 | ContextChainHolderImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.2.Carbon | Device openflow:35776560030 connected.<br/> 2018-01-08 14:21:35,601 | WARN | entLoopGroup-7-8 | ContextChainHolderImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.2.Carbon | Device openflow:35776560030 is already in termination state, closing all incoming connections.<br/> 2018-01-08 14:21:35,601 | DEBUG | entLoopGroup-7-8 | ConnectionContextImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.2.Carbon | Actively closing connection: device=openflow:35776560030 | auxiliaryId=0 | connectionState=RIP<br/> 2018-01-08 14:21:35,601 | DEBUG | entLoopGroup-7-8 | ConnectionContextImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.2.Carbon | Trying unregister outbound queue handler registration for node openflow:35776560030<br/> 2018-01-08 14:21:35,601 | DEBUG | entLoopGroup-7-8 | ConnectionContextImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.2.Carbon | Trying closing handshake context for node openflow:35776560030<br/> 2018-01-08 14:21:43,691 | DEBUG | ntLoopGroup-7-10 | HandshakeListenerImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.2.Carbon | succeeded by getting sweep barrier after post-handshake for device openflow:35776560030<br/> 2018-01-08 14:21:43,692 | INFO | ntLoopGroup-7-10 | ContextChainHolderImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.2.Carbon | Device openflow:35776560030 connected.<br/> 2018-01-08 14:21:43,692 | WARN | ntLoopGroup-7-10 | ContextChainHolderImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.2.Carbon | Device openflow:35776560030 is already trying to connect, wait until succeeded or disconnected.<br/> 2018-01-08 14:21:43,692 | DEBUG | ntLoopGroup-7-10 | ConnectionContextImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.2.Carbon | Actively closing connection: device=openflow:35776560030 | auxiliaryId=0 | connectionState=RIP<br/> 2018-01-08 14:21:43,692 | DEBUG | ntLoopGroup-7-10 | ConnectionContextImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.2.Carbon | Trying unregister outbound queue handler registration for node openflow:35776560030<br/> 2018-01-08 14:21:43,692 | DEBUG | ntLoopGroup-7-10 | ConnectionContextImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.2.Carbon | Trying closing handshake context for node openflow:35776560030<br/> 2018-01-08 14:21:51,812 | DEBUG | ntLoopGroup-7-12 | HandshakeListenerImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.2.Carbon | succeeded by getting sweep barrier after post-handshake for device openflow:35776560030<br/> 2018-01-08 14:21:51,812 | INFO | ntLoopGroup-7-12 | ContextChainHolderImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.2.Carbon | Device openflow:35776560030 connected.<br/> 2018-01-08 14:21:51,813 | WARN | ntLoopGroup-7-12 | ContextChainHolderImpl | 220 - org.opendaylight.openflowplugin.impl - 0.4.2.Carbon | Device openflow:35776560030 is already trying to connect, wait until succeeded or disconnected.</p> <p> </p> <p>I tried to analyze it, it seems sometimes the failure in setting master role on switches would cause this issue.</p> <p>The pattern is like: Failure in setting master role -&gt; destroying context chain -&gt; Closing connection as device is idle. -&gt; disconnected -&gt; conencted -&gt; already in termination state -&gt; connected -&gt; already trying to connect -&gt; connected -&gt; already trying to connect -&gt; ...</p> <p>Operating System: Ubuntu 14.04<br/> Platform: PC</p> OPNFLWPLUG-970

Unable to establish connections with switches after frequent connecting and disconnecting

Bug Highest Resolved Done Chien-Hsin Chen Chien-Hsin Chen Mon, 8 Jan 2018 07:12:52 +0000 Tue, 19 Jun 2018 10:04:06 +0000 Tue, 19 Jun 2018 10:04:06 +0000 Carbon-SR3 Carbon-SR2 Oxygen openflowplugin-impl 0 6 <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jamescch" class="user-hover" rel="jamescch">jamescch</a> Are you planning to work on this issue?</p> <p>I'm afraid I can't because of limited knowledge on openflowplugin.</p> <p>But I can assist in debugging.</p> <p>I am able to reproduce this issue with just one switch disconnect and reconnect. It doesn't happen all the time, but once it happen, the state cannot recover. I will look into this issue.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=kevinxw" class="user-hover" rel="kevinxw">kevinxw</a> I have found the root cause. I'm going to fix this.</p> <p>Do you mind assigning back to me?</p> <p>Please go ahead. I am busy with other task recently and haven't really made much progress so far.</p> <p>Change has been opened <a href="https://git.opendaylight.org/gerrit/#/c/70093/" class="external-link" target="_blank" rel="nofollow noopener">here</a>.</p> <p>Experiencing what appears to be the same issue with Nitrogen:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> 2018-04-04 22:03:14,179 | WARN | ntLoopGroup-16-7 | ContextChainHolderImpl | 305 - org.opendaylight.openflowplugin.impl - 0.5.1 | Device openflow:218639756982094 already connected. Closing previous connection 2018-04-04 22:03:16,412 | WARN | ntLoopGroup-16-4 | ContextChainImpl | 305 - org.opendaylight.openflowplugin.impl - 0.5.1 | Not able to start clustering services <span class="code-keyword">for</span> node openflow:231865136161864 2018-04-04 22:03:16,414 | WARN | ofppool-11 | ContextChainHolderImpl | 305 - org.opendaylight.openflowplugin.impl - 0.5.1 | Not able to set MASTER role on device openflow:231865136161864, reason: java.lang.RuntimeException: Failed to initialize device openflow:231865136161864 in 9s: java.util.concurrent.TimeoutException 2018-04-04 22:03:16,414 | WARN | ofppool-11 | ContextChainHolderImpl | 305 - org.opendaylight.openflowplugin.impl - 0.5.1 | This mastering is mandatory, destroying context chain and closing connection <span class="code-keyword">for</span> device openflow:231865136161864. </pre> </div></div> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jamescch" class="user-hover" rel="jamescch">jamescch</a>, what is the root cause of this issue?</p> <p>The root cause is that when destoryContextChain in class ContextChainHolderImpl is called, class DeviceContextImpl will try to iterate all requestContexts it has created in a collection to close them.</p> <p>However, those requestContexts will remove themselves from the collection.</p> <p>And this caused the CurrentModificationException and the thread was aborted.</p> <p>The status in ContextChainImpl for this switch remained CLOSING and changed to ALREADY_CONNECTED later and blocked all incoming connections.</p> <p>And the reason why we didn't see the error in the karaf console is that thread is executed using ExecutorService.submit() which didn't expose the exception.</p> <p>I uploaded the stack trace so that you can see how it goes into this situation.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jamescch" class="user-hover" rel="jamescch">jamescch</a> Can you please cherry-pick the changes to stable/oxygen and master branch as well. We have seen similar issue with those branches as well.</p> Duplicate OPNFLWPLUG-1018 Development Rank 0|i039bz: