OpenDaylight JIRA

OpenDaylight JIRA https://jira.opendaylight.org This file is an XML representation of an issue en-us

8.20.10

820010

22-06-2022

[OPNFLWPLUG-1018] OVS not able to connect to ODL - contextChain stuck in CLOSED state https://jira.opendaylight.org/browse/OPNFLWPLUG-1018 OpenFlowPlugin <p>We have seen sporadic failures where OVS on a compute node fails to connect to ODL when set-controller is executed on the compute node. These failures usually occur during deployment.</p> <p>When this failure occurs, OVS on the compute node continues attempting to connect to ODL. Openflowplugin immediately closes the new incoming connection, as it thinks the connection is in termination state. This cycle repeats endlessly once we are in this state.</p> <p> </p> <p>Since this is a random, intermittent, sorta difficult to reproduce bug (and would take forever having to try and hit this by having to redeploy every time), I wrote a small test script to reproduce this issue. Script is attached as test.sh.</p> <p>I enabled TRACE logs for openflowplugin.impl.lifecycle, and was able to reproduce the issue. I've been looking at the logs and code, but would be really great if someone on the openflowplugin team could jump in and take a look.</p> <p>I've included a snippet of some relevant karaf and ovs-vswitchd logs below. Full karaf log and ovs-vswitchd logs are attached. </p> <p>The timestamp of interest in the karaf logs starts around timestamp 2018-06-08T22:32:11,465, which is line number 4661.</p> <p> </p> <p><ins><b>Setup</b></ins></p> <p>Devstack setup, with 1 control node (VM) and 2 compute nodes (VMs) on 1 bm server.</p> <p>Devstack is stable/queens.</p> <p>ODL is stable/oxygen.</p> <p> </p> <p><ins><b>Karaf logs</b></ins></p> <p>2018-06-08T22:32:11,465 | WARN | epollEventLoopGroup-9-6 | ContextChainHolderImpl | 401 - org.opendaylight.openflowplugin.impl - 0.7.0.S<br/> NAPSHOT | Device openflow:220918713701936 is already in termination state, closing all incoming connections. <br/> 2018-06-08T22:32:11,870 | INFO | epollEventLoopGroup-9-7 | AbstractConnectionAdapter | 410 - org.opendaylight.openflowplugin.openflowjava.o<br/> penflow-protocol-impl - 0.7.0.SNAPSHOT | The channel outbound queue size:1024 <br/> 2018-06-08T22:32:11,871 | INFO | epollEventLoopGroup-9-7 | ConnectionAdapterImpl | 410 - org.opendaylight.openflowplugin.openflowjava.o<br/> penflow-protocol-impl - 0.7.0.SNAPSHOT | Hello received <br/> 2018-06-08T22:32:11,873 | INFO | epollEventLoopGroup-9-7 | ContextChainHolderImpl | 401 - org.opendaylight.openflowplugin.impl - 0.7.0.S<br/> NAPSHOT | Device openflow:220918713701936 connected. <br/> 2018-06-08T22:32:11,873 | WARN | epollEventLoopGroup-9-7 | ContextChainHolderImpl | 401 - org.opendaylight.openflowplugin.impl - 0.7.0.S<br/> NAPSHOT | Device openflow:220918713701936 is already in termination state, closing all incoming connections.</p> <p>2018-06-08T22:32:13,868 | INFO | epollEventLoopGroup-9-8 | AbstractConnectionAdapter | 410 - org.opendaylight.openflowplugin.openflowjava.o<br/> penflow-protocol-impl - 0.7.0.SNAPSHOT | The channel outbound queue size:1024 <br/> 2018-06-08T22:32:13,869 | INFO | epollEventLoopGroup-9-8 | ConnectionAdapterImpl | 410 - org.opendaylight.openflowplugin.openflowjava.o<br/> penflow-protocol-impl - 0.7.0.SNAPSHOT | Hello received <br/> 2018-06-08T22:32:13,872 | INFO | epollEventLoopGroup-9-8 | ContextChainHolderImpl | 401 - org.opendaylight.openflowplugin.impl - 0.7.0.S<br/> NAPSHOT | Device openflow:220918713701936 connected. <br/> 2018-06-08T22:32:13,873 | WARN | epollEventLoopGroup-9-8 | ContextChainHolderImpl | 401 - org.opendaylight.openflowplugin.impl - 0.7.0.S<br/> NAPSHOT | Device openflow:220918713701936 is already in termination state, closing all incoming connections.</p> <p> </p> <p><ins><b>ovs-vswitchd logs</b></ins></p> <p>2018-06-08T23:04:42.855Z|13767|rconn|INFO|br-int<->tcp:10.8.125.230:6653: connected <br/> 2018-06-08T23:04:42.858Z|13768|rconn|INFO|br-int<->tcp:10.8.125.230:6653: connection closed by peer <br/> 2018-06-08T23:04:50.854Z|13769|rconn|INFO|br-int<->tcp:10.8.125.230:6653: connected <br/> 2018-06-08T23:04:50.857Z|13770|rconn|INFO|br-int<->tcp:10.8.125.230:6653: connection closed by peer <br/> 2018-06-08T23:04:58.854Z|13771|rconn|INFO|br-int<->tcp:10.8.125.230:6653: connected <br/> 2018-06-08T23:04:58.857Z|13772|rconn|INFO|br-int<->tcp:10.8.125.230:6653: connection closed by peer <br/> 2018-06-08T23:05:06.855Z|13773|rconn|INFO|br-int<->tcp:10.8.125.230:6653: connected <br/> 2018-06-08T23:05:06.858Z|13774|rconn|INFO|br-int<->tcp:10.8.125.230:6653: connection closed by peer <br/> 2018-06-08T23:05:14.853Z|13775|rconn|INFO|br-int<->tcp:10.8.125.230:6653: connected <br/> 2018-06-08T23:05:14.856Z|13776|rconn|INFO|br-int<->tcp:10.8.125.230:6653: connection closed by peer <br/> 2018-06-08T23:05:22.854Z|13777|rconn|INFO|br-int<->tcp:10.8.125.230:6653: connected <br/> 2018-06-08T23:05:22.856Z|13778|rconn|INFO|br-int<->tcp:10.8.125.230:6653: connection closed by peer <br/> 2018-06-08T23:05:30.855Z|13779|rconn|INFO|br-int<->tcp:10.8.125.230:6653: connected <br/> 2018-06-08T23:05:30.858Z|13780|rconn|INFO|br-int<->tcp:10.8.125.230:6653: connection closed by peer <br/> 2018-06-08T23:05:38.853Z|13781|rconn|INFO|br-int<->tcp:10.8.125.230:6653: connected <br/> 2018-06-08T23:05:38.855Z|13782|rconn|INFO|br-int<->tcp:10.8.125.230:6653: connection closed by peer <br/> 2018-06-08T23:05:46.855Z|13783|rconn|INFO|br-int<->tcp:10.8.125.230:6653: connected</p> <p> </p> <p> </p> <p> </p> <p> </p> OPNFLWPLUG-1018

OVS not able to connect to ODL - contextChain stuck in CLOSED state

Bug High Resolved Done Anil Vishnoi Victor Pickard Fri, 8 Jun 2018 19:51:52 +0000 Mon, 25 Jun 2018 17:45:15 +0000 Mon, 25 Jun 2018 17:45:15 +0000 Oxygen Fluorine openflowplugin-impl 0 2 <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=vpickard" class="user-hover" rel="vpickard">vpickard</a> <a href="https://jira.opendaylight.org/browse/OPNFLWPLUG-970" title="Unable to establish connections with switches after frequent connecting and disconnecting" class="issue-link" data-issue-key="OPNFLWPLUG-970"><del>OPNFLWPLUG-970</del></a> is similar issue and there are patches raised for that isuse for stable/carbon. We will cherry-pick it to stable/oxygen and probably you can test with those patch to see if this resolves the issue.</p> <p>Anil,</p> <p>Thanks. I will test with the new changes once they are cherry-picked.</p> <p>Just FYI, I also saw the ConcurrentModificationException as noted in <a href="https://jira.opendaylight.org/browse/OPNFLWPLUG-970" class="external-link" rel="nofollow">https://jira.opendaylight.org/browse/OPNFLWPLUG-970</a>, so would need </p> <p><a href="https://git.opendaylight.org/gerrit/#/c/70093/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/70093/</a></p> <p>to be cherry-picked as well.</p> <p> </p> <p>Thanks,</p> <p>Vic</p> <p> </p> <p>I cherry-picked the patches and did some local testing. I was not able to reproduce this issue with these patches, looks good. Thanks Anil!</p> <p> </p> <p>Vic</p> <p> </p> <p>Attached updated version of test script (test2.sh). I noticed that sometimes, after doing a set-controller, it took a little longer than usual for ovs to connect to odl. Updated script to account for those extra few seconds.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=vpickard" class="user-hover" rel="vpickard">vpickard</a> I think this issue is resolved? Can you open another Jira to trace the issue you mentioned and can you please provide the logs as well.</p> <p>Anil,</p> <p>Yes, this issue is resolved with the 2 patches. </p> <p>The issue I mentioned (Exception) was also resolved by the patch, so no new issue to open Jira for.</p> <p>Should be good to mark this bug as resolved.</p> <p> </p> <p>Thanks</p> Duplicate OPNFLWPLUG-970 Development Rank 0|i03fhj: