https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/OPNFLWPLUG-1065 OpenFlowPlugin <p>curl -u admin:admin <a href="http://192.168.0.5:8181/diagstatus" class="external-link" target="_blank" rel="nofollow noopener">http://192.168.0.5:8181/diagstatus</a> shows openflowplugin wasn't up successfully, here are some error information.</p> <p> "statusSummary": [<br/> </p> { "serviceName": "OPENFLOW", "effectiveStatus": "ERROR", "reportedStatusDescription": "OF::PORTS:: 6653 and 6633 are not up yet", "statusTimestamp": "2019-03-04T01:54:44.411Z", "errorCause": "" } <p>,</p> <p>Opendaylight Fluorine SR1 + Openstack Rocky</p> OPNFLWPLUG-1065

openflowplugin failed to initialize because of dependency issue

Bug Highest Resolved Duplicate Anil Vishnoi Yi Yang Mon, 4 Mar 2019 02:12:18 +0000 Mon, 6 May 2019 15:25:51 +0000 Sun, 5 May 2019 00:15:28 +0000 0 4 <p>the attachment karaf.log.error-for-1065 is karaf log, I have encountered this many times, but restarting it can fix issue, I doubt it is initialization order issue of depended features.</p> <p>Looks like your cluster setup is not coming up properly. Initially when 192.168.0.5 node comes, it tries to reach .6 and .7 machine and the connection is refused, so node started as a singleton cluster (pretty much useless)</p> <p> </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> 2019-03-04T01:28:19,714 | INFO  | opendaylight-cluster-data-akka.actor.<span class="code-keyword">default</span>-dispatcher-2 | Cluster(akka:<span class="code-comment">//opendaylight-cluster-data) | 83 - com.typesafe.akka.slf4j - 2.5.11 | Cluster Node [akka.tcp://opendaylight-cluster-data@192.168.0.5:2550] - Started up successfully </span>2019-03-04T01:28:19,921 | WARN  | opendaylight-cluster-data-akka.actor.<span class="code-keyword">default</span>-dispatcher-18 | NettyTransport                   | 83 - com.typesafe.akka.slf4j - 2.5.11 | Remote connection to [<span class="code-keyword">null</span>] failed with java.net.ConnectException: Connection refused: /192.168.0.7:2550 2019-03-04T01:28:19,927 | WARN  | opendaylight-cluster-data-akka.actor.<span class="code-keyword">default</span>-dispatcher-18 | NettyTransport                   | 83 - com.typesafe.akka.slf4j - 2.5.11 | Remote connection to [<span class="code-keyword">null</span>] failed with java.net.ConnectException: Connection refused: /192.168.0.6:2550 2019-03-04T01:28:19,939 | WARN  | opendaylight-cluster-data-akka.actor.<span class="code-keyword">default</span>-dispatcher-18 | ReliableDeliverySupervisor       | 83 - com.typesafe.akka.slf4j - 2.5.11 | Association with remote system [akka.tcp:<span class="code-comment">//opendaylight-cluster-data@192.168.0.6:2550] has failed, address is now gated <span class="code-keyword">for</span> [5000] ms. Reason: [Association failed with [akka.tcp://opendaylight-cluster-data@192.168.0.6:2550]] Caused by: [Connection refused: /192.168.0.6:2550] </span>2019-03-04T01:28:19,941 | WARN  | opendaylight-cluster-data-akka.actor.<span class="code-keyword">default</span>-dispatcher-18 | ReliableDeliverySupervisor       | 83 - com.typesafe.akka.slf4j - 2.5.11 | Association with remote system [akka.tcp:<span class="code-comment">//opendaylight-cluster-data@192.168.0.7:2550] has failed, address is now gated <span class="code-keyword">for</span> [5000] ms. Reason: [Association failed with [akka.tcp://opendaylight-cluster-data@192.168.0.7:2550]] Caused by: [Connection refused: /192.168.0.7:2550]</span> </pre> </div></div> <p> </p> <p> </p> <p>and after around 5:30 minutes, node 0.6 become reachable </p> <p> </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> 2019-03-04T01:33:50,275 | INFO  | opendaylight-cluster-data-akka.actor.<span class="code-keyword">default</span>-dispatcher-44 | Cluster(akka:<span class="code-comment">//opendaylight-cluster-data) | 83 - com.typesafe.akka.slf4j - 2.5.11 | Cluster Node [akka.tcp://opendaylight-cluster-data@192.168.0.5:2550] - Received InitJoin message from [Actor[akka.tcp://opendaylight-cluster-data@192.168.0.6:2550/system/cluster/core/daemon/joinSeedNodeProcess-1#1720025309]] to [akka.tcp://opendaylight-cluster-data@192.168.0.5:2550] </span>2019-03-04T01:33:50,276 | INFO  | opendaylight-cluster-data-akka.actor.<span class="code-keyword">default</span>-dispatcher-44 | Cluster(akka:<span class="code-comment">//opendaylight-cluster-data) | 83 - com.typesafe.akka.slf4j - 2.5.11 | Cluster Node [akka.tcp://opendaylight-cluster-data@192.168.0.5:2550] - Sending InitJoinAck message from node [akka.tcp://opendaylight-cluster-data@192.168.0.5:2550] to [Actor[akka.tcp://opendaylight-cluster-data@192.168.0.6:2550/system/cluster/core/daemon/joinSeedNodeProcess-1#1720025309]] </span>2019-03-04T01:33:50,452 | INFO  | opendaylight-cluster-data-akka.actor.<span class="code-keyword">default</span>-dispatcher-44 | Cluster(akka:<span class="code-comment">//opendaylight-cluster-data) | 83 - com.typesafe.akka.slf4j - 2.5.11 | Cluster Node [akka.tcp://opendaylight-cluster-data@192.168.0.5:2550] - Node [akka.tcp://opendaylight-cluster-data@192.168.0.6:2550] is JOINING, roles [member-2, dc-<span class="code-keyword">default</span>]</span> </pre> </div></div> <p>and than initialization of mdsal, aaa and eventually openflowplugin triggers, but seems like some of the components are still not initialized properly yet.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> 2019-03-04T01:34:15,330 | INFO  | Blueprint Extender: 3 | OpenFlowPluginProviderFactoryImpl | 370 - org.opendaylight.openflowplugin.impl - 0.7.1 | Initializing <span class="code-keyword">new</span> OFP southbound.</pre> </div></div> <p>Looks like once diagstats give up after some timeout, it does not retry to check (or don't get notifications to retry in case bundle initialization happens with delay) that status again. But irrespective of that, controller goes in the bad state because controller didn't converge initially because of the non availability of other nodes in the cluster. So it's not really openflowplugin issue, but related to the proper cluster initialization.</p> <p> </p> <p> </p> <p>Anil, thank you so much for your comments, yes, it is cluster-related issue, so whom do you think it should be reassigned to?</p> <p>Hi Yi,</p> <p> </p> <p>I would suggest to first check your environment to see why controllers 192.168.0.6/7 are not reachable to 192.168.0.5, that seems to be the root cause of the issue. I have seen that you are creating br-int on the same node, i am wondering if your OVSDB configuration on the 192.168.0.6/7 is interfering with the networking between your cluster VMs. In most of the cases i have seen that if CDS don't initialize properly, the only option you have is restart your cluster.</p> <p>Anil, this is an error ha_l2.robot resulted in, I can make the cluster work normally by restarting every cluster node in order, but this isn't what we expect.</p> <p>In reality, such situation maybe often happens, so we think akka-based clustering isn't reliable, why can't it fix this automatically? </p> <p>Actually ha_l2.robot is very friendly, Robert said MDSAL cluster has more rigorous than this, but it can work normally, do you know which robot script can torture MDSAL cluster ?</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=yangyi01" class="user-hover" rel="yangyi01">yangyi01</a> I believe out of order restart issue is something akka's limitation and to fix this, i would suggest you open a ticket to the controller project.</p> <p>I don't really have any information regarding the script that can stress mdsal, probably md-sal project or integration/test project folks can help you better here.</p> <p>Thanks Anil, I'll open a new bug for controller and close this one, do you know how I can change this to controller project? or I have to re-create new one for it?</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=yangyi01" class="user-hover" rel="yangyi01">yangyi01</a> are you going to close this? what is the bug id in the controller project?</p> <p><a href="https://jira.opendaylight.org/browse/CONTROLLER-1892" class="external-link" rel="nofollow">https://jira.opendaylight.org/browse/CONTROLLER-1892</a> has been there, so close this one, this should be a controller-specific bug.</p> <p>Jamo, I created <a href="https://jira.opendaylight.org/browse/CONTROLLER-1892" class="external-link" rel="nofollow">https://jira.opendaylight.org/browse/CONTROLLER-1892</a> for it, so close this one.</p> <p>thanks!</p> Development Priority Rank 0|i03mvb: