https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-1906 controller <p>Regression was detected here:</p> <p><a href="https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-3node-clustering-only-sodium/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-3node-clustering-only-sodium/</a></p> <p>To reproduce just connect OVS switch to 3 controllers and file an RPC like this from all instances:<br/> POST http://<tt>controller</tt>:8181/restconf/operations/sal-flow:add-flow</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> &lt;?xml version=<span class="code-quote">"1.0"</span> encoding=<span class="code-quote">"UTF-8"</span> standalone=<span class="code-quote">"no"</span>?&gt; &lt;input xmlns=<span class="code-quote">"urn:opendaylight:flow:service"</span>&gt; &lt;node xmlns:inv=<span class="code-quote">"urn:opendaylight:inventory"</span>&gt;/inv:nodes/inv:node[inv:id=<span class="code-quote">"openflow:1"</span>]&lt;/node&gt; &lt;table_id&gt;0&lt;/table_id&gt; &lt;priority&gt;2&lt;/priority&gt; &lt;match&gt; &lt;ethernet-match&gt; &lt;ethernet-type&gt; &lt;type&gt;2048&lt;/type&gt; &lt;/ethernet-type&gt; &lt;/ethernet-match&gt; &lt;ipv4-destination&gt;10.0.1.0/24&lt;/ipv4-destination&gt; &lt;/match&gt; &lt;instructions&gt; &lt;instruction&gt; &lt;order&gt;0&lt;/order&gt; &lt;apply-actions&gt; &lt;action&gt; &lt;output-action&gt; &lt;output-node-connector&gt;1&lt;/output-node-connector&gt; &lt;/output-action&gt; &lt;order&gt;0&lt;/order&gt; &lt;/action&gt; &lt;/apply-actions&gt; &lt;/instruction&gt; &lt;/instructions&gt; &lt;/input&gt; </pre> </div></div> <p>At least 1 instance will complain with this message:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> &lt;errors xmlns=<span class="code-quote">"urn:ietf:params:xml:ns:yang:ietf-restconf"</span>&gt; &lt;error&gt; &lt;error-type&gt;application&lt;/error-type&gt; &lt;error-tag&gt;operation-failed&lt;/error-tag&gt; &lt;error-message&gt;The operation encountered an unexpected error <span class="code-keyword">while</span> executing.&lt;/error-message&gt; &lt;error-info&gt;Ask timed out on [Actor[akka.tcp:<span class="code-comment">//opendaylight-cluster-data@10.18.130.162:2550/user/rpc/broker#-516941188]] after [15000 ms]. Message of type [org.opendaylight.controller.remote.rpc.messages.ExecuteRpc]. A typical reason <span class="code-keyword">for</span> `AskTimeoutException` is that the recipient actor didn't send a reply.&lt;/error-info&gt; </span> &lt;/error&gt; &lt;/errors&gt; </pre> </div></div> CONTROLLER-1906

OFP RPC does not work from all instances in the cluster

Bug Highest Resolved Done Emmett Cox Luis Gomez Thu, 18 Jul 2019 21:25:34 +0000 Wed, 14 Aug 2019 10:32:23 +0000 Wed, 14 Aug 2019 10:32:23 +0000 Sodium Sodium clustering 0 3 <p>Candidate in this area is <a href="https://git.opendaylight.org/gerrit/#/c/82283/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/82283/</a> – <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=EmmettCox" class="user-hover" rel="EmmettCox">EmmettCox</a> can you take a look?</p> <p>Taking a look now - trying to recreate bug first, setting up environment, might be missing a few things - if anyone could point out some mistakes in my flow/logic, would be useful:</p> <p>-spun up 3 vms, each running an opendaylight controller and clustered together </p> <p>-installed feature odl-openflowplugin-flow-services-rest on each odl </p> <ul class="alternate" type="square"> <li>created a test-bridge on one of the vm's and added the 3 controllers to bridge:</li> </ul> <p>    sudo ovs-vsctl dd-br br-test </p> <p>    sudo ovs-vsctl set-controller br-test &lt; controller ips ports etc &gt; </p> <p>-ran above command through postman explorer, got an error back saying  No implementation of RPC AbsolouteSchemaPath &lt;SchemaPath&gt;add-flow available</p> <p> </p> <p>Think either I might be missing a feature install or might be configuring the bridge wrong... any ideas?</p> <p>Just to confirm the regression was introduced by the patch reverted here:</p> <p><a href="https://git.opendaylight.org/gerrit/c/controller/+/83212" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/c/controller/+/83212</a></p> <p>Any ETA for the fix? I believe OFP patch verification is impacted by this problem.</p> <p>Got a pointer to the verify job failing?</p> <p>Aside from that, I do not currently have the cycles over the next two weeks to test this out locally, but if I get some data at logs.opendaylight.org, I may be able to look into the cause. Can we get an CSIT run with sal-remoterpc-connector debugs store there?</p> <p>The failing job in the ticket description is a good example if a job used to verify patches:</p> <p><a href="https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-3node-clustering-only-sodium/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-3node-clustering-only-sodium/</a></p> <p>I can run the test with the debug on.</p> <p>This run has the DEBUG but I do not see anything when the test fails:</p> <p><a href="https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-3node-gate-clustering-only-sodium/104/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-3node-gate-clustering-only-sodium/104/</a></p> <p>got some logs from running csit tests in my local environment:</p> <p><span class="error">&#91;^karaf_debug.log&#93;</span></p> <p> </p> <p>WIll point out that i had to run the csit test tests twice before it failed - it succeeded perfectly the first time. Also from looking through the previous runs on jenkins the number of fail tests can range from 2-6, with only two tests failing consistently every time - makes it annoying to debug, and implies the problem is an inconsistent one...</p> <p>The specific error is listed below:</p> <p>2019-07-31T09:42:10,746 | TRACE | qtp533277569-75  | BrokerFacade                     | 298 - org.opendaylight.netconf.restconf-nb-bierman02 - 1.10.0.SNAPSHOT | Read OPERATIONAL via Restconf: /(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)network-topology<br/> 2019-07-31T09:42:11,423 | WARN  | opendaylight-cluster-data-akka.actor.default-dispatcher-30 | NettyTransport                   | 88 - com.typesafe.akka.slf4j - 2.5.21 | Remote connection to <span class="error">&#91;null&#93;</span> failed with java.net.ConnectException: Connection refused: /192.168.67.48:2550<br/> 2019-07-31T09:42:11,431 | WARN  | opendaylight-cluster-data-akka.actor.default-dispatcher-30 | ReliableDeliverySupervisor       | 88 - com.typesafe.akka.slf4j - 2.5.21 | Association with remote system <span class="error">&#91;akka.tcp://opendaylight-cluster-data@192.168.67.48:2550&#93;</span> has failed, address is now gated for <span class="error">&#91;5000&#93;</span> ms. Reason: [Association failed with <span class="error">&#91;akka.tcp://opendaylight-cluster-data@192.168.67.48:2550&#93;</span>] Caused by: <span class="error">&#91;java.net.ConnectException: Connection refused: /192.168.67.48:2550&#93;</span><br/> 2019-07-31T09:42:12,458 | DEBUG | qtp533277569-68  | RestconfImpl                     | 298 - org.opendaylight.netconf.restconf-nb-bierman02 - 1.10.0.SNAPSHOT | Execution RpcError: <br/> java.util.concurrent.ExecutionException: akka.pattern.AskTimeoutException: Ask timed out on [Actor<span class="error">&#91;akka.tcp://opendaylight-cluster-data@192.168.67.48:2550/user/rpc/broker#1272405160&#93;</span>] after <span class="error">&#91;15000 ms&#93;</span>. Message of type <span class="error">&#91;org.opendaylight.controller.remote.rpc.messages.ExecuteRpc&#93;</span>. A typical reason for `AskTimeoutException` is that the recipient actor didn't send a reply.</p> <p> </p> <p> </p> <p>I wonder about the warning regarding the Connection refused for one of the nodes.... I've noticed the same warning appear for some of the other tests that succeed, but those tests take a few seconds longer to execute.... could it simply be akka timing out when it' would work if given a few extra seconds? not that it should take so long, but...</p> <p>discovered that I was missing the debug option for remote rpc logs, so there's a little bit more being logged now....</p> <p>going to include the logs from all 3 nodes, give me a min to add them.... </p> <p> </p> <p>Discovered the root of the issue is to do with the OpsRegistrar changes made as part of my commit.</p> <p>Part of the changes removed functionality that removed and closed old rpc registrations, which caused the rpc's to not update correctly and fail when a node was shutdown.</p> <p>I'm in the midst of making some code changes to fix the bug.</p> <p>Hi Emmett,</p> <p>FYI, I started a verification on this patch, considering this is the candidate fix:</p> <p><a href="https://git.opendaylight.org/gerrit/c/controller/+/83530" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/c/controller/+/83530</a></p> <p>I think the issue is fixed now: <a href="https://jenkins.opendaylight.org/sandbox/job/openflowplugin-csit-3node-clustering-only-sodium/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/sandbox/job/openflowplugin-csit-3node-clustering-only-sodium/</a></p> Development Rank 0|i03ofz: