https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-1848 controller <p>The suite seems to kill and restart the "brt" (not sure what brt means yet) owner and at that point there is<br/> some trouble (501 response) when trying to hit the "/restconf/operations/basic-rpc-test:basic-global"<br/> endpoint.</p> <p>the response text is:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> {<span class="code-quote">"errors"</span>:{<span class="code-quote">"error"</span>:[{<span class="code-quote">"error-type"</span>:<span class="code-quote">"application"</span>,<span class="code-quote">"error-tag"</span>:<span class="code-quote">"operation-not-supported"</span>,<span class="code-quote">"error-message"</span>:<span class="code-quote">"No implementation of RPC AbsoluteSchemaPath{path=[(urn:opendaylight:controller:basic-rpc-test?revision=2016-01-20)basic-global]} available"</span>}]}} </pre> </div></div> CONTROLLER-1848

Sporadic CSIT failure in RPC tests (501 when hitting /restconf/operations/basic-rpc-test:basic-global)

Bug Medium Resolved Duplicate Tom Pantelis Victor Pickard csit:3node Tue, 3 Jul 2018 14:36:37 +0000 Wed, 1 Aug 2018 12:22:43 +0000 Mon, 30 Jul 2018 17:50:58 +0000 0 4 <p>That message is normal from akka - it's reporting a node is unreachable which I assume is part of the test. </p> <p>as I commented in the kernel call, this jira was opened because of a failing test. I think <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=vpickard" class="user-hover" rel="vpickard">vpickard</a> was just trying to find some<br/> more specific reason for the failure (to be helpful) and not neccessarily calling out that error message.</p> <p>maybe it will be better to use more generic jira summaries like "CSIT test case XYZ intermittently failing" with some comments<br/> showing certain suspect logs like "Leader can not perform its duties" so we can keep better track of our issues? just an idea,<br/> but I'm not sure if this one getting closed resulted in a new one being open, because I doubt the intermittent test failure was<br/> addressed yet.</p> <p>Sure. You can reopen this and change summary etc or open a new one - doesn't really matter. I thought this was created b/c you saw that ominous message in the log and thought it was an issue.</p> <p>here is the original bug description for posterity</p> <blockquote> <p>Controller clustering CSIT is failing intermittently, I see this in the karaf logs 7 times for this particular run <span class="error">&#91;1&#93;</span></p> <p>2018-07-03T06:49:24,867 | INFO | opendaylight-cluster-data-akka.actor.default-dispatcher-33 | Cluster(akka://opendaylight-cluster-data) | 51 - com.typesafe.akka.slf4j - 2.5.11 | Cluster Node <span class="error">&#91;akka.tcp://opendaylight-cluster-data@10.30.170.209:2550&#93;</span> - Leader can currently not perform its duties, reachability status: [akka.tcp://opendaylight-cluster-data@10.30.170.209:2550 -&gt; akka.tcp://opendaylight-cluster-data@10.30.170.221:2550: Unreachable <span class="error">&#91;Unreachable&#93;</span> (1), akka.tcp://opendaylight-cluster-data@10.30.170.220:2550 -&gt; akka.tcp://opendaylight-cluster-data@10.30.170.221:2550: Unreachable <span class="error">&#91;Unreachable&#93;</span> (1)], member status: <span class="error">&#91;akka.tcp://opendaylight-cluster-data@10.30.170.209:2550 Up seen=true, akka.tcp://opendaylight-cluster-data@10.30.170.220:2550 Up seen=true, akka.tcp://opendaylight-cluster-data@10.30.170.221:2550 Up seen=false&#93;</span></p> <p><span class="error">&#91;1&#93;</span> <a href="https://jenkins.opendaylight.org/sandbox/job/controller-csit-3node-clustering-vpickard-all-oxygen/13/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/sandbox/job/controller-csit-3node-clustering-vpickard-all-oxygen/13/</a></p></blockquote> <p>thanks <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=tpantelis" class="user-hover" rel="tpantelis">tpantelis</a>, I just repopened this jira. the description had a link to the failing job, so I use that to<br/> add a few details.</p> <p>we need to keep digging from here.</p> <p>Archived logs since this is a sandbox job</p> <p> </p> <p><a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/163/controller-csit-3node-clustering-vpickard-all-oxygen/13/" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/163/controller-csit-3node-clustering-vpickard-all-oxygen/13/</a></p> <p>BRT stands for "Basic RPC Test" <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>We need to understand what the test does and need the log files in order to do post-mortem analysis.</p> <p>The archived logs are here:</p> <p><a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/163/controller-csit-3node-clustering-vpickard-all-oxygen/13/" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/163/controller-csit-3node-clustering-vpickard-all-oxygen/13/</a></p> <p>So the "old owner" was odl3. There are no errors in the odl3 karaf log after restart around 2018-07-03T06:50:20. In fact there's no log messages at all after 6:52:28 - the robot log.html indicates the basic-global RPC was sent around 6:54. We need to enable debug for org.opendaylight.controller.remote.rpc and try to reproduce.</p> <p>If possible, in the Restart_Killed_Member step, it would be helpful to save off the karaf.log and delete it so the node starts with a clean log. </p> <p>Looking at the Run_Rpc keyword, it issues the RPC once. When a node restarts, it takes a bit of time for it to learn about RPC registrations from other nodes via the gossip functionality. So I suspect the test issued the RPC before odl3 knew about the remote RPC implementation, hence the 501 "No implementation" error. So the test should retry via Wait Until Keyword Succeeds. </p> <p>what's a reasonable time for this though. looking at the log <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=vpickard" class="user-hover" rel="vpickard">vpickard</a> posted in the comment above, the<br/> killed memeber is restarted, and it polls for ~2 minutes until syncstatus is true, then it's another 1.5m before<br/> the RPC is issued. So, 3.5m from the time of restart, but probably more importantly, 1.5m since the cluster<br/> sync was said to be ok.</p> <p>we can certainly add a WUKS to that step, but it seems like it's already happening quite a long time later<br/> for a cluster member to not know about the RPC. right?</p> <p>I didn't notice 1.5m passed. That should be plenty of time. Does it clean the data dir before restarting? If so, then it's re-installing the world so that can take time. It was just an idea...</p> <p>We need a failure run with org.opendaylight.controller.remote.rpc debug enabled. Maybe pull out that one suite and run over and over in the sandbox or whatever?</p> <p>The few times I've seen this reproduced it seems that it is caused by 2/3 of the nodes being down or otherwise unreachable. Manual explicit tests of this have produced the same outcome - when 2/3 nodes are down (or in "deadlock") rpc calls return 501.</p> <p>So the BasicRpcTestProvider implements the basic-global RPC. It is also a ClusterSingletonService - the RPC is registered when it gets the singleton lock and is spun up. Therefore with 2/3 nodes down and the prior lock owner was one of those 2 then consensus is lost and the remaining node isn't able to become the lock owner. Therefore there is no BasicRpcTestProvider instance running and no basic-global RPC implementation.</p> <p>Thanks!</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jhershbe" class="user-hover" rel="jhershbe">jhershbe</a>looked back through the history of Oxygen jobs and found that the failures<br/> seen are coming because of two existing bugs:</p> <p><a href="https://jira.opendaylight.org/browse/CONTROLLER-1849" title="controller not coming up healthy after being killed and restarted (401 after 5m)" class="issue-link" data-issue-key="CONTROLLER-1849"><del>CONTROLLER-1849</del></a> and <a href="https://jira.opendaylight.org/browse/CONTROLLER-1742" title="RetiredGenerationException in cluster sanity suite" class="issue-link" data-issue-key="CONTROLLER-1742"><del>CONTROLLER-1742</del></a></p> <p>marking this as duplicate because of that.</p> <p>From what I can tell this works via EOS. Not a question, just recording this for posterity.</p> <p>yes - EOS via ClusterSingletonService</p> Relates NETVIRT-1315 Development Rank 0|i03gef: