That message is normal from akka - it's reporting a node is unreachable which I assume is part of the test.
as I commented in the kernel call, this jira was opened because of a failing test. I think vpickard was just trying to find some
more specific reason for the failure (to be helpful) and not neccessarily calling out that error message.
maybe it will be better to use more generic jira summaries like "CSIT test case XYZ intermittently failing" with some comments
showing certain suspect logs like "Leader can not perform its duties" so we can keep better track of our issues? just an idea,
but I'm not sure if this one getting closed resulted in a new one being open, because I doubt the intermittent test failure was
addressed yet.
Sure. You can reopen this and change summary etc or open a new one - doesn't really matter. I thought this was created b/c you saw that ominous message in the log and thought it was an issue.
here is the original bug description for posterity
Controller clustering CSIT is failing intermittently, I see this in the karaf logs 7 times for this particular run [1]
2018-07-03T06:49:24,867 | INFO | opendaylight-cluster-data-akka.actor.default-dispatcher-33 | Cluster(akka://opendaylight-cluster-data) | 51 - com.typesafe.akka.slf4j - 2.5.11 | Cluster Node [akka.tcp://opendaylight-cluster-data@10.30.170.209:2550] - Leader can currently not perform its duties, reachability status: [akka.tcp://opendaylight-cluster-data@10.30.170.209:2550 -> akka.tcp://opendaylight-cluster-data@10.30.170.221:2550: Unreachable [Unreachable] (1), akka.tcp://opendaylight-cluster-data@10.30.170.220:2550 -> akka.tcp://opendaylight-cluster-data@10.30.170.221:2550: Unreachable [Unreachable] (1)], member status: [akka.tcp://opendaylight-cluster-data@10.30.170.209:2550 Up seen=true, akka.tcp://opendaylight-cluster-data@10.30.170.220:2550 Up seen=true, akka.tcp://opendaylight-cluster-data@10.30.170.221:2550 Up seen=false]
thanks tpantelis, I just repopened this jira. the description had a link to the failing job, so I use that to
add a few details.
we need to keep digging from here.
Archived logs since this is a sandbox job
BRT stands for "Basic RPC Test" 
We need to understand what the test does and need the log files in order to do post-mortem analysis.
The archived logs are here:
So the "old owner" was odl3. There are no errors in the odl3 karaf log after restart around 2018-07-03T06:50:20. In fact there's no log messages at all after 6:52:28 - the robot log.html indicates the basic-global RPC was sent around 6:54. We need to enable debug for org.opendaylight.controller.remote.rpc and try to reproduce.
If possible, in the Restart_Killed_Member step, it would be helpful to save off the karaf.log and delete it so the node starts with a clean log.
Looking at the Run_Rpc keyword, it issues the RPC once. When a node restarts, it takes a bit of time for it to learn about RPC registrations from other nodes via the gossip functionality. So I suspect the test issued the RPC before odl3 knew about the remote RPC implementation, hence the 501 "No implementation" error. So the test should retry via Wait Until Keyword Succeeds.
what's a reasonable time for this though. looking at the log vpickard posted in the comment above, the
killed memeber is restarted, and it polls for ~2 minutes until syncstatus is true, then it's another 1.5m before
the RPC is issued. So, 3.5m from the time of restart, but probably more importantly, 1.5m since the cluster
sync was said to be ok.
we can certainly add a WUKS to that step, but it seems like it's already happening quite a long time later
for a cluster member to not know about the RPC. right?
I didn't notice 1.5m passed. That should be plenty of time. Does it clean the data dir before restarting? If so, then it's re-installing the world so that can take time. It was just an idea...
We need a failure run with org.opendaylight.controller.remote.rpc debug enabled. Maybe pull out that one suite and run over and over in the sandbox or whatever?
The few times I've seen this reproduced it seems that it is caused by 2/3 of the nodes being down or otherwise unreachable. Manual explicit tests of this have produced the same outcome - when 2/3 nodes are down (or in "deadlock") rpc calls return 501.
So the BasicRpcTestProvider implements the basic-global RPC. It is also a ClusterSingletonService - the RPC is registered when it gets the singleton lock and is spun up. Therefore with 2/3 nodes down and the prior lock owner was one of those 2 then consensus is lost and the remaining node isn't able to become the lock owner. Therefore there is no BasicRpcTestProvider instance running and no basic-global RPC implementation.
Thanks!
jhershbelooked back through the history of Oxygen jobs and found that the failures
seen are coming because of two existing bugs:
CONTROLLER-1849 and CONTROLLER-1742
marking this as duplicate because of that.
From what I can tell this works via EOS. Not a question, just recording this for posterity.
yes - EOS via ClusterSingletonService