https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-1273 controller <p>I'm seeing strange behavior when a node in a 3 node cluster is restarted. It somehow causes the Shard and ShardManager actors in the other nodes in the cluster to terminate. I see these messages in the log:</p> <p>2015-04-22 13:11:08,317 | INFO | lt-dispatcher-19 | Shard | 177 | 227 - org.opendaylight.controller.sal-akka-raft - 1.2.0.SNAPSHOT | | Stopping Shard member-1-shard-topology-operational</p> <p>2015-04-22 13:11:08,323 | INFO | lt-dispatcher-18 | ShardManager | 159 | 234 - org.opendaylight.controller.sal-distributed-datastore - 1.2.0.SNAPSHOT | | Stopping ShardManager</p> <p>There's no other messages in the log except the usual akka INFO messages about node addresses gated and nodes leaving and joining the cluster.</p> <p>Note that this occurs after the node is started up and not after it is shutdown. Right after the Shard stopping messages above I see the akka message that the downed node is now re-joining:</p> <p>2015-04-22 13:14:08,412 | INFO | lt-dispatcher-22 | receive$1$$anonfun$applyOrElse$3 | 74 | 220 - com.typesafe.akka.slf4j - 2.3.9 | | Cluster Node <span class="error">&#91;akka.tcp://odl-cluster-rpc@127.0.0.1:2551&#93;</span> - Node <span class="error">&#91;akka.tcp://odl-cluster-rpc@127.0.0.1:2555&#93;</span> is JOINING, roles []</p> <p>I have all 3 nodes running in the same VM on different ports so I'm not sure if that's a factor but I've been running with this setup for a while without seeing this issue.</p> <p>Operating System: All<br/> Platform: All</p> CONTROLLER-1273

Clustering: Shard actors are terminated when another cluster node is restarted

Bug Resolved Done Unassigned Tom Pantelis Thu, 23 Apr 2015 05:44:22 +0000 Tue, 25 Jul 2023 08:24:01 +0000 Tue, 5 May 2015 16:45:18 +0000 mdsal 0 1 <p>Can you attach the full log? Besides the logging what other behavior are you observing?</p> <p>The Shards and ShardManager disappear from JConsole.</p> <p>I'm curious if you can reproduce this as well or if there's something fluky going on in my environment. I haven't dug into this yet with all the other patches going on right now. </p> <p>(In reply to Moiz Raja from comment #1)<br/> &gt; Can you attach the full log? Besides the logging what other behavior are you<br/> &gt; observing?</p> <p>I haven't seen this problem yet <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/> The ActorSystem was shutdown possibly. Why this happened may be in the logs.</p> <p>I found this link <a href="https://groups.google.com/forum/#!topic/akka-user/jleFC7P66ao" class="external-link" target="_blank" rel="nofollow noopener">https://groups.google.com/forum/#!topic/akka-user/jleFC7P66ao</a> which appears to be recent. This outlines the same issue I see - it's a bug in akka. The issue is related to a node becoming reachable after being auto-downed.</p> <p>On my controller instances I still had the auto-down-unreachable-after setting to 10s. So I set it really high and the actor system shutdown did not occur. That's good news. So it seems auto-downing is problematic at this point.</p> <p>However, the 3rd node which I had restarted didn't join back into the cluster. On the other 2 nodes I see this INFO message every few seconds: "Existing member <span class="error">&#91;address of 3rd node&#93;</span> is trying to join, ignoring". Not sure what's going on there. Interestingly the restarted node did become a follower as it appears the leader was able to send heartbeats b/c we cache the remote actor address in the RaftActorContext. However the restarted node did not have peer addresses for the other 2 so it did not get ClusterMemberUp messages.</p> <p>Also, akka's ClusterState mbean reported the stopped node as Unreachable which makes sense. However it also listed it in the members list as Up which doesn't seem right. It continued to report this state even after the node was restarted. </p> <p>So it seems the cluster leader didn't Up the restarted node or the endpoint layer didn't report reachability as evidenced by the "ignoring" message. It's unclear what it was ignoring and why.</p> <p>It seems akka's clustering code is either really funky or really buggy.</p> <p>After reading other posts, e.g. <a href="https://groups.google.com/forum/#!topic/akka-user/AdRSv2yuwo4" class="external-link" target="_blank" rel="nofollow noopener">https://groups.google.com/forum/#!topic/akka-user/AdRSv2yuwo4</a>, it seems clear that a node with same host:port can't re-join until the previous member with same host:port has been removed from the cluster, which happens after auto-downing it. However, auto-downing seems to cause the endpoint layer bug that shuts down the actor system.</p> <p>I really don't get the rationale behind their design.</p> <p>Going through the 2.3.10 release notes (<a href="http://akka.io/news/2015/04/23/akka-2.3.10-released.html" class="external-link" target="_blank" rel="nofollow noopener">http://akka.io/news/2015/04/23/akka-2.3.10-released.html</a>) I spotted this,</p> <ul> <li>remove wrong assertion in remoting, which could lead to ActorSystem termination when restarted remote ActorSystem connects after being quarantined</li> </ul> <p>Maybe we should switch to akka 2.3.10 it has a bunch of remoting/clustering related fixes.</p> <p>Yup - that's the issue. We should upgrade. </p> <p>(In reply to Moiz Raja from comment #6)<br/> &gt; Going through the 2.3.10 release notes<br/> &gt; (<a href="http://akka.io/news/2015/04/23/akka-2.3.10-released.html" class="external-link" target="_blank" rel="nofollow noopener">http://akka.io/news/2015/04/23/akka-2.3.10-released.html</a>) I spotted this,<br/> &gt; <br/> &gt; * remove wrong assertion in remoting, which could lead to ActorSystem<br/> &gt; termination when restarted remote ActorSystem connects after being<br/> &gt; quarantined<br/> &gt; <br/> &gt; Maybe we should switch to akka 2.3.10 it has a bunch of remoting/clustering<br/> &gt; related fixes.</p> <p>More details <a href="https://github.com/akka/akka/issues/17213" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/akka/akka/issues/17213</a></p> <p>I tested upgrading to 2.3.10. I had auto-down-unreachable-after set to 10s and, after stopping and restarting a node after auto-down, the actor system shutdown issue didn't occur and the node successfully rejoined the cluster. So that issue is fixed.</p> <p>I also tested with auto-down-unreachable-after set high. After restarting a node, it successfully rejoined the cluster. So that issue is fixed as well. Tried it twice. </p> <p>Akka's ClusterState mbean still reported the member as both Up and Unreachable but I can live with that.</p> <p>Looks like we have a stable akka release wrt to remoting.</p> Development External issue ID 3049 External issue URL Issue Type ODL SR Target Milestone Rank 0|i02pjz: