Can you attach the full log? Besides the logging what other behavior are you observing?
The Shards and ShardManager disappear from JConsole.
I'm curious if you can reproduce this as well or if there's something fluky going on in my environment. I haven't dug into this yet with all the other patches going on right now.
(In reply to Moiz Raja from comment #1)
> Can you attach the full log? Besides the logging what other behavior are you
> observing?
I haven't seen this problem yet 
The ActorSystem was shutdown possibly. Why this happened may be in the logs.
I found this link https://groups.google.com/forum/#!topic/akka-user/jleFC7P66ao which appears to be recent. This outlines the same issue I see - it's a bug in akka. The issue is related to a node becoming reachable after being auto-downed.
On my controller instances I still had the auto-down-unreachable-after setting to 10s. So I set it really high and the actor system shutdown did not occur. That's good news. So it seems auto-downing is problematic at this point.
However, the 3rd node which I had restarted didn't join back into the cluster. On the other 2 nodes I see this INFO message every few seconds: "Existing member [address of 3rd node] is trying to join, ignoring". Not sure what's going on there. Interestingly the restarted node did become a follower as it appears the leader was able to send heartbeats b/c we cache the remote actor address in the RaftActorContext. However the restarted node did not have peer addresses for the other 2 so it did not get ClusterMemberUp messages.
Also, akka's ClusterState mbean reported the stopped node as Unreachable which makes sense. However it also listed it in the members list as Up which doesn't seem right. It continued to report this state even after the node was restarted.
So it seems the cluster leader didn't Up the restarted node or the endpoint layer didn't report reachability as evidenced by the "ignoring" message. It's unclear what it was ignoring and why.
It seems akka's clustering code is either really funky or really buggy.
After reading other posts, e.g. https://groups.google.com/forum/#!topic/akka-user/AdRSv2yuwo4, it seems clear that a node with same host:port can't re-join until the previous member with same host:port has been removed from the cluster, which happens after auto-downing it. However, auto-downing seems to cause the endpoint layer bug that shuts down the actor system.
I really don't get the rationale behind their design.
Going through the 2.3.10 release notes (http://akka.io/news/2015/04/23/akka-2.3.10-released.html) I spotted this,
- remove wrong assertion in remoting, which could lead to ActorSystem termination when restarted remote ActorSystem connects after being quarantined
Maybe we should switch to akka 2.3.10 it has a bunch of remoting/clustering related fixes.
Yup - that's the issue. We should upgrade.
(In reply to Moiz Raja from comment #6)
> Going through the 2.3.10 release notes
> (http://akka.io/news/2015/04/23/akka-2.3.10-released.html) I spotted this,
>
> * remove wrong assertion in remoting, which could lead to ActorSystem
> termination when restarted remote ActorSystem connects after being
> quarantined
>
> Maybe we should switch to akka 2.3.10 it has a bunch of remoting/clustering
> related fixes.
More details https://github.com/akka/akka/issues/17213
I tested upgrading to 2.3.10. I had auto-down-unreachable-after set to 10s and, after stopping and restarting a node after auto-down, the actor system shutdown issue didn't occur and the node successfully rejoined the cluster. So that issue is fixed.
I also tested with auto-down-unreachable-after set high. After restarting a node, it successfully rejoined the cluster. So that issue is fixed as well. Tried it twice.
Akka's ClusterState mbean still reported the member as both Up and Unreachable but I can live with that.
Looks like we have a stable akka release wrt to remoting.