Current raft implementation seems to be unstable when dynamically adding peers when new nodes come up.

Scenario were trying to do:

We want to have a shard started with replicas on all cluster members(even ones added at any time in the future).

Currently the raft implementation seems to quite fragile when dynamically adding peers, when you send an AddServer message to the actor on MemberUp/Reachable events the followers seem to always ignore it leading to each peer having different set of peers which leads to never ending elections.
The current implementation seems to rely either on having hardcoded replicas in the module conf. If we went similar route like the ClusterAdminService addReplica does this also seems to only update peers on the leader which leads to the neverending elections due to different peers for each actor.