Uploaded image for project: 'controller'
  1. controller
  2. CONTROLLER-1817

Shard leader does not join back Cluster when it is isolated and rejoined

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Cannot Reproduce
    • Icon: High High
    • None
    • None
    • clustering
    • None

      Shard leader does not join back Cluster when it is isolated and rejoined
      Steps to recreate:

      1. Bring up 3 node cluster and make node 2 as Shard leader using "http://{{controller-ip}}:{{restconf-port}}/restconf/operations/cluster-admin:make-leader-local" API
      2. Check if Node 2 is the leader for all shards and data-store-type.
      3. Apply ACL on Node 1 and Node 3 to isolate Node 2.
      4. Check if Node 1 or Node 3 is elected as Leader and Node 2 is "Isolated Leader"
      5. When Active Leader marks Node 2 as terminated to resume Leader capability, remove the ACL applied on Node 1 and Node 3
      Note - For Step 3, also tried applying ACL only on Node 2 to isolate Node 1 & 3. Same behavior.
      

      Expected Behavior

      Node 2 should join back the cluster as "Follower"
      

      Actual Behavior

      Node 2 remains as "Isolated Leader" and all 3 nodes there is "Quarantined address is still unreachable or has not been restarted" message which is seen for close to 50 minutes. After which Node 2 joins back the cluster and cluster operations are resumed.
      

       

            tpantelis Tom Pantelis
            chethana7 Chethana Lakshmanappa
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: