Uploaded image for project: 'controller'
  1. controller
  2. CONTROLLER-1083

Clustering: 2 leaders in a cluster

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved
    • Resolution: Cannot Reproduce
    • Post-Helium
    • None
    • clustering
    • None
    • Operating System: All
      Platform: All

    • 2557

    Description

      While adding flows to config store, in a 3 node cluster, the cluster went to a state with 2 leaders.

      Node 1: leader; Node 3: One switch connected
      -Connect debugger to node 1 with breakpoint in AbstractListeningCommitter.onDataChanged()

      • Use restconf on Node 1 to add a flow

      When the control reaches the breakpoint in debugger, leadership changes. After this, I "resume program" in debugger. I had other breakpoints but control doesnt stop there.

      Node 2 becomes leader while other 2 are follower.

      Restconf call does not return yet.

      LastApplied and LastLogIndex increments on Node 2 and Node 3. Node 1 still shows 1 uncommitted entry.

      After a while Restconf call returns with akka timeout at which point Node 1 becomes leader and syncs up with other nodes in cluster. Node 2 still stays to be a leader.

      Node 3 initially flip-floped between node 1 and node 2 as the leader and synced data with the corresponding leader. Later it stuck with node 2 as the leader. However, add flow on node 1 synced LastApplied, LastLogIndex with that of node 1 but leadership and data (seen with restconf GET) was synced with node 2.

      The LastLogTerm stayed constant on all 3 nodes.

      I have attached a screenshot displaying current state of jmx counters on all 3 nodes.
      Node 1 = 148
      Node 2 = 151
      Node 3 = 150

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            Unassigned Unassigned
            abhishk2@cisco.com Abhishek Kumar
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: