[CONTROLLER-1083] Clustering: 2 leaders in a cluster

Thu, 19 Oct 2017 21:27:04 +0000

While adding flows to config store, in a 3 node cluster, the cluster went to a state with 2 leaders.

Node 1: leader; Node 3: One switch connected
-Connect debugger to node 1 with breakpoint in AbstractListeningCommitter.onDataChanged()

Use restconf on Node 1 to add a flow

When the control reaches the breakpoint in debugger, leadership changes. After this, I "resume program" in debugger. I had other breakpoints but control doesnt stop there.

Node 2 becomes leader while other 2 are follower.

Restconf call does not return yet.

LastApplied and LastLogIndex increments on Node 2 and Node 3. Node 1 still shows 1 uncommitted entry.

After a while Restconf call returns with akka timeout at which point Node 1 becomes leader and syncs up with other nodes in cluster. Node 2 still stays to be a leader.

Node 3 initially flip-floped between node 1 and node 2 as the leader and synced data with the corresponding leader. Later it stuck with node 2 as the leader. However, add flow on node 1 synced LastApplied, LastLogIndex with that of node 1 but leadership and data (seen with restconf GET) was synced with node 2.

The LastLogTerm stayed constant on all 3 nodes.

I have attached a screenshot displaying current state of jmx counters on all 3 nodes.
Node 1 = 148
Node 2 = 151
Node 3 = 150

OpenDaylight JIRA

[CONTROLLER-1083] Clustering: 2 leaders in a cluster