Details
-
Bug
-
Status: Resolved
-
Resolution: Cannot Reproduce
-
Post-Helium
-
None
-
None
-
Operating System: All
Platform: All
-
2557
Description
While adding flows to config store, in a 3 node cluster, the cluster went to a state with 2 leaders.
Node 1: leader; Node 3: One switch connected
-Connect debugger to node 1 with breakpoint in AbstractListeningCommitter.onDataChanged()
- Use restconf on Node 1 to add a flow
When the control reaches the breakpoint in debugger, leadership changes. After this, I "resume program" in debugger. I had other breakpoints but control doesnt stop there.
Node 2 becomes leader while other 2 are follower.
Restconf call does not return yet.
LastApplied and LastLogIndex increments on Node 2 and Node 3. Node 1 still shows 1 uncommitted entry.
After a while Restconf call returns with akka timeout at which point Node 1 becomes leader and syncs up with other nodes in cluster. Node 2 still stays to be a leader.
Node 3 initially flip-floped between node 1 and node 2 as the leader and synced data with the corresponding leader. Later it stuck with node 2 as the leader. However, add flow on node 1 synced LastApplied, LastLogIndex with that of node 1 but leadership and data (seen with restconf GET) was synced with node 2.
The LastLogTerm stayed constant on all 3 nodes.
I have attached a screenshot displaying current state of jmx counters on all 3 nodes.
Node 1 = 148
Node 2 = 151
Node 3 = 150