[CONTROLLER-1786] Jolokia lookup says leader exists but karaf logs says no leader Created: 07/Nov/17  Updated: 02/Aug/18  Resolved: 02/Aug/18

Status: Resolved
Project: controller
Component/s: clustering
Affects Version/s: None
Fix Version/s: Fluorine, Neon

Type: Bug Priority: High
Reporter: Sai MarapaReddy Assignee: Unassigned
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File jolokia output.txt     Text File leader_not_found.txt    
Issue Links:
Relates
relates to NETVIRT-1315 Troubleshooting Controller CSIT In Progress
relates to CONTROLLER-1856 CSIT test Local_Leader_Shutdown fails... Resolved
relates to OVSDB-434 br-int not created after failing over... Resolved
Epic Link: Clustering Stability

 Description   

Scenario of 3-node cluster:-
Bring all nodes up, transactions (car test) go through.
Shut down leader (default-config) in 3 nodes (say node 1)
Shut down leader (default-config) in 2 nodes. (say node 2)

Bring up any of the shut down node (say node 2)
Karaf logs says node2 didn't join cluster, but jolokia lookup [1] for shard-default-config shows
it has a leader.
When we try transactions (car tests - check karaf logs ) it fails .

 Comments   
Comment by Sai MarapaReddy [ 07/Nov/17 ]

For jolokia the leader information is fetched from [1] which in turn fetches from [2]
[1] controller/opendaylight/md-sal/sal-distributed-datastore/src/main/java/org/opendaylight/controller/cluster/datastore/jmx/mbeans/shard/ShardStats.java
[2] controller/opendaylight/md-sal/sal-akka-raft/src/main/java/org/opendaylight/controller/cluster/raft/client/messages/OnDemandRaftState.java
jolokia output - jolokia output.txt

karaf logs
says Shard member-3-shard-default-config currently has no leader. Attached snippet of the karaf log leader_not_found.txt

Comment by Sai MarapaReddy [ 07/Nov/17 ]

tpantelis
The code path in karaf logs differ from the code path that jolokia fetches the leader from. Are they expected to differ ?

Comment by Tom Pantelis [ 02/Aug/18 ]

I'm not really clear on the purpose of this Jira - it seems it's really asking a question rather than reporting a bug. 

The code paths to obtain the leader info in the 2 scenarios differ. jolokia is an on-demand read from the MBean. For transactions, the shard leader info is resolved and cached on the front-end for efficiency by the ShardManager via messages sent from the shard.  

 

Generated at Wed Feb 07 19:56:27 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.