[CONTROLLER-936] Clustering:ShardManager does not return the leader of the Shard in FindPrimary Created: 15/Oct/14 Updated: 26/Mar/15 Resolved: 26/Mar/15 |
|
| Status: | Resolved |
| Project: | controller |
| Component/s: | mdsal |
| Affects Version/s: | Post-Helium |
| Fix Version/s: | None |
| Type: | Bug | ||
| Reporter: | Kamal Rameshan | Assignee: | Tom Pantelis |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
Operating System: All |
||
| External issue ID: | 2194 |
| Description |
|
In ShardManager's FindPrimary, it attemps to return a shard replica , which may or may not be the leader of the shard. The logic to return it from the local exists, but to find the actual leader of the shard is missing. For a 3 node cluster, this might not be introducing a significant dent in the performance, as each node has atleast one replica of each shard. But when we move to more fine-grained sharding, the number of hops will be 1 more. |
| Comments |
| Comment by Mark Mozolewski [ 30/Oct/14 ] |
|
(Proposed Fix - Implemented & Testing) The Shard’s Leader (aka “primary”) is reported to each Shard while adjusting its behavior (changing leadership) and to report in the ShardStats MBean for JMX access. The ShardManager needs some way to identify the Shard leader without coupling the Shard and ShardMananger or being part of the election process that the RaftActor is running. My proposed fix that I've implemented and am testing is to allow the Shard to register Leader changed listeners and report a Leader change event to those registered listeners. In this case the ShardManager will register a Leader change listener with its local shard upon that Shard’s creation (if it has one). When ShardManager#findPrimary is called it may know who the Leader is and can report that. There will still be cases where it doesn’t know who the Leader is, e.g. it doesn’t have a local shard for this shard configuration or the information has not been reported yet. In those cases #findPrimary will function as it does today where from the local shard configuration file it knows which nodes in the system have shard replicas and it reports one of those as primary to allow them complete the findPrimary request. (Note: A Shard will forward any requests to the Leader if contacted, we are just trying to optimize how often the the Leader can be identified correctly.) |
| Comment by Moiz Raja [ 30/Oct/14 ] |
|
In the case where we do not have a local shard how about forwarding the findprimary to a shardmanager on a node which we do know has a local shard? |
| Comment by Tom Pantelis [ 11/Mar/15 ] |
|
The ShardManager now listens for RoleChangedNotificatons so it knows the raft state of each shard. I have also added a LeaderStateChange notification with https://git.opendaylight.org/gerrit/#/c/16328/1 so the ShardManager knows the current leaderId. With these changes and https://git.opendaylight.org/gerrit/#/c/16164/14, the ShardManager now returns the actual shard leader actor path from FindPrimary. Also if there is no leader yet, it waits a period of time (shardInitializationTimeout) for a leader to be elected before returning a response. If there is no leader after the timeout, it returns a NoShardLeaderException failure response. With https://git.opendaylight.org/gerrit/#/c/16194/7, if there's no local shard it forwards the FindPrimary message to another remote ShardManager. |