Operating System: All
Platform: All

We could use JMX actions and jolokia's REST API. I would think jolokia supports POST for actions.

(In reply to Tom Pantelis from comment #1)
> We could use JMX actions and jolokia's REST API. I would think jolokia
> supports POST for actions.

Is performing all our configuration changes on the Cluster subsystem via JMX (with Jolokia for REST/HTTP access) in line with CONTROLLER-1064 bringing all our clustering configuration under the Config Subsystem?

I'd like to make sure all existing config (like distributed-datastore-provider.yang) and new config for cluster/shard formation, and maybe even Akka are accessed in a consistent way. Thanks.

Yeah - we'll have to see if the config system will be sufficient for what we want to do. The config system does support JMX but currently the CDS JMX beans aren't integrated with the config system. The config system has limitations. Also if/how the config system deals with upgrades is not clear.

I think regardless of how the JMX beans are registered (via config system or not) I think we could use JMX actions as we'll likely want to apply the config changes to the running system as well as persist it. Either that or use a yang-defined RPC accessed via restconf.

This bug covers two things,

1. Dynamically adding/removing new controller nodes to a cluster
2. Dynamically adding/removing shard replicas

Basically grow//shrink cluster.

Attachment CONTROLLER-934_design.docx has been added with description: Design approach for dynamic addition/removal of controller nodes

If this involves change the membership of a given shard, that is going to be a very complex change which requires, among other things, storing the membership information and it's modifications as part of the shards themselves.

>>> Memebr addition
>>>The leader for each shard replicates it data to the shard replica member created in the new node.

Comment : When we have a large shard to be replicated, the node would be unusable - right ? Ie. when above replication process takes longer time, the new node would not be able to participate in any in-flight or incoming transactions - is my understanding right ? I understand that this is not specific to dynamic addition. Bu it can become more prominent when new "empty" node is added to a shard

>>> The listeners in the cluster group controller nodes listening for changes in the ‘shards’ list informs ShardManager to add shard replicas for its set of local shards

Comment : Since updating ShardManager is a node-local action, it would not be a problem if we use clustered datatree change listeners. It must just be ensured that above listener does not perform any clusterwide-operation again with any other shards CDS operations because they would be executed more than once which may not be desirable

>>> Dynamic removal of controller node from cluster

Comment :
a) When a node is removed, do we honor the replication criterion of the shard-definition or not ? Eg. Shard S1 is configured with 3 replicas on members M1, M2 and M3. Now we bring down M3. Now, if we examine the "shard-status" via RESTCONF , we would see only M1 and M2 as members of S1 - right ?

b) We might have to "warn"/"fail" "administrator" action when an attempt is made to remove all members of a shard.

One possible solution to (b):
We can define something called "min-live-shard-members" parameter and keep that as, say, 2 and when there is a removal attempt when only 2 members of shard are present, we can fail the removal-request with error-message.

>>> JMX mbean API will be exposed to allow dynamic removal of a controller node from the cluster group

Comment : Since the model is very much available via RESTCONF for modifications, would it not be sufficient to just use that or RPC (as indicated by Tom) instead of JMX ?

Would it be better if we have a FSM which depicts various states through which a new node transitions before it becomes available for transactions ?

Initial patches:

https://git.opendaylight.org/gerrit/#/c/28018/
https://git.opendaylight.org/gerrit/#/c/27742/
https://git.opendaylight.org/gerrit/#/c/28596/

https://gist.github.com/moizr/9401c6c7f427ec4bbff4