https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-934 controller <p>Cluster members are defined statically in akka.conf. New cluster configurations requires changing this file on each system and restarting the controllers.</p> <p>Update to provide a way to programmatically configure a cluster including options that are currently part of akka.conf and potentially others we want to support. In general:</p> <ul class="alternate" type="square"> <li>Controller's address (IP/Hostname)</li> <li>Other cluster member's addresses (IP/Hostname) including Akka "seed nodes"</li> <li>Akka-specific config (ActorSystem config, logging, serializer, etc.)</li> </ul> <p>Provide a REST interface to create/destroy a team?</p> <p>Operating System: All<br/> Platform: All</p> CONTROLLER-934

Clustering : Programmatic Cluster configuration

Improvement Resolved Done Kalaiselvi Mark Mozolewski Tue, 14 Oct 2014 16:22:54 +0000 Thu, 7 Jan 2016 15:07:22 +0000 Thu, 7 Jan 2016 15:07:22 +0000 Helium clustering 0 8 <p>We could use JMX actions and jolokia's REST API. I would think jolokia supports POST for actions.</p> <p>(In reply to Tom Pantelis from comment #1)<br/> &gt; We could use JMX actions and jolokia's REST API. I would think jolokia<br/> &gt; supports POST for actions.</p> <p>Is performing all our configuration changes on the Cluster subsystem via JMX (with Jolokia for REST/HTTP access) in line with <a href="https://jira.opendaylight.org/browse/CONTROLLER-1064" title="Clustering : Move all clustering configuration to config sub-system" class="issue-link" data-issue-key="CONTROLLER-1064"><del>CONTROLLER-1064</del></a> bringing all our clustering configuration under the Config Subsystem? </p> <p>I'd like to make sure all existing config (like distributed-datastore-provider.yang) and new config for cluster/shard formation, and maybe even Akka are accessed in a consistent way. Thanks.</p> <p>Yeah - we'll have to see if the config system will be sufficient for what we want to do. The config system does support JMX but currently the CDS JMX beans aren't integrated with the config system. The config system has limitations. Also if/how the config system deals with upgrades is not clear.</p> <p>I think regardless of how the JMX beans are registered (via config system or not) I think we could use JMX actions as we'll likely want to apply the config changes to the running system as well as persist it. Either that or use a yang-defined RPC accessed via restconf.</p> <p>This bug covers two things,</p> <p>1. Dynamically adding/removing new controller nodes to a cluster<br/> 2. Dynamically adding/removing shard replicas</p> <p>Basically grow//shrink cluster.</p> <p>Attachment <a href="https://jira.opendaylight.org/browse/CONTROLLER-934" title="Clustering : Programmatic Cluster configuration" class="issue-link" data-issue-key="CONTROLLER-934"><del>CONTROLLER-934</del></a>_design.docx has been added with description: Design approach for dynamic addition/removal of controller nodes</p> <p>If this involves change the membership of a given shard, that is going to be a very complex change which requires, among other things, storing the membership information and it's modifications as part of the shards themselves.</p> <p>&gt;&gt;&gt; Memebr addition<br/> &gt;&gt;&gt;The leader for each shard replicates it data to the shard replica member created in the new node.</p> <p>Comment : When we have a large shard to be replicated, the node would be unusable - right ? Ie. when above replication process takes longer time, the new node would not be able to participate in any in-flight or incoming transactions - is my understanding right ? I understand that this is not specific to dynamic addition. Bu it can become more prominent when new "empty" node is added to a shard</p> <p>&gt;&gt;&gt; The listeners in the cluster group controller nodes listening for changes in the ‘shards’ list informs ShardManager to add shard replicas for its set of local shards</p> <p>Comment : Since updating ShardManager is a node-local action, it would not be a problem if we use clustered datatree change listeners. It must just be ensured that above listener does not perform any clusterwide-operation again with any other shards CDS operations because they would be executed more than once which may not be desirable</p> <p>&gt;&gt;&gt; Dynamic removal of controller node from cluster</p> <p>Comment : <br/> a) When a node is removed, do we honor the replication criterion of the shard-definition or not ? Eg. Shard S1 is configured with 3 replicas on members M1, M2 and M3. Now we bring down M3. Now, if we examine the "shard-status" via RESTCONF <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/help_16.png" height="16" width="16" align="absmiddle" alt="" border="0"/>, we would see only M1 and M2 as members of S1 - right ? </p> <p>b) We might have to "warn"/"fail" "administrator" action when an attempt is made to remove all members of a shard.</p> <p>One possible solution to (b):<br/> We can define something called "min-live-shard-members" parameter and keep that as, say, 2 and when there is a removal attempt when only 2 members of shard are present, we can fail the removal-request with error-message.</p> <p>&gt;&gt;&gt; JMX mbean API will be exposed to allow dynamic removal of a controller node from the cluster group</p> <p>Comment : Since the model is very much available via RESTCONF for modifications, would it not be sufficient to just use that or RPC (as indicated by Tom) instead of JMX ?</p> <p>Would it be better if we have a FSM which depicts various states through which a new node transitions before it becomes available for transactions ?</p> <p>Initial patches:</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/28018/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/28018/</a><br/> <a href="https://git.opendaylight.org/gerrit/#/c/27742/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/27742/</a><br/> <a href="https://git.opendaylight.org/gerrit/#/c/28596/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/28596/</a></p> <p><a href="https://gist.github.com/moizr/9401c6c7f427ec4bbff4" class="external-link" target="_blank" rel="nofollow noopener">https://gist.github.com/moizr/9401c6c7f427ec4bbff4</a></p> Development External issue ID 2187 External issue URL Issue Type ODL SR Target Milestone Rank 0|i02ngn: