[CONTROLLER-1217] Clustering : unable to configure persistence using restconf(netconf connector) on 3 node cluster Created: 17/Mar/15  Updated: 25/Jul/23  Resolved: 28/Dec/15

Status: Resolved
Project: controller
Component/s: clustering
Affects Version/s: None
Fix Version/s: None

Type: Bug
Reporter: Peter Gubka Assignee: Unassigned
Resolution: Won't Do Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: Linux
Platform: PC

Attachments: Text File karaf.log     Text File persistent_change_steps.txt    
External issue ID: 2858

 Description   

odl: https://nexus.opendaylight.org/content/repositories/opendaylight.snapshot/org/opendaylight/integration/distribution-karaf/0.3.0-SNAPSHOT/distribution-karaf-0.3.0-20150317.094607-668.tar.gz

installed features: odl-restconf odl-mdsal-clustering odl-restconf-all odl-openflowplugin-all odl-netconf-connector-all
installed bundles: mvn:org.jolokia/jolokia-osgi/1.1.5

-Xms128M -Xmx2048m -XX:MaxPermSize=512m

VM: fedora18, 6G ram,
java -version
java version "1.7.0_67"
Java(TM) SE Runtime Environment (build 1.7.0_67-b01)
Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)

i have installed 3 node cluster with config datastore shart persistence=false.
i wanted to switch it on (persistence=true) using restconf and postman

method: POST

url:http://g37:8181/restconf/config/network-topology:network-topology/topology/topology-netconf/node/controller-config/yang-ext:mount/

data:
<modules xmlns="urn:opendaylight:params:xml:ns:yang:controller:config">
<module xmlns="urn:opendaylight:params:xml:ns:yang:controller:config">
<type
xmlns:x="urn:opendaylight:params:xml:ns:yang:controller:config:distributed-datastore-provider">x:distributed-config-datastore-provider
</type>
<name>distributed-config-store-module</name>
<config-properties
xmlns="urn:opendaylight:params:xml:ns:yang:controller:config:distributed-datastore-provider">
<shard-transaction-idle-timeout-in-minutes>10</shard-transaction-idle-timeout-in-minutes>
<shard-journal-recovery-log-batch-size>5000</shard-journal-recovery-log-batch-size>
<persistent>true</persistent>
<operation-timeout-in-seconds>5</operation-timeout-in-seconds>
<shard-transaction-commit-timeout-in-seconds>30</shard-transaction-commit-timeout-in-seconds>
<shard-election-timeout-factor>2</shard-election-timeout-factor>
<shard-initialization-timeout-in-seconds>300</shard-initialization-timeout-in-seconds>
<enable-metric-capture>false</enable-metric-capture>
<transaction-creation-initial-rate-limit>100</transaction-creation-initial-rate-limit>
<shard-snapshot-data-threshold-percentage>12</shard-snapshot-data-threshold-percentage>
<shard-heartbeat-interval-in-millis>500</shard-heartbeat-interval-in-millis>
<bounded-mailbox-capacity>1000</bounded-mailbox-capacity>
<shard-leader-election-timeout-in-seconds>30</shard-leader-election-timeout-in-seconds>
<max-shard-data-change-executor-queue-size>1000</max-shard-data-change-executor-queue-size>
<max-shard-data-change-executor-pool-size>20</max-shard-data-change-executor-pool-size>
<shard-isolated-leader-check-interval-in-millis>5000</shard-isolated-leader-check-interval-in-millis>
<max-shard-data-store-executor-queue-size>5000</max-shard-data-store-executor-queue-size>
<shard-snapshot-batch-count>20000</shard-snapshot-batch-count>
<shard-batched-modification-count>100</shard-batched-modification-count>
<max-shard-data-change-listener-queue-size>1000</max-shard-data-change-listener-queue-size>
<shard-transaction-commit-queue-capacity>20000</shard-transaction-commit-queue-capacity>
</config-properties>
<config-schema-service
xmlns="urn:opendaylight:params:xml:ns:yang:controller:config:distributed-datastore-provider">
<name>yang-schema-service</name>
<type
xmlns:x="urn:opendaylight:params:xml:ns:yang:controller:md:sal:dom">x:schema-service
</type>
</config-schema-service>
</module>
</modules>

headers: accept and content-type to application/xml

I got 204 but using GET i always see persistence to be false. In the log lots ex exceptions appeared.

I expected to be able to put data to /restconf/config.

 Comments   
Comment by Peter Gubka [ 17/Mar/15 ]

Attachment karaf.log has been added with description: karaf.log

Comment by Peter Gubka [ 17/Mar/15 ]

Attachment persistent_change_steps.txt has been added with description: steps with console output

Comment by Tom Pantelis [ 09/Apr/15 ]

I tried the repro steps and it does save the new config to etc/opendaylight/current/controller.currentconfig.xml and it stops the clustered config data store but it appears to fail to restart it as I no longer see the mbean in JConsole. A lot of exceptions occur in the log related to other modules. Restarting the the clustered config data store module will (attempt to) restart modules dependent on it.

I am not able to read back the configs via restconf due to CONTROLLER-1241.

I restarted karaf and the clustered config data store starts up fine but doesn't have the updated config (the config is reported in a JXM bean).

So this is an issue with the config system.

Frankly, I don't recommend updating the clustering config via restconf. It will restart most modules which is basically a crap-shoot as app modules may not handling shutdown/restart properly. Also the entire config for all modules is copied to controller.currentconfig.xml, even if you just changed one. This can break upgrades (assuming the config system properly obtains the configs from controller.currentconfig.xml as intended).

For these and other reasons, we now have an etc/org.opendaylight.controller.cluster.datastore.cfg properties file that users can modify which is overlayed on top of the config xml settings. We also dynamically react to changes without restarting the clustering modules.

Comment by Tom Pantelis [ 09/Apr/15 ]

I just noticed that while the clustered config data store started up after karaf restart, the clustered operational data store did not (again lots of exceptions config system).

Comment by Peter Gubka [ 09/Apr/15 ]

(In reply to Tom Pantelis from comment #2)
> I tried the repro steps and it does save the new config to
> etc/opendaylight/current/controller.currentconfig.xml and it stops the
> clustered config data store but it appears to fail to restart it as I no
> longer see the mbean in JConsole. A lot of exceptions occur in the log
> related to other modules. Restarting the the clustered config data store
> module will (attempt to) restart modules dependent on it.
>
> I am not able to read back the configs via restconf due to CONTROLLER-1241.
>
> I restarted karaf and the clustered config data store starts up fine but
> doesn't have the updated config (the config is reported in a JXM bean).
>
> So this is an issue with the config system.
>
> Frankly, I don't recommend updating the clustering config via restconf. It
> will restart most modules which is basically a crap-shoot as app modules may
> not handling shutdown/restart properly. Also the entire config for all
> modules is copied to controller.currentconfig.xml, even if you just changed
> one. This can break upgrades (assuming the config system properly obtains
> the configs from controller.currentconfig.xml as intended).
>
> For these and other reasons, we now have an
> etc/org.opendaylight.controller.cluster.datastore.cfg properties file that
> users can modify which is overlayed on top of the config xml settings. We
> also dynamically react to changes without restarting the clustering modules.

I understand that it is not recommended to change clustering config, but once it is possible, there will always be someone, who just tries it.

Comment by Tom Pantelis [ 09/Apr/15 ]

Yes - people can try it, assuming one knows how to do it

My point is that even if the config system handles on-the-fly restarts properly, all apps have to handle it properly as well. That's a tall order - it's not something that people generally test and it will always be a moving target.

A more fundamental issue is that the entire config for all modules is copied to controller.currentconfig.xml when one module is changed via restconf. That is a significant flaw that can affect upgrades coupled with the fact that internal code wiring config is combined with user-facing config (they can be separated but apps don't do it). IMO, the config system needs to be re-thought/re-designed wrt user-facing config.

Comment by Moiz Raja [ 10/Apr/15 ]

Moving this to Beryllium for now. I think we can safely assuming people will not be changing configuration for datastore this was in Lithium.

Generated at Wed Feb 07 19:54:58 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.