Restarting controller after configuring netconf connector fails

I configured a netconf connector as outlined in https://wiki.opendaylight.org/view/OpenDaylight_Controller:Config:Examples:Netconf#Spawning_Additional_Netconf_Connectors_While_the_Controller_is_Running. The config was saved to etc/opendaylight/current/controller.currentconfig.xml (along with all the other module configs).

I verified a GET on http://localhost:8181/restconf/config/network-topology:network-topology/topology/topology-netconf/node/controller-config/yang-ext:mount/config:modules returns all the modules correctly including the new netconf connector.

I then restarted the controller but it failed miserably with several exceptions:

Exception in thread "config-pusher" java.lang.IllegalStateException: Failed to send commit for configuration 05-clustering.xml(odl-netconf-connector,odl-mdsal-broker)
at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.sendRequestGetResponseCheckIsOK(ConfigPusherImpl.java:285)
at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.pushConfig(ConfigPusherImpl.java:239)
at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.pushConfigWithConflictingVersionRetries(ConfigPusherImpl.java:128)
at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.internalPushConfigs(ConfigPusherImpl.java:101)
at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.process(ConfigPusherImpl.java:76)
at org.opendaylight.controller.netconf.persist.impl.osgi.ConfigPersisterActivator$InnerCustomizer$1.run(ConfigPersisterActivator.java:181)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.IllegalStateException: Error - getInstance() failed for ModuleIdentifier

{factoryName='sal-netconf-connector', instanceName='new-netconf-device'} in transaction TransactionIdentifier{name='ConfigTransaction-46-48'}
at org.opendaylight.controller.config.manager.impl.ConfigTransactionControllerImpl.secondPhaseCommit(ConfigTransactionControllerImpl.java:405)
at org.opendaylight.controller.config.manager.impl.ConfigRegistryImpl.secondPhaseCommit(ConfigRegistryImpl.java:280)
at org.opendaylight.controller.config.manager.impl.ConfigRegistryImpl.commitConfig(ConfigRegistryImpl.java:227)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)
at sun.reflect.GeneratedMethodAccessor36.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279)
at com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(ConvertingMethod.java:193)
at com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(ConvertingMethod.java:175)
at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:117)
at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:54)
at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
at com.sun.jmx.mbeanserver.MXBeanProxy$InvokeHandler.invoke(MXBeanProxy.java:150)
at com.sun.jmx.mbeanserver.MXBeanProxy.invoke(MXBeanProxy.java:167)
at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:252)
at com.sun.proxy.$Proxy16.commitConfig(Unknown Source)
at org.opendaylight.controller.config.util.ConfigRegistryJMXClient.commitConfig(ConfigRegistryJMXClient.java:102)
at org.opendaylight.controller.netconf.confignetconfconnector.transactions.TransactionProvider.commitTransaction(TransactionProvider.java:138)
at org.opendaylight.controller.netconf.confignetconfconnector.operations.Commit.handleWithNoSubsequentOperations(Commit.java:54)
at org.opendaylight.controller.netconf.util.mapping.AbstractLastNetconfOperation.handle(AbstractLastNetconfOperation.java:33)
at org.opendaylight.controller.netconf.util.mapping.AbstractNetconfOperation.handle(AbstractNetconfOperation.java:100)
at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.sendRequestGetResponseCheckIsOK(ConfigPusherImpl.java:280)
... 6 more
Caused by: java.lang.IllegalStateException: cannot create children while terminating or terminated
at akka.actor.dungeon.Children$class.makeChild(Children.scala:199)
at akka.actor.dungeon.Children$class.attachChild(Children.scala:41)
at akka.actor.ActorCell.attachChild(ActorCell.scala:369)
at akka.actor.ActorSystemImpl.actorOf(ActorSystem.scala:553)
at org.opendaylight.controller.cluster.datastore.DistributedDataStore.<init>(DistributedDataStore.java:80)
at org.opendaylight.controller.cluster.datastore.DistributedDataStoreFactory.createInstance(DistributedDataStoreFactory.java:34)
at org.opendaylight.controller.config.yang.config.distributed_datastore_provider.DistributedOperationalDataStoreProviderModule.createInstance(DistributedOperationalDataStoreProviderModule.java:65)
at org.opendaylight.controller.config.spi.AbstractModule.getInstance(AbstractModule.java:73)
at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.opendaylight.controller.config.manager.impl.dependencyresolver.DependencyResolverManager$ModuleInvocationHandler.handleInvocation(DependencyResolverManager.java:150)
at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
at com.sun.proxy.$Proxy74.getInstance(Unknown Source)
at org.opendaylight.controller.config.manager.impl.dependencyresolver.DependencyResolverImpl.resolveInstance(DependencyResolverImpl.java:159)
at org.opendaylight.controller.config.yang.config.concurrent_data_broker.AbstractDomConcurrentDataBrokerModule.resolveDependencies(AbstractDomConcurrentDataBrokerModule.java:91)
at org.opendaylight.controller.config.spi.AbstractModule.getInstance(AbstractModule.java:72)
at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.opendaylight.controller.config.manager.impl.dependencyresolver.DependencyResolverManager$ModuleInvocationHandler.handleInvocation(DependencyResolverManager.java:150)
at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
at com.sun.proxy.$Proxy45.getInstance(Unknown Source)
at org.opendaylight.controller.config.manager.impl.dependencyresolver.DependencyResolverImpl.resolveInstance(DependencyResolverImpl.java:159)
at org.opendaylight.controller.config.yang.md.sal.dom.impl.AbstractDomBrokerImplModule.resolveDependencies(AbstractDomBrokerImplModule.java:70)
at org.opendaylight.controller.config.spi.AbstractModule.getInstance(AbstractModule.java:72)
at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.opendaylight.controller.config.manager.impl.dependencyresolver.DependencyResolverManager$ModuleInvocationHandler.handleInvocation(DependencyResolverManager.java:150)
at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
at com.sun.proxy.$Proxy45.getInstance(Unknown Source)
at org.opendaylight.controller.config.manager.impl.dependencyresolver.DependencyResolverImpl.resolveInstance(DependencyResolverImpl.java:159)
at org.opendaylight.controller.config.yang.md.sal.connector.netconf.AbstractNetconfConnectorModule.resolveDependencies(AbstractNetconfConnectorModule.java:110)
at org.opendaylight.controller.config.spi.AbstractModule.getInstance(AbstractModule.java:72)
at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.opendaylight.controller.config.manager.impl.dependencyresolver.DependencyResolverManager$ModuleInvocationHandler.handleInvocation(DependencyResolverManager.java:150)
at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
at com.sun.proxy.$Proxy74.getInstance(Unknown Source)
at org.opendaylight.controller.config.manager.impl.ConfigTransactionControllerImpl.secondPhaseCommit(ConfigTransactionControllerImpl.java:399)
... 36 more


The clustering modules started up originally but then were shutdown by the config system. I assume the reason is somewhere in the exception stack traces.

It's curious that it fails trying to commit the 05-clustering.xml config. It's my understanding that the controller.currentconfig.xml overrides the initial configs in etc/opendaylight/karaf.

Also it's strange that the traces indicate the 05-clustering.xml config failure was caused by a failure in "ModuleIdentifier{factoryName='sal-netconf-connector', instanceName='new-netconf-device'}

". I don't see the causual relationship there.