[NETCONF-206] Netconf throws exception on initial start in 3 N cluster - prevents self mount

Fri, 15 Mar 2019 22:22:24 +0000

We have discovered that if you configure 3 Node clustering BEFORE you starting any of the nodes, and then you start all three nodes at the same time, 2 of them will always fail with a transaction chain exception. The two nodes that fail fail int he 99 netconf provide xml in the CSS which breaks the rest of the CSS configuring (i.e. no self mounting netconf).

The work around is to start all three nodes WITHOUT clustering first, and then stop, configure clustering, and restart, or restart all three nodes in a configured cluster after you see that hte first one works.

This is happening because of a race condition in the providers which are trying to create the nodes in the tree. See below for stack trace and some initial comments from Tom Pantelis.

=== Problem Versions ===
This worked fine on BE. It fails on BE-SR2
We have not tested yet on BO, but see comment from Tom P on that.

=== Requested Fix Version ===
We are requesting a fix in stable/beryllium

=== Reproduction ====

1) Set up a three node cluster - NOTE - do NOT start the nodes. Configure the akka.conf BEFORE starting any of the nodes
2) Start each node together
3) 2 of the nodes will eventually get exceptions when trying to push the 99-netconf connection CSS xml. 1 will work successfully.
4) Once you see that restart all three nodes and you should be good.

==== Comments on stack trace from Tom P =====
So In Be, there are 2 adapters, one for inventory Nodes and one for topology. The tx for the first one fails which causes the ISE on submit for the second. In master the inventory Nodes one is gone so mounts can only go under topology. So this issue wouldn't occur in master - the topology tx wouldn't fail on the submit call but may fail async when committed which would log an ex (one time) but would/should otherwise be benign.

=== Stack trace =====
2016-05-18 07:55:50,934 | INFO | config-pusher | ConfigPusherImpl | 131 - org.opendaylight.controller.config-persister-impl - 0.4.2.Beryllium-SR2 | Pushing configuration snapshot 99-netconf-connector.xml(bvc-base,bvc-base)
2016-05-18 07:55:51,241 | ERROR | ult-dispatcher-2 | TransactionChainProxy | 179 - org.opendaylight.controller.sal-distributed-datastore - 1.3.2.Beryllium-SR2 | Tx: member-2-chn-1-txn-2-1463583351233 - ready future failed for previous Tx member-2-chn-1-txn-2-1463583351233
2016-05-18 07:55:51,242 | WARN | lt-dispatcher-19 | ConcurrentDOMDataBroker | 179 - org.opendaylight.controller.sal-distributed-datastore - 1.3.2.Beryllium-SR2 | Tx: DOM-CHAIN-0-0 Error during phase CAN_COMMIT, starting Abort
OptimisticLockFailedException

{message=Optimistic lock failed., errorList=[RpcError [message=Optimistic lock failed., severity=ERROR, errorType=APPLICATION, tag=resource-denied, applicationTag=null, info=null, cause=org.opendaylight.yangtools.yang.data.api.schema.tree.ConflictingModificationAppliedException: Node was created by other transaction.]]}

at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.canCommit(SimpleShardDataTreeCohort.java:54)
at org.opendaylight.controller.cluster.datastore.ChainedCommitCohort.canCommit(ChainedCommitCohort.java:52)
at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator$CohortEntry.canCommit(ShardCommitCoordinator.java:662)
at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.doCanCommit(ShardCommitCoordinator.java:330)
at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.maybeProcessNextCohortEntry(ShardCommitCoordinator.java:557)
at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.currentTransactionComplete(ShardCommitCoordinator.java:538)
at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.doCanCommit(ShardCommitCoordinator.java:358)
at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.maybeProcessNextCohortEntry(ShardCommitCoordinator.java:557)
at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.currentTransactionComplete(ShardCommitCoordinator.java:538)
at org.opendaylight.controller.cluster.datastore.Shard.finishCommit(Shard.java:379)
at org.opendaylight.controller.cluster.datastore.Shard.finishCommit(Shard.java:415)
at org.opendaylight.controller.cluster.datastore.Shard.applyState(Shard.java:663)
at org.opendaylight.controller.cluster.raft.RaftActor.handleCommand(RaftActor.java:224)
at org.opendaylight.controller.cluster.common.actor.AbstractUntypedPersistentActor.onReceiveCommand(AbstractUntypedPersistentActor.java:36)
at org.opendaylight.controller.cluster.datastore.Shard.onReceiveCommand(Shard.java:275)
at akka.persistence.UntypedPersistentActor.onReceive(Eventsourced.scala:430)
at org.opendaylight.controller.cluster.common.actor.MeteringBehavior.apply(MeteringBehavior.java:97)
at akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:534)
at akka.persistence.Recovery$State$class.process(Recovery.scala:30)
at akka.persistence.ProcessorImpl$$anon$2.process(Processor.scala:103)
at akka.persistence.ProcessorImpl$$anon$2.aroundReceive(Processor.scala:114)
at akka.persistence.Recovery$class.aroundReceive(Recovery.scala:265)
at akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(Eventsourced.scala:428)
at akka.persistence.Eventsourced$$anon$2.doAroundReceive(Eventsourced.scala:82)
at akka.persistence.Eventsourced$$anon$2.aroundReceive(Eventsourced.scala:78)
at akka.persistence.Eventsourced$class.aroundReceive(Eventsourced.scala:369)
at akka.persistence.UntypedPersistentActor.aroundReceive(Eventsourced.scala:428)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
at akka.dispatch.Mailbox.run(Mailbox.scala:220)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: org.opendaylight.yangtools.yang.data.api.schema.tree.ConflictingModificationAppliedException: Node was created by other transaction.
at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkWriteApplicable(SchemaAwareApplyOperation.java:173)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:127)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:300)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkMergeApplicable(AbstractNodeContainerModificationStrategy.java:308)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.MinMaxElementsValidation.checkMergeApplicable(MinMaxElementsValidation.java:117)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:130)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:300)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkMergeApplicable(AbstractNodeContainerModificationStrategy.java:308)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:130)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.StructuralContainerModificationStrategy.checkApplicable(StructuralContainerModificationStrategy.java:98)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:300)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkTouchApplicable(AbstractNodeContainerModificationStrategy.java:283)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:124)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.RootModificationApplyOperation.checkApplicable(RootModificationApplyOperation.java:72)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractDataTreeTip.validate(AbstractDataTreeTip.java:38)
at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.canCommit(SimpleShardDataTreeCohort.java:48)
... 35 more
2016-05-18 07:55:51,247 | ERROR | CommitFutures-1 | NetconfDeviceSalProvider | 250 - org.opendaylight.netconf.sal-netconf-connector - 1.3.2.Beryllium-SR2 | RemoteDevice

{controller-config}: TransactionChain(org.opendaylight.controller.md.sal.binding.impl.BindingDOMTransactionChainAdapter@2aee71df) DOM-CHAIN-0-0 FAILED!
OptimisticLockFailedException{message=Optimistic lock failed., errorList=[RpcError [message=Optimistic lock failed., severity=ERROR, errorType=APPLICATION, tag=resource-denied, applicationTag=null, info=null, cause=org.opendaylight.yangtools.yang.data.api.schema.tree.ConflictingModificationAppliedException: Node was created by other transaction.]]}
at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.canCommit(SimpleShardDataTreeCohort.java:54)
at org.opendaylight.controller.cluster.datastore.ChainedCommitCohort.canCommit(ChainedCommitCohort.java:52)
at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator$CohortEntry.canCommit(ShardCommitCoordinator.java:662)
at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.doCanCommit(ShardCommitCoordinator.java:330)
at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.maybeProcessNextCohortEntry(ShardCommitCoordinator.java:557)
at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.currentTransactionComplete(ShardCommitCoordinator.java:538)
at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.doCanCommit(ShardCommitCoordinator.java:358)
at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.maybeProcessNextCohortEntry(ShardCommitCoordinator.java:557)
at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.currentTransactionComplete(ShardCommitCoordinator.java:538)
at org.opendaylight.controller.cluster.datastore.Shard.finishCommit(Shard.java:379)
at org.opendaylight.controller.cluster.datastore.Shard.finishCommit(Shard.java:415)
at org.opendaylight.controller.cluster.datastore.Shard.applyState(Shard.java:663)
at org.opendaylight.controller.cluster.raft.RaftActor.handleCommand(RaftActor.java:224)
at org.opendaylight.controller.cluster.common.actor.AbstractUntypedPersistentActor.onReceiveCommand(AbstractUntypedPersistentActor.java:36)
at org.opendaylight.controller.cluster.datastore.Shard.onReceiveCommand(Shard.java:275)
at akka.persistence.UntypedPersistentActor.onReceive(Eventsourced.scala:430)
at org.opendaylight.controller.cluster.common.actor.MeteringBehavior.apply(MeteringBehavior.java:97)
at akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:534)
at akka.persistence.Recovery$State$class.process(Recovery.scala:30)
at akka.persistence.ProcessorImpl$$anon$2.process(Processor.scala:103)
at akka.persistence.ProcessorImpl$$anon$2.aroundReceive(Processor.scala:114)
at akka.persistence.Recovery$class.aroundReceive(Recovery.scala:265)
at akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(Eventsourced.scala:428)
at akka.persistence.Eventsourced$$anon$2.doAroundReceive(Eventsourced.scala:82)
at akka.persistence.Eventsourced$$anon$2.aroundReceive(Eventsourced.scala:78)
at akka.persistence.Eventsourced$class.aroundReceive(Eventsourced.scala:369)
at akka.persistence.UntypedPersistentActor.aroundReceive(Eventsourced.scala:428)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
at akka.dispatch.Mailbox.run(Mailbox.scala:220)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: org.opendaylight.yangtools.yang.data.api.schema.tree.ConflictingModificationAppliedException: Node was created by other transaction.
at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkWriteApplicable(SchemaAwareApplyOperation.java:173)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:127)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:300)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkMergeApplicable(AbstractNodeContainerModificationStrategy.java:308)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.MinMaxElementsValidation.checkMergeApplicable(MinMaxElementsValidation.java:117)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:130)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:300)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkMergeApplicable(AbstractNodeContainerModificationStrategy.java:308)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:130)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.StructuralContainerModificationStrategy.checkApplicable(StructuralContainerModificationStrategy.java:98)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:300)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkTouchApplicable(AbstractNodeContainerModificationStrategy.java:283)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:124)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.RootModificationApplyOperation.checkApplicable(RootModificationApplyOperation.java:72)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractDataTreeTip.validate(AbstractDataTreeTip.java:38)
at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.canCommit(SimpleShardDataTreeCohort.java:48)
... 35 more
2016-05-18 07:55:51,247 | ERROR | CommitFutures-0 | NetconfDeviceDatastoreAdapter | 250 - org.opendaylight.netconf.sal-netconf-connector - 1.3.2.Beryllium-SR2 | RemoteDevice{controller-config}

: Transaction(init) DOM-CHAIN-0-0 FAILED!
OptimisticLockFailedException

at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.canCommit(SimpleShardDataTreeCohort.java:54)
at org.opendaylight.controller.cluster.datastore.ChainedCommitCohort.canCommit(ChainedCommitCohort.java:52)
at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator$CohortEntry.canCommit(ShardCommitCoordinator.java:662)
at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.doCanCommit(ShardCommitCoordinator.java:330)
at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.maybeProcessNextCohortEntry(ShardCommitCoordinator.java:557)
at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.currentTransactionComplete(ShardCommitCoordinator.java:538)
at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.doCanCommit(ShardCommitCoordinator.java:358)
at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.maybeProcessNextCohortEntry(ShardCommitCoordinator.java:557)
at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.currentTransactionComplete(ShardCommitCoordinator.java:538)
at org.opendaylight.controller.cluster.datastore.Shard.finishCommit(Shard.java:379)
at org.opendaylight.controller.cluster.datastore.Shard.finishCommit(Shard.java:415)
at org.opendaylight.controller.cluster.datastore.Shard.applyState(Shard.java:663)
at org.opendaylight.controller.cluster.raft.RaftActor.handleCommand(RaftActor.java:224)
at org.opendaylight.controller.cluster.common.actor.AbstractUntypedPersistentActor.onReceiveCommand(AbstractUntypedPersistentActor.java:36)
at org.opendaylight.controller.cluster.datastore.Shard.onReceiveCommand(Shard.java:275)
at akka.persistence.UntypedPersistentActor.onReceive(Eventsourced.scala:430)
at org.opendaylight.controller.cluster.common.actor.MeteringBehavior.apply(MeteringBehavior.java:97)
at akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:534)
at akka.persistence.Recovery$State$class.process(Recovery.scala:30)
at akka.persistence.ProcessorImpl$$anon$2.process(Processor.scala:103)
at akka.persistence.ProcessorImpl$$anon$2.aroundReceive(Processor.scala:114)
at akka.persistence.Recovery$class.aroundReceive(Recovery.scala:265)
at akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(Eventsourced.scala:428)
at akka.persistence.Eventsourced$$anon$2.doAroundReceive(Eventsourced.scala:82)
at akka.persistence.Eventsourced$$anon$2.aroundReceive(Eventsourced.scala:78)
at akka.persistence.Eventsourced$class.aroundReceive(Eventsourced.scala:369)
at akka.persistence.UntypedPersistentActor.aroundReceive(Eventsourced.scala:428)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
at akka.dispatch.Mailbox.run(Mailbox.scala:220)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: org.opendaylight.yangtools.yang.data.api.schema.tree.ConflictingModificationAppliedException: Node was created by other transaction.
at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkWriteApplicable(SchemaAwareApplyOperation.java:173)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:127)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:300)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkMergeApplicable(AbstractNodeContainerModificationStrategy.java:308)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.MinMaxElementsValidation.checkMergeApplicable(MinMaxElementsValidation.java:117)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:130)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:300)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkMergeApplicable(AbstractNodeContainerModificationStrategy.java:308)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:130)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.StructuralContainerModificationStrategy.checkApplicable(StructuralContainerModificationStrategy.java:98)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:300)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkTouchApplicable(AbstractNodeContainerModificationStrategy.java:283)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:124)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.RootModificationApplyOperation.checkApplicable(RootModificationApplyOperation.java:72)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractDataTreeTip.validate(AbstractDataTreeTip.java:38)
at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.canCommit(SimpleShardDataTreeCohort.java:48)
... 35 more
2016-05-18 07:55:51,332 | ERROR | config-pusher | ConfigTransactionControllerImpl | 122 - org.opendaylight.controller.config-manager - 0.4.2.Beryllium-SR2 | Commit failed on ModuleIdentifier

{factoryName='sal-netconf-connector', instanceName='controller-config'} in transaction TransactionIdentifier{name='ConfigTransaction-65-67'}
java.lang.IllegalStateException: Transaction chain has failed
at com.google.common.base.Preconditions.checkState(Preconditions.java:173)
at org.opendaylight.controller.cluster.databroker.DOMBrokerTransactionChain.checkNotFailed(DOMBrokerTransactionChain.java:73)
at org.opendaylight.controller.cluster.databroker.DOMBrokerTransactionChain.submit(DOMBrokerTransactionChain.java:84)
at org.opendaylight.controller.cluster.databroker.AbstractDOMBrokerWriteTransaction.submit(AbstractDOMBrokerWriteTransaction.java:137)
at org.opendaylight.controller.md.sal.binding.impl.AbstractWriteTransaction.doSubmit(AbstractWriteTransaction.java:134)
at org.opendaylight.controller.md.sal.binding.impl.BindingDOMWriteTransactionAdapter.submit(BindingDOMWriteTransactionAdapter.java:83)
at org.opendaylight.controller.md.sal.binding.impl.BindingDOMTransactionChainAdapter$2.submit(BindingDOMTransactionChainAdapter.java:80)
at org.opendaylight.netconf.sal.connect.netconf.sal.NetconfDeviceTopologyAdapter.commitTransaction(NetconfDeviceTopologyAdapter.java:215)
at org.opendaylight.netconf.sal.connect.netconf.sal.NetconfDeviceTopologyAdapter.initDeviceData(NetconfDeviceTopologyAdapter.java:121)
at org.opendaylight.netconf.sal.connect.netconf.sal.NetconfDeviceTopologyAdapter.<init>(NetconfDeviceTopologyAdapter.java:89)
at org.opendaylight.netconf.sal.connect.netconf.sal.NetconfDeviceSalProvider.onSessionInitiated(NetconfDeviceSalProvider.java:109)
at org.opendaylight.controller.md.sal.binding.util.BindingContextUtils.createProviderContextAndInitialize(BindingContextUtils.java:56)
at org.opendaylight.controller.sal.binding.impl.RootBindingAwareBroker.registerProvider(RootBindingAwareBroker.java:187)
at org.opendaylight.netconf.sal.connect.netconf.sal.NetconfDeviceSalFacade.registerToSal(NetconfDeviceSalFacade.java:44)
at org.opendaylight.netconf.sal.connect.netconf.sal.NetconfDeviceSalFacade.<init>(NetconfDeviceSalFacade.java:39)
at org.opendaylight.controller.config.yang.md.sal.connector.netconf.NetconfConnectorModule.createInstance(NetconfConnectorModule.java:199)
at org.opendaylight.controller.config.spi.AbstractModule.getInstance(AbstractModule.java:82)
at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.7.0_95]
at java.lang.reflect.Method.invoke(Method.java:606)[:1.7.0_95]
at org.opendaylight.controller.config.manager.impl.dependencyresolver.DependencyResolverManager$ModuleInvocationHandler.handleInvocation(DependencyResolverManager.java:149)
at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
at com.sun.proxy.$Proxy44.getInstance(Unknown Source)
at org.opendaylight.controller.config.manager.impl.ConfigTransactionControllerImpl.secondPhaseCommit(ConfigTransactionControllerImpl.java:421)
at org.opendaylight.controller.config.manager.impl.ConfigRegistryImpl.secondPhaseCommit(ConfigRegistryImpl.java:287)
at org.opendaylight.controller.config.manager.impl.ConfigRegistryImpl.commitConfig(ConfigRegistryImpl.java:227)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.7.0_95]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)[:1.7.0_95]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.7.0_95]
at java.lang.reflect.Method.invoke(Method.java:606)[:1.7.0_95]
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)[:1.7.0_95]
at sun.reflect.GeneratedMethodAccessor32.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.7.0_95]
at java.lang.reflect.Method.invoke(Method.java:606)[:1.7.0_95]
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279)[:1.7.0_95]
at com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(ConvertingMethod.java:193)[:1.7.0_95]
at com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(ConvertingMethod.java:175)[:1.7.0_95]
at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:117)[:1.7.0_95]
at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:54)[:1.7.0_95]
at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)[:1.7.0_95]
at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)[:1.7.0_95]
at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)[:1.7.0_95]
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)[:1.7.0_95]
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)[:1.7.0_95]
at com.sun.jmx.mbeanserver.MXBeanProxy$InvokeHandler.invoke(MXBeanProxy.java:150)[:1.7.0_95]
at com.sun.jmx.mbeanserver.MXBeanProxy.invoke(MXBeanProxy.java:167)[:1.7.0_95]
at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:258)[:1.7.0_95]
at com.sun.proxy.$Proxy21.commitConfig(Unknown Source)[64:org.opendaylight.controller.config-api:0.4.2.Beryllium-SR2]
at org.opendaylight.controller.config.util.ConfigRegistryJMXClient.commitConfig(ConfigRegistryJMXClient.java:110)[90:org.opendaylight.controller.config-util:0.4.2.Beryllium-SR2]
at org.opendaylight.controller.config.facade.xml.transactions.TransactionProvider.commitTransaction(TransactionProvider.java:142)[123:org.opendaylight.controller.config-manager-facade-xml:0.4.2.Beryllium-SR2]
at org.opendaylight.controller.config.facade.xml.ConfigSubsystemFacade.commitSilentTransaction(ConfigSubsystemFacade.java:138)[123:org.opendaylight.controller.config-manager-facade-xml:0.4.2.Beryllium-SR2]
at org.opendaylight.controller.config.persist.impl.ConfigPusherImpl.pushConfig(ConfigPusherImpl.java:312)[131:org.opendaylight.controller.config-persister-impl:0.4.2.Beryllium-SR2]
at org.opendaylight.controller.config.persist.impl.ConfigPusherImpl.pushConfigWithConflictingVersionRetries(ConfigPusherImpl.java:164)[131:org.opendaylight.controller.config-persister-impl:0.4.2.Beryllium-SR2]
at org.opendaylight.controller.config.persist.impl.ConfigPusherImpl.internalPushConfigs(ConfigPusherImpl.java:129)[131:org.opendaylight.controller.config-persister-impl:0.4.2.Beryllium-SR2]
at org.opendaylight.controller.config.persist.impl.ConfigPusherImpl.processSingle(ConfigPusherImpl.java:83)[131:org.opendaylight.controller.config-persister-impl:0.4.2.Beryllium-SR2]
at org.opendaylight.controller.config.persist.impl.ConfigPusherImpl.process(ConfigPusherImpl.java:74)[131:org.opendaylight.controller.config-persister-impl:0.4.2.Beryllium-SR2]
at org.opendaylight.controller.config.persist.impl.osgi.ConfigPersisterActivator$2.run(ConfigPersisterActivator.java:131)[131:org.opendaylight.controller.config-persister-impl:0.4.2.Beryllium-SR2]
at java.lang.Thread.run(Thread.java:745)[:1.7.0_95]
2016-05-18 07:55:51,342 | ERROR | config-pusher | ConfigRegistryImpl | 122 - org.opendaylight.controller.config-manager - 0.4.2.Beryllium-SR2 | Configuration Transaction failed on 2PC, server is unhealthy
java.lang.IllegalStateException: Error - getInstance() failed for ModuleIdentifier{factoryName='sal-netconf-connector', instanceName='controller-config'}

in transaction TransactionIdentifier

{name='ConfigTransaction-65-67'}

at org.opendaylight.controller.config.manager.impl.ConfigTransactionControllerImpl.secondPhaseCommit(ConfigTransactionControllerImpl.java:427)
at org.opendaylight.controller.config.manager.impl.ConfigRegistryImpl.secondPhaseCommit(ConfigRegistryImpl.java:287)
at org.opendaylight.controller.config.manager.impl.ConfigRegistryImpl.commitConfig(ConfigRegistryImpl.java:227)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.7.0_95]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)[:1.7.0_95]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.7.0_95]
at java.lang.reflect.Method.invoke(Method.java:606)[:1.7.0_95]
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)[:1.7.0_95]
at sun.reflect.GeneratedMethodAccessor32.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.7.0_95]
at java.lang.reflect.Method.invoke(Method.java:606)[:1.7.0_95]
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279)[:1.7.0_95]
at com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(ConvertingMethod.java:193)[:1.7.0_95]
at com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(ConvertingMethod.java:175)[:1.7.0_95]
at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:117)[:1.7.0_95]
at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:54)[:1.7.0_95]
at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)[:1.7.0_95]
at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)[:1.7.0_95]
at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)[:1.7.0_95]
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)[:1.7.0_95]
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)[:1.7.0_95]
at com.sun.jmx.mbeanserver.MXBeanProxy$InvokeHandler.invoke(MXBeanProxy.java:150)[:1.7.0_95]
at com.sun.jmx.mbeanserver.MXBeanProxy.invoke(MXBeanProxy.java:167)[:1.7.0_95]
at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:258)[:1.7.0_95]
at com.sun.proxy.$Proxy21.commitConfig(Unknown Source)[64:org.opendaylight.controller.config-api:0.4.2.Beryllium-SR2]
at org.opendaylight.controller.config.util.ConfigRegistryJMXClient.commitConfig(ConfigRegistryJMXClient.java:110)[90:org.opendaylight.controller.config-util:0.4.2.Beryllium-SR2]
at org.opendaylight.controller.config.facade.xml.transactions.TransactionProvider.commitTransaction(TransactionProvider.java:142)[123:org.opendaylight.controller.config-manager-facade-xml:0.4.2.Beryllium-SR2]
at org.opendaylight.controller.config.facade.xml.ConfigSubsystemFacade.commitSilentTransaction(ConfigSubsystemFacade.java:138)[123:org.opendaylight.controller.config-manager-facade-xml:0.4.2.Beryllium-SR2]
at org.opendaylight.controller.config.persist.impl.ConfigPusherImpl.pushConfig(ConfigPusherImpl.java:312)[131:org.opendaylight.controller.config-persister-impl:0.4.2.Beryllium-SR2]
at org.opendaylight.controller.config.persist.impl.ConfigPusherImpl.pushConfig...

OpenDaylight JIRA

[NETCONF-206] Netconf throws exception on initial start in 3 N cluster - prevents self mount