https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-975 controller <p>This looks like a direct analogue of <a href="https://bugs.opendaylight.org/show_bug.cgi?id=1535" class="external-link" target="_blank" rel="nofollow noopener">https://bugs.opendaylight.org/show_bug.cgi?id=1535</a></p> <p>Attached compressed log, where:<br/> 0. Karaf is re-started after being gracefully shut down with bunch of features installed.<br/> 1. PCEP services starts up successfully, sessions are established.<br/> 2. Clustering features finish booting up.<br/> 3. A (maybe unrelated?) controller configuration change triggers a nasty error.</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/10014/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/10014/</a> looks like easily transferrable fix.</p> <p>This bug is filed against PCEP, but BGP has it too; who knows how many other projects could use the fix too.</p> <p>Operating System: All<br/> Platform: All</p> CONTROLLER-975

Pcep and clustering, starting at once, lead to Transaction factory was closed.

Bug Resolved Done Maros Marsalek Vratko Polak Thu, 30 Oct 2014 17:18:30 +0000 Tue, 25 Jul 2023 08:24:15 +0000 Tue, 3 Feb 2015 12:11:18 +0000 config 0 8 <p>Attachment cluster_20141029.log.xz has been added with description: XZipped complete karaf.log file</p> <p>Workaround: Before re-starting karaf, delete data/cache,<br/> after starting karaf: install clustering features, wait for them to became fully ready, only then install bgpcep features.</p> <p>Both protocols have it, because its clustering issue.</p> <p>I'm not clear as to what the issue is. Can you please paste the relevant error info from the log into this bug and elaborate? I can't load the attachment (I've never even heard of XZip).</p> <p><a href="http://en.wikipedia.org/wiki/Xz" class="external-link" target="_blank" rel="nofollow noopener">http://en.wikipedia.org/wiki/Xz</a> - unxz works fine for unzipping</p> <p>2014-10-29 17:44:55,742 | ERROR | config-pusher | ConfigRegistryImpl | 110 - org.opendaylight.controller.config-manager - 0.2.5.Helium | Configuration Transaction failed on 2PC, server is unhealthy<br/> java.lang.IllegalStateException: Error - getInstance() failed for ModuleIdentifier</p> {factoryName='pcep-topology-provider', instanceName='pcep1-topology'} <p> in transaction TransactionIdentifier</p> {name='ConfigTransaction-114-116'} <p> at org.opendaylight.controller.config.manager.impl.ConfigTransactionControllerImpl.secondPhaseCommit(ConfigTransactionControllerImpl.java:402)<br/> at org.opendaylight.controller.config.manager.impl.ConfigRegistryImpl.secondPhaseCommit(ConfigRegistryImpl.java:282)<br/> at org.opendaylight.controller.config.manager.impl.ConfigRegistryImpl.commitConfig(ConfigRegistryImpl.java:229)<br/> at sun.reflect.GeneratedMethodAccessor54.invoke(Unknown Source)<br/> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)<span class="error">&#91;:1.7.0_67&#93;</span><br/> at java.lang.reflect.Method.invoke(Unknown Source)<span class="error">&#91;:1.7.0_67&#93;</span><br/> at sun.reflect.misc.Trampoline.invoke(Unknown Source)<span class="error">&#91;:1.7.0_67&#93;</span><br/> at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)<br/> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)<span class="error">&#91;:1.7.0_67&#93;</span><br/> at java.lang.reflect.Method.invoke(Unknown Source)<span class="error">&#91;:1.7.0_67&#93;</span><br/> at sun.reflect.misc.MethodUtil.invoke(Unknown Source)<span class="error">&#91;:1.7.0_67&#93;</span><br/> at com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(Unknown Source)<span class="error">&#91;:1.7.0_67&#93;</span><br/> at com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(Unknown Source)<span class="error">&#91;:1.7.0_67&#93;</span><br/> at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(Unknown Source)<span class="error">&#91;:1.7.0_67&#93;</span><br/> at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(Unknown Source)<span class="error">&#91;:1.7.0_67&#93;</span><br/> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(Unknown Source)<span class="error">&#91;:1.7.0_67&#93;</span><br/> at com.sun.jmx.mbeanserver.PerInterface.invoke(Unknown Source)<span class="error">&#91;:1.7.0_67&#93;</span><br/> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(Unknown Source)<span class="error">&#91;:1.7.0_67&#93;</span><br/> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Unknown Source)<span class="error">&#91;:1.7.0_67&#93;</span><br/> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown Source)<span class="error">&#91;:1.7.0_67&#93;</span><br/> at com.sun.jmx.mbeanserver.MXBeanProxy$InvokeHandler.invoke(Unknown Source)<span class="error">&#91;:1.7.0_67&#93;</span><br/> at com.sun.jmx.mbeanserver.MXBeanProxy.invoke(Unknown Source)<span class="error">&#91;:1.7.0_67&#93;</span><br/> at javax.management.MBeanServerInvocationHandler.invoke(Unknown Source)<span class="error">&#91;:1.7.0_67&#93;</span><br/> at com.sun.proxy.$Proxy15.commitConfig(Unknown Source)<span class="error">&#91;104:org.opendaylight.controller.config-api:0.2.5.Helium&#93;</span><br/> at org.opendaylight.controller.config.util.ConfigRegistryJMXClient.commitConfig(ConfigRegistryJMXClient.java:102)<span class="error">&#91;105:org.opendaylight.controller.config-util:0.2.5.Helium&#93;</span><br/> at org.opendaylight.controller.netconf.confignetconfconnector.transactions.TransactionProvider.commitTransaction(TransactionProvider.java:110)<span class="error">&#91;122:org.opendaylight.controller.config-netconf-connector:0.2.5.Helium&#93;</span><br/> at org.opendaylight.controller.netconf.confignetconfconnector.operations.Commit.handleWithNoSubsequentOperations(Commit.java:55)<span class="error">&#91;122:org.opendaylight.controller.config-netconf-connector:0.2.5.Helium&#93;</span><br/> at org.opendaylight.controller.netconf.util.mapping.AbstractLastNetconfOperation.handle(AbstractLastNetconfOperation.java:33)<span class="error">&#91;121:org.opendaylight.controller.netconf-util:0.2.5.Helium&#93;</span><br/> at org.opendaylight.controller.netconf.util.mapping.AbstractNetconfOperation.handle(AbstractNetconfOperation.java:105)<span class="error">&#91;121:org.opendaylight.controller.netconf-util:0.2.5.Helium&#93;</span><br/> at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.sendRequestGetResponseCheckIsOK(ConfigPusherImpl.java:285)<span class="error">&#91;125:org.opendaylight.controller.config-persister-impl:0.2.5.Helium&#93;</span><br/> at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.pushConfig(ConfigPusherImpl.java:244)<span class="error">&#91;125:org.opendaylight.controller.config-persister-impl:0.2.5.Helium&#93;</span><br/> at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.pushConfigWithConflictingVersionRetries(ConfigPusherImpl.java:133)<span class="error">&#91;125:org.opendaylight.controller.config-persister-impl:0.2.5.Helium&#93;</span><br/> at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.internalPushConfigs(ConfigPusherImpl.java:109)<span class="error">&#91;125:org.opendaylight.controller.config-persister-impl:0.2.5.Helium&#93;</span><br/> at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.process(ConfigPusherImpl.java:76)<span class="error">&#91;125:org.opendaylight.controller.config-persister-impl:0.2.5.Helium&#93;</span><br/> at org.opendaylight.controller.netconf.persist.impl.osgi.ConfigPersisterActivator$InnerCustomizer$1.run(ConfigPersisterActivator.java:178)<span class="error">&#91;125:org.opendaylight.controller.config-persister-impl:0.2.5.Helium&#93;</span><br/> at java.lang.Thread.run(Unknown Source)<span class="error">&#91;:1.7.0_67&#93;</span><br/> Caused by: java.lang.IllegalStateException: Failed to acquire instruction queue<br/> at org.opendaylight.bgpcep.programming.impl.ProgrammingServiceImpl.&lt;init&gt;(ProgrammingServiceImpl.java:125)<br/> at org.opendaylight.controller.config.yang.programming.impl.InstructionSchedulerImplModule.createInstance(InstructionSchedulerImplModule.java:57)<br/> at org.opendaylight.controller.config.yang.programming.impl.AbstractInstructionSchedulerImplModule.getInstance(AbstractInstructionSchedulerImplModule.java:96)<br/> at sun.reflect.GeneratedMethodAccessor32.invoke(Unknown Source)<br/> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)<span class="error">&#91;:1.7.0_67&#93;</span><br/> at java.lang.reflect.Method.invoke(Unknown Source)<span class="error">&#91;:1.7.0_67&#93;</span><br/> at org.opendaylight.controller.config.manager.impl.dependencyresolver.DependencyResolverManager$1.handleInvocation(DependencyResolverManager.java:152)<br/> at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:70)<br/> at com.sun.proxy.$Proxy30.getInstance(Unknown Source)<br/> at org.opendaylight.controller.config.manager.impl.dependencyresolver.DependencyResolverImpl.resolveInstance(DependencyResolverImpl.java:169)<br/> at org.opendaylight.controller.config.yang.pcep.topology.provider.AbstractPCEPTopologyProviderModule.getInstance(AbstractPCEPTopologyProviderModule.java:100)<br/> at sun.reflect.GeneratedMethodAccessor32.invoke(Unknown Source)<br/> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)<span class="error">&#91;:1.7.0_67&#93;</span><br/> at java.lang.reflect.Method.invoke(Unknown Source)<span class="error">&#91;:1.7.0_67&#93;</span><br/> at org.opendaylight.controller.config.manager.impl.dependencyresolver.DependencyResolverManager$1.handleInvocation(DependencyResolverManager.java:152)<br/> at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:70)<br/> at com.sun.proxy.$Proxy30.getInstance(Unknown Source)<br/> at org.opendaylight.controller.config.manager.impl.ConfigTransactionControllerImpl.secondPhaseCommit(ConfigTransactionControllerImpl.java:396)<br/> ... 35 more<br/> Caused by: java.util.concurrent.ExecutionException: ReadFailedException</p> {message=Error reading data for path /(urn:opendaylight:params:xml:ns:yang:programming?revision=2013-09-30)instructions-queue, errorList=[RpcError [message=Error read ing data for path /(urn:opendaylight:params:xml:ns:yang:programming?revision=2013-09-30)instructions-queue, severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=org.opendaylight.controller.cl uster.datastore.exceptions.TimeoutException: Sending message class org.opendaylight.controller.protobuff.messages.transaction.ShardTransactionMessages$CreateTransaction to actor ActorSelection[Anchor(akka://opendaylight-cluster-data/), P ath(/user/shardmanager-operational/member-1-shard-default-operational)] failed]]} <p> at org.opendaylight.yangtools.util.concurrent.MappingCheckedFuture.wrapInExecutionException(MappingCheckedFuture.java:64)<br/> at org.opendaylight.yangtools.util.concurrent.MappingCheckedFuture.get(MappingCheckedFuture.java:77)<br/> at org.opendaylight.bgpcep.programming.impl.ProgrammingServiceImpl.&lt;init&gt;(ProgrammingServiceImpl.java:123)<br/> ... 52 more<br/> Caused by: ReadFailedException</p> {message=Error reading data for path /(urn:opendaylight:params:xml:ns:yang:programming?revision=2013-09-30)instructions-queue, errorList=[RpcError [message=Error reading data for path /(urn:opendaylight:params:xml:ns:yang:programming?revision=2013-09-30)instructions-queue, severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=org.opendaylight.controller.cluster.datastore.exceptions.TimeoutException: Sending message class org.opendaylight.controller.protobuff.messages.transaction.ShardTransactionMessages$CreateTransaction to actor ActorSelection[Anchor(akka://opendaylight-cluster-data/), Path(/user/shardmanager-operational/member-1-shard-default-operational)] failed]]} <p> at org.opendaylight.controller.cluster.datastore.TransactionProxy$NoOpTransactionContext.readData(TransactionProxy.java:836)<br/> at org.opendaylight.controller.cluster.datastore.TransactionProxy.read(TransactionProxy.java:244)<br/> at org.opendaylight.controller.md.sal.dom.broker.impl.DOMForwardedReadWriteTransaction.read(DOMForwardedReadWriteTransaction.java:47)<br/> at org.opendaylight.controller.md.sal.binding.impl.AbstractForwardedTransaction.doRead(AbstractForwardedTransaction.java:63)<br/> at org.opendaylight.controller.md.sal.binding.impl.BindingDataReadWriteTransactionImpl.read(BindingDataReadWriteTransactionImpl.java:31)<br/> ... 53 more<br/> Caused by: org.opendaylight.controller.cluster.datastore.exceptions.TimeoutException: Sending message class org.opendaylight.controller.protobuff.messages.transaction.ShardTransactionMessages$CreateTransaction to actor ActorSelection<span class="error">&#91;Anchor(akka://opendaylight-cluster-data/), Path(/user/shardmanager-operational/member-1-shard-default-operational)&#93;</span> failed<br/> at org.opendaylight.controller.cluster.datastore.utils.ActorContext.executeRemoteOperation(ActorContext.java:189)<br/> at org.opendaylight.controller.cluster.datastore.utils.ActorContext.executeShardOperation(ActorContext.java:243)<br/> at org.opendaylight.controller.cluster.datastore.TransactionProxy.createTransactionIfMissing(TransactionProxy.java:382)<br/> at org.opendaylight.controller.cluster.datastore.TransactionProxy.read(TransactionProxy.java:242)<br/> ... 56 more<br/> Caused by: java.lang.IllegalStateException: Could not find leader so transaction cannot be created<br/> at org.opendaylight.controller.cluster.datastore.Shard.onReceiveCommand(Shard.java:224)<span class="error">&#91;253:org.opendaylight.controller.sal-distributed-datastore:1.1.0.Helium&#93;</span><br/> at akka.persistence.UntypedPersistentActor.onReceive(Eventsourced.scala:430)<span class="error">&#91;243:com.typesafe.akka.persistence.experimental:2.3.4&#93;</span><br/> at org.opendaylight.controller.cluster.common.actor.MeteringBehavior.apply(MeteringBehavior.java:80)<span class="error">&#91;245:org.opendaylight.controller.sal-clustering-commons:1.1.0.Helium&#93;</span><br/> at akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:534)<span class="error">&#91;238:com.typesafe.akka.actor:2.3.4&#93;</span><br/> at akka.persistence.Recovery$State$class.process(Recovery.scala:30)<span class="error">&#91;243:com.typesafe.akka.persistence.experimental:2.3.4&#93;</span><br/> at akka.persistence.ProcessorImpl$$anon$2.process(Processor.scala:103)<span class="error">&#91;243:com.typesafe.akka.persistence.experimental:2.3.4&#93;</span><br/> at akka.persistence.ProcessorImpl$$anon$2.aroundReceive(Processor.scala:114)<span class="error">&#91;243:com.typesafe.akka.persistence.experimental:2.3.4&#93;</span><br/> at akka.persistence.Recovery$class.aroundReceive(Recovery.scala:256)<span class="error">&#91;243:com.typesafe.akka.persistence.experimental:2.3.4&#93;</span><br/> at akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(Eventsourced.scala:428)<span class="error">&#91;243:com.typesafe.akka.persistence.experimental:2.3.4&#93;</span><br/> at akka.persistence.Eventsourced$$anon$2.doAroundReceive(Eventsourced.scala:82)<span class="error">&#91;243:com.typesafe.akka.persistence.experimental:2.3.4&#93;</span><br/> at akka.persistence.Eventsourced$$anon$2.aroundReceive(Eventsourced.scala:78)<span class="error">&#91;243:com.typesafe.akka.persistence.experimental:2.3.4&#93;</span><br/> at akka.persistence.Eventsourced$class.aroundReceive(Eventsourced.scala:369)<span class="error">&#91;243:com.typesafe.akka.persistence.experimental:2.3.4&#93;</span><br/> at akka.persistence.UntypedPersistentActor.aroundReceive(Eventsourced.scala:428)<span class="error">&#91;243:com.typesafe.akka.persistence.experimental:2.3.4&#93;</span><br/> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)<span class="error">&#91;238:com.typesafe.akka.actor:2.3.4&#93;</span><br/> at akka.actor.ActorCell.invoke(ActorCell.scala:487)<span class="error">&#91;238:com.typesafe.akka.actor:2.3.4&#93;</span><br/> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)<span class="error">&#91;238:com.typesafe.akka.actor:2.3.4&#93;</span><br/> at akka.dispatch.Mailbox.run(Mailbox.scala:220)<span class="error">&#91;238:com.typesafe.akka.actor:2.3.4&#93;</span><br/> at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)<span class="error">&#91;238:com.typesafe.akka.actor:2.3.4&#93;</span><br/> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)<span class="error">&#91;235:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b&#93;</span><br/> at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)<span class="error">&#91;235:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b&#93;</span><br/> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)<span class="error">&#91;235:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b&#93;</span><br/> at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)<span class="error">&#91;235:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b&#93;</span></p> <p>Ok - the underlying root cause error is "java.lang.IllegalStateException: Could not find leader so transaction cannot be created". This can happen on startup if the shard has not elected a leader yet when a transaction is created. This has been addressed by <a href="https://git.opendaylight.org/gerrit/#/c/12354/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/12354/</a> which was merged on Oct 29th. The build that this error was seen did not have these changes (as evidenced by TransactionProxy.createTransactionIfMissing in the trace).</p> <p>This issue should be alleviated now. Please retest.</p> <p>Attachment karaf_20141107.log.xz has been added with description: xz of full log of 2014-11-07 re-test</p> <p>Re-tested on this image:<br/> <a href="https://nexus.opendaylight.org/content/groups/staging/org/opendaylight/integration/distribution-karaf/0.2.1-Helium-SR1/distribution-karaf-0.2.1-Helium-SR1.tar.gz" class="external-link" target="_blank" rel="nofollow noopener">https://nexus.opendaylight.org/content/groups/staging/org/opendaylight/integration/distribution-karaf/0.2.1-Helium-SR1/distribution-karaf-0.2.1-Helium-SR1.tar.gz</a></p> <p>Titular error is still there, but now it happens before clustering finishes booting up.<br/> I vaguely remember a gerrit change described similarly to "separate config modules from datastore" which I think may be related to this error, but my search skills are not good enough to find and check status of that gerrit.</p> <p>Here is relevant piece of (compressed attached) log:</p> <p>2014-11-07 15:40:22,301 | INFO | config-pusher | RootBindingAwareBroker | 218 - org.opendaylight.controller.sal-binding-broker-impl - 1.1.1.Helium-SR1 | Starting Binding Aware Broker: binding-broker-impl<br/> 2014-11-07 15:40:22,303 | ERROR | config-pusher | ctInstructionSchedulerImplModule | 284 - org.opendaylight.bgpcep.programming-impl - 0.3.2.Helium-SR1 | An error occurred while closing old instance org.opendaylight.controller.config.yang.programming.impl.InstructionSchedulerImplModule$1ProgrammingServiceImplCloseable@6f15ff44<br/> java.lang.IllegalStateException: Transaction factory was closed. No further operations allowed.<br/> at com.google.common.base.Preconditions.checkState(Preconditions.java:149)<br/> at org.opendaylight.controller.md.sal.dom.broker.impl.AbstractDOMForwardedTransactionFactory.checkNotClosed(AbstractDOMForwardedTransactionFactory.java:209)<br/> at org.opendaylight.controller.md.sal.dom.broker.impl.AbstractDOMForwardedTransactionFactory.newWriteOnlyTransaction(AbstractDOMForwardedTransactionFactory.java:125)<br/> at org.opendaylight.controller.sal.dom.broker.osgi.DOMDataBrokerProxy.newWriteOnlyTransaction(DOMDataBrokerProxy.java:33)<br/> at org.opendaylight.controller.md.sal.binding.impl.ForwardedBindingDataBroker.newWriteOnlyTransaction(ForwardedBindingDataBroker.java:49)<br/> at org.opendaylight.bgpcep.programming.impl.ProgrammingServiceImpl.close(ProgrammingServiceImpl.java:345)<br/> at org.opendaylight.controller.config.yang.programming.impl.InstructionSchedulerImplModule$1ProgrammingServiceImplCloseable.close(InstructionSchedulerImplModule.java:68)<br/> at org.opendaylight.controller.config.yang.programming.impl.AbstractInstructionSchedulerImplModule.getInstance(AbstractInstructionSchedulerImplModule.java:91)<br/> at sun.reflect.GeneratedMethodAccessor32.invoke(Unknown Source)<br/> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)<span class="error">&#91;:1.7.0_67&#93;</span><br/> at java.lang.reflect.Method.invoke(Unknown Source)<span class="error">&#91;:1.7.0_67&#93;</span><br/> at org.opendaylight.controller.config.manager.impl.dependencyresolver.DependencyResolverManager$1.handleInvocation(DependencyResolverManager.java:152)<br/> at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:70)<br/> at com.sun.proxy.$Proxy67.getInstance(Unknown Source)<br/> at org.opendaylight.controller.config.manager.impl.ConfigTransactionControllerImpl.secondPhaseCommit(ConfigTransactionControllerImpl.java:396)<br/> at org.opendaylight.controller.config.manager.impl.ConfigRegistryImpl.secondPhaseCommit(ConfigRegistryImpl.java:282)<br/> at org.opendaylight.controller.config.manager.impl.ConfigRegistryImpl.commitConfig(ConfigRegistryImpl.java:229)<br/> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)<span class="error">&#91;:1.7.0_67&#93;</span><br/> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)<span class="error">&#91;:1.7.0_67&#93;</span><br/> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)<span class="error">&#91;:1.7.0_67&#93;</span><br/> at java.lang.reflect.Method.invoke(Unknown Source)<span class="error">&#91;:1.7.0_67&#93;</span><br/> at sun.reflect.misc.Trampoline.invoke(Unknown Source)<span class="error">&#91;:1.7.0_67&#93;</span><br/> at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)<br/> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)<span class="error">&#91;:1.7.0_67&#93;</span><br/> at java.lang.reflect.Method.invoke(Unknown Source)<span class="error">&#91;:1.7.0_67&#93;</span><br/> at sun.reflect.misc.MethodUtil.invoke(Unknown Source)<span class="error">&#91;:1.7.0_67&#93;</span><br/> at com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(Unknown Source)<span class="error">&#91;:1.7.0_67&#93;</span><br/> at com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(Unknown Source)<span class="error">&#91;:1.7.0_67&#93;</span><br/> at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(Unknown Source)<span class="error">&#91;:1.7.0_67&#93;</span><br/> at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(Unknown Source)<span class="error">&#91;:1.7.0_67&#93;</span><br/> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(Unknown Source)<span class="error">&#91;:1.7.0_67&#93;</span><br/> at com.sun.jmx.mbeanserver.PerInterface.invoke(Unknown Source)<span class="error">&#91;:1.7.0_67&#93;</span><br/> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(Unknown Source)<span class="error">&#91;:1.7.0_67&#93;</span><br/> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Unknown Source)<span class="error">&#91;:1.7.0_67&#93;</span><br/> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown Source)<span class="error">&#91;:1.7.0_67&#93;</span><br/> at com.sun.jmx.mbeanserver.MXBeanProxy$InvokeHandler.invoke(Unknown Source)<span class="error">&#91;:1.7.0_67&#93;</span><br/> at com.sun.jmx.mbeanserver.MXBeanProxy.invoke(Unknown Source)<span class="error">&#91;:1.7.0_67&#93;</span><br/> at javax.management.MBeanServerInvocationHandler.invoke(Unknown Source)<span class="error">&#91;:1.7.0_67&#93;</span><br/> at com.sun.proxy.$Proxy31.commitConfig(Unknown Source)<span class="error">&#91;175:org.opendaylight.controller.config-api:0.2.6.Helium-SR1&#93;</span><br/> at org.opendaylight.controller.config.util.ConfigRegistryJMXClient.commitConfig(ConfigRegistryJMXClient.java:102)<span class="error">&#91;176:org.opendaylight.controller.config-util:0.2.6.Helium-SR1&#93;</span><br/> at org.opendaylight.controller.netconf.confignetconfconnector.transactions.TransactionProvider.commitTransaction(TransactionProvider.java:110)<span class="error">&#91;193:org.opendaylight.controller.config-netconf-connector:0.2.6.Helium-SR1&#93;</span><br/> at org.opendaylight.controller.netconf.confignetconfconnector.operations.Commit.handleWithNoSubsequentOperations(Commit.java:55)<span class="error">&#91;193:org.opendaylight.controller.config-netconf-connector:0.2.6.Helium-SR1&#93;</span><br/> at org.opendaylight.controller.netconf.util.mapping.AbstractLastNetconfOperation.handle(AbstractLastNetconfOperation.java:33)<span class="error">&#91;192:org.opendaylight.controller.netconf-util:0.2.6.Helium-SR1&#93;</span><br/> at org.opendaylight.controller.netconf.util.mapping.AbstractNetconfOperation.handle(AbstractNetconfOperation.java:105)<span class="error">&#91;192:org.opendaylight.controller.netconf-util:0.2.6.Helium-SR1&#93;</span><br/> at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.sendRequestGetResponseCheckIsOK(ConfigPusherImpl.java:285)<span class="error">&#91;196:org.opendaylight.controller.config-persister-impl:0.2.6.Helium-SR1&#93;</span><br/> at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.pushConfig(ConfigPusherImpl.java:244)<span class="error">&#91;196:org.opendaylight.controller.config-persister-impl:0.2.6.Helium-SR1&#93;</span><br/> at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.pushConfigWithConflictingVersionRetries(ConfigPusherImpl.java:133)<span class="error">&#91;196:org.opendaylight.controller.config-persister-impl:0.2.6.Helium-SR1&#93;</span><br/> at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.internalPushConfigs(ConfigPusherImpl.java:109)<span class="error">&#91;196:org.opendaylight.controller.config-persister-impl:0.2.6.Helium-SR1&#93;</span><br/> at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.process(ConfigPusherImpl.java:76)<span class="error">&#91;196:org.opendaylight.controller.config-persister-impl:0.2.6.Helium-SR1&#93;</span><br/> at org.opendaylight.controller.netconf.persist.impl.osgi.ConfigPersisterActivator$InnerCustomizer$1.run(ConfigPersisterActivator.java:178)<span class="error">&#91;196:org.opendaylight.controller.config-persister-impl:0.2.6.Helium-SR1&#93;</span><br/> at java.lang.Thread.run(Unknown Source)<span class="error">&#91;:1.7.0_67&#93;</span></p> <p>&gt; Ok - the underlying root cause error is "java.lang.IllegalStateException: Could not find leader so transaction cannot be created".</p> <p>I would pattern-match this more to <a href="https://jira.opendaylight.org/browse/CONTROLLER-976" title="Clustering: Leaderless default shard during feature installation." class="issue-link" data-issue-key="CONTROLLER-976"><del>CONTROLLER-976</del></a> than to this one. And yes, that is perhaps fixed.<br/> I have not tested that precisely, as <a href="https://jira.opendaylight.org/browse/CONTROLLER-976" title="Clustering: Leaderless default shard during feature installation." class="issue-link" data-issue-key="CONTROLLER-976"><del>CONTROLLER-976</del></a> has different steps to reproduce, namely there we make sure the other ODL instances are not up yet.</p> <p>The "java.lang.IllegalStateException: Transaction factory was closed. No further operations allowed." errors occurs when the PCEP module is closed before spinning up a new instance. This emanates from the broker and not the data store itself and occurs when the clustering feature is installed as it configures a new broker instance. This was seen with the toaster and will occur for any module that tries to interact with the broker on close.</p> <p>However, IMO, this is not really an issue with the clustered store but a general issue with the config system. The config system should not close dependencies (e.g. the broker) until all its dependents have been closed. This is what other dependency injection frameworks (e.g. Spring) do. Also this error can occur on controller shutdown or if the IMDS or CDS is reconfigured on the fly via restconf or netconf.</p> <p>Taking this as it is in fact a config subsystem issue as Tom pointed out. For now it can be "fixed" in pcep by adding a try catch block when cleaning up resources from md-sal.</p> <p>Will assign it back to Maros, once the workaround in pcep is pushed to stable/helium.</p> <p>bgp/pcep workaround </p> <p><a href="https://git.opendaylight.org/gerrit/#/c/12927/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/12927/</a> - stable/helium<br/> <a href="https://git.opendaylight.org/gerrit/#/c/12915/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/12915/</a> - master</p> <p>Fixed ordering in which close and createInstance methods are called by config subsystem when performing reconfiguration.</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/13343/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/13343/</a></p> <p>the change seems merged, could someone test and mark this as resolved?</p> <p>This is not resolved. Only the workarounds were merged, the real fix is still on review: </p> <p><a href="https://git.opendaylight.org/gerrit/#/c/13343/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/13343/</a></p> <p>Marking this as CONFIRMED (as WAITING FOR REVIEW cannot be set from RESOLVED ??? )</p> <p>The fix broke API contracts, so it cannot be applied to Helium SR2. Retargetted to Li M2.</p> Blocks BGPCEP-181 Development External issue ID 2283 External issue URL Issue Type ODL SR Target Milestone Rank 0|i02npr: