[CONTROLLER-975] Pcep and clustering, starting at once, lead to Transaction factory was closed. Created: 30/Oct/14  Updated: 25/Jul/23  Resolved: 03/Feb/15

Status: Resolved
Project: controller
Component/s: config
Affects Version/s: None
Fix Version/s: None

Type: Bug
Reporter: Vratko Polak Assignee: Maros Marsalek
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: All
Platform: All


Attachments: File cluster_20141029.log.xz     File karaf_20141107.log.xz    
Issue Links:
Blocks
blocks BGPCEP-181 Milestone: Implement RFC5575 Resolved
External issue ID: 2283

 Description   

This looks like a direct analogue of https://bugs.opendaylight.org/show_bug.cgi?id=1535

Attached compressed log, where:
0. Karaf is re-started after being gracefully shut down with bunch of features installed.
1. PCEP services starts up successfully, sessions are established.
2. Clustering features finish booting up.
3. A (maybe unrelated?) controller configuration change triggers a nasty error.

https://git.opendaylight.org/gerrit/#/c/10014/ looks like easily transferrable fix.

This bug is filed against PCEP, but BGP has it too; who knows how many other projects could use the fix too.



 Comments   
Comment by Vratko Polak [ 30/Oct/14 ]

Attachment cluster_20141029.log.xz has been added with description: XZipped complete karaf.log file

Comment by Vratko Polak [ 30/Oct/14 ]

Workaround: Before re-starting karaf, delete data/cache,
after starting karaf: install clustering features, wait for them to became fully ready, only then install bgpcep features.

Comment by Dana Kutenicsova [ 03/Nov/14 ]

Both protocols have it, because its clustering issue.

Comment by Tom Pantelis [ 07/Nov/14 ]

I'm not clear as to what the issue is. Can you please paste the relevant error info from the log into this bug and elaborate? I can't load the attachment (I've never even heard of XZip).

Comment by Dana Kutenicsova [ 07/Nov/14 ]

http://en.wikipedia.org/wiki/Xz - unxz works fine for unzipping

2014-10-29 17:44:55,742 | ERROR | config-pusher | ConfigRegistryImpl | 110 - org.opendaylight.controller.config-manager - 0.2.5.Helium | Configuration Transaction failed on 2PC, server is unhealthy
java.lang.IllegalStateException: Error - getInstance() failed for ModuleIdentifier

{factoryName='pcep-topology-provider', instanceName='pcep1-topology'}

in transaction TransactionIdentifier

{name='ConfigTransaction-114-116'}

at org.opendaylight.controller.config.manager.impl.ConfigTransactionControllerImpl.secondPhaseCommit(ConfigTransactionControllerImpl.java:402)
at org.opendaylight.controller.config.manager.impl.ConfigRegistryImpl.secondPhaseCommit(ConfigRegistryImpl.java:282)
at org.opendaylight.controller.config.manager.impl.ConfigRegistryImpl.commitConfig(ConfigRegistryImpl.java:229)
at sun.reflect.GeneratedMethodAccessor54.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)[:1.7.0_67]
at java.lang.reflect.Method.invoke(Unknown Source)[:1.7.0_67]
at sun.reflect.misc.Trampoline.invoke(Unknown Source)[:1.7.0_67]
at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)[:1.7.0_67]
at java.lang.reflect.Method.invoke(Unknown Source)[:1.7.0_67]
at sun.reflect.misc.MethodUtil.invoke(Unknown Source)[:1.7.0_67]
at com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(Unknown Source)[:1.7.0_67]
at com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(Unknown Source)[:1.7.0_67]
at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(Unknown Source)[:1.7.0_67]
at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(Unknown Source)[:1.7.0_67]
at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(Unknown Source)[:1.7.0_67]
at com.sun.jmx.mbeanserver.PerInterface.invoke(Unknown Source)[:1.7.0_67]
at com.sun.jmx.mbeanserver.MBeanSupport.invoke(Unknown Source)[:1.7.0_67]
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Unknown Source)[:1.7.0_67]
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown Source)[:1.7.0_67]
at com.sun.jmx.mbeanserver.MXBeanProxy$InvokeHandler.invoke(Unknown Source)[:1.7.0_67]
at com.sun.jmx.mbeanserver.MXBeanProxy.invoke(Unknown Source)[:1.7.0_67]
at javax.management.MBeanServerInvocationHandler.invoke(Unknown Source)[:1.7.0_67]
at com.sun.proxy.$Proxy15.commitConfig(Unknown Source)[104:org.opendaylight.controller.config-api:0.2.5.Helium]
at org.opendaylight.controller.config.util.ConfigRegistryJMXClient.commitConfig(ConfigRegistryJMXClient.java:102)[105:org.opendaylight.controller.config-util:0.2.5.Helium]
at org.opendaylight.controller.netconf.confignetconfconnector.transactions.TransactionProvider.commitTransaction(TransactionProvider.java:110)[122:org.opendaylight.controller.config-netconf-connector:0.2.5.Helium]
at org.opendaylight.controller.netconf.confignetconfconnector.operations.Commit.handleWithNoSubsequentOperations(Commit.java:55)[122:org.opendaylight.controller.config-netconf-connector:0.2.5.Helium]
at org.opendaylight.controller.netconf.util.mapping.AbstractLastNetconfOperation.handle(AbstractLastNetconfOperation.java:33)[121:org.opendaylight.controller.netconf-util:0.2.5.Helium]
at org.opendaylight.controller.netconf.util.mapping.AbstractNetconfOperation.handle(AbstractNetconfOperation.java:105)[121:org.opendaylight.controller.netconf-util:0.2.5.Helium]
at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.sendRequestGetResponseCheckIsOK(ConfigPusherImpl.java:285)[125:org.opendaylight.controller.config-persister-impl:0.2.5.Helium]
at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.pushConfig(ConfigPusherImpl.java:244)[125:org.opendaylight.controller.config-persister-impl:0.2.5.Helium]
at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.pushConfigWithConflictingVersionRetries(ConfigPusherImpl.java:133)[125:org.opendaylight.controller.config-persister-impl:0.2.5.Helium]
at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.internalPushConfigs(ConfigPusherImpl.java:109)[125:org.opendaylight.controller.config-persister-impl:0.2.5.Helium]
at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.process(ConfigPusherImpl.java:76)[125:org.opendaylight.controller.config-persister-impl:0.2.5.Helium]
at org.opendaylight.controller.netconf.persist.impl.osgi.ConfigPersisterActivator$InnerCustomizer$1.run(ConfigPersisterActivator.java:178)[125:org.opendaylight.controller.config-persister-impl:0.2.5.Helium]
at java.lang.Thread.run(Unknown Source)[:1.7.0_67]
Caused by: java.lang.IllegalStateException: Failed to acquire instruction queue
at org.opendaylight.bgpcep.programming.impl.ProgrammingServiceImpl.<init>(ProgrammingServiceImpl.java:125)
at org.opendaylight.controller.config.yang.programming.impl.InstructionSchedulerImplModule.createInstance(InstructionSchedulerImplModule.java:57)
at org.opendaylight.controller.config.yang.programming.impl.AbstractInstructionSchedulerImplModule.getInstance(AbstractInstructionSchedulerImplModule.java:96)
at sun.reflect.GeneratedMethodAccessor32.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)[:1.7.0_67]
at java.lang.reflect.Method.invoke(Unknown Source)[:1.7.0_67]
at org.opendaylight.controller.config.manager.impl.dependencyresolver.DependencyResolverManager$1.handleInvocation(DependencyResolverManager.java:152)
at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:70)
at com.sun.proxy.$Proxy30.getInstance(Unknown Source)
at org.opendaylight.controller.config.manager.impl.dependencyresolver.DependencyResolverImpl.resolveInstance(DependencyResolverImpl.java:169)
at org.opendaylight.controller.config.yang.pcep.topology.provider.AbstractPCEPTopologyProviderModule.getInstance(AbstractPCEPTopologyProviderModule.java:100)
at sun.reflect.GeneratedMethodAccessor32.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)[:1.7.0_67]
at java.lang.reflect.Method.invoke(Unknown Source)[:1.7.0_67]
at org.opendaylight.controller.config.manager.impl.dependencyresolver.DependencyResolverManager$1.handleInvocation(DependencyResolverManager.java:152)
at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:70)
at com.sun.proxy.$Proxy30.getInstance(Unknown Source)
at org.opendaylight.controller.config.manager.impl.ConfigTransactionControllerImpl.secondPhaseCommit(ConfigTransactionControllerImpl.java:396)
... 35 more
Caused by: java.util.concurrent.ExecutionException: ReadFailedException

{message=Error reading data for path /(urn:opendaylight:params:xml:ns:yang:programming?revision=2013-09-30)instructions-queue, errorList=[RpcError [message=Error read ing data for path /(urn:opendaylight:params:xml:ns:yang:programming?revision=2013-09-30)instructions-queue, severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=org.opendaylight.controller.cl uster.datastore.exceptions.TimeoutException: Sending message class org.opendaylight.controller.protobuff.messages.transaction.ShardTransactionMessages$CreateTransaction to actor ActorSelection[Anchor(akka://opendaylight-cluster-data/), P ath(/user/shardmanager-operational/member-1-shard-default-operational)] failed]]}

at org.opendaylight.yangtools.util.concurrent.MappingCheckedFuture.wrapInExecutionException(MappingCheckedFuture.java:64)
at org.opendaylight.yangtools.util.concurrent.MappingCheckedFuture.get(MappingCheckedFuture.java:77)
at org.opendaylight.bgpcep.programming.impl.ProgrammingServiceImpl.<init>(ProgrammingServiceImpl.java:123)
... 52 more
Caused by: ReadFailedException

{message=Error reading data for path /(urn:opendaylight:params:xml:ns:yang:programming?revision=2013-09-30)instructions-queue, errorList=[RpcError [message=Error reading data for path /(urn:opendaylight:params:xml:ns:yang:programming?revision=2013-09-30)instructions-queue, severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=org.opendaylight.controller.cluster.datastore.exceptions.TimeoutException: Sending message class org.opendaylight.controller.protobuff.messages.transaction.ShardTransactionMessages$CreateTransaction to actor ActorSelection[Anchor(akka://opendaylight-cluster-data/), Path(/user/shardmanager-operational/member-1-shard-default-operational)] failed]]}

at org.opendaylight.controller.cluster.datastore.TransactionProxy$NoOpTransactionContext.readData(TransactionProxy.java:836)
at org.opendaylight.controller.cluster.datastore.TransactionProxy.read(TransactionProxy.java:244)
at org.opendaylight.controller.md.sal.dom.broker.impl.DOMForwardedReadWriteTransaction.read(DOMForwardedReadWriteTransaction.java:47)
at org.opendaylight.controller.md.sal.binding.impl.AbstractForwardedTransaction.doRead(AbstractForwardedTransaction.java:63)
at org.opendaylight.controller.md.sal.binding.impl.BindingDataReadWriteTransactionImpl.read(BindingDataReadWriteTransactionImpl.java:31)
... 53 more
Caused by: org.opendaylight.controller.cluster.datastore.exceptions.TimeoutException: Sending message class org.opendaylight.controller.protobuff.messages.transaction.ShardTransactionMessages$CreateTransaction to actor ActorSelection[Anchor(akka://opendaylight-cluster-data/), Path(/user/shardmanager-operational/member-1-shard-default-operational)] failed
at org.opendaylight.controller.cluster.datastore.utils.ActorContext.executeRemoteOperation(ActorContext.java:189)
at org.opendaylight.controller.cluster.datastore.utils.ActorContext.executeShardOperation(ActorContext.java:243)
at org.opendaylight.controller.cluster.datastore.TransactionProxy.createTransactionIfMissing(TransactionProxy.java:382)
at org.opendaylight.controller.cluster.datastore.TransactionProxy.read(TransactionProxy.java:242)
... 56 more
Caused by: java.lang.IllegalStateException: Could not find leader so transaction cannot be created
at org.opendaylight.controller.cluster.datastore.Shard.onReceiveCommand(Shard.java:224)[253:org.opendaylight.controller.sal-distributed-datastore:1.1.0.Helium]
at akka.persistence.UntypedPersistentActor.onReceive(Eventsourced.scala:430)[243:com.typesafe.akka.persistence.experimental:2.3.4]
at org.opendaylight.controller.cluster.common.actor.MeteringBehavior.apply(MeteringBehavior.java:80)[245:org.opendaylight.controller.sal-clustering-commons:1.1.0.Helium]
at akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:534)[238:com.typesafe.akka.actor:2.3.4]
at akka.persistence.Recovery$State$class.process(Recovery.scala:30)[243:com.typesafe.akka.persistence.experimental:2.3.4]
at akka.persistence.ProcessorImpl$$anon$2.process(Processor.scala:103)[243:com.typesafe.akka.persistence.experimental:2.3.4]
at akka.persistence.ProcessorImpl$$anon$2.aroundReceive(Processor.scala:114)[243:com.typesafe.akka.persistence.experimental:2.3.4]
at akka.persistence.Recovery$class.aroundReceive(Recovery.scala:256)[243:com.typesafe.akka.persistence.experimental:2.3.4]
at akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(Eventsourced.scala:428)[243:com.typesafe.akka.persistence.experimental:2.3.4]
at akka.persistence.Eventsourced$$anon$2.doAroundReceive(Eventsourced.scala:82)[243:com.typesafe.akka.persistence.experimental:2.3.4]
at akka.persistence.Eventsourced$$anon$2.aroundReceive(Eventsourced.scala:78)[243:com.typesafe.akka.persistence.experimental:2.3.4]
at akka.persistence.Eventsourced$class.aroundReceive(Eventsourced.scala:369)[243:com.typesafe.akka.persistence.experimental:2.3.4]
at akka.persistence.UntypedPersistentActor.aroundReceive(Eventsourced.scala:428)[243:com.typesafe.akka.persistence.experimental:2.3.4]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)[238:com.typesafe.akka.actor:2.3.4]
at akka.actor.ActorCell.invoke(ActorCell.scala:487)[238:com.typesafe.akka.actor:2.3.4]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)[238:com.typesafe.akka.actor:2.3.4]
at akka.dispatch.Mailbox.run(Mailbox.scala:220)[238:com.typesafe.akka.actor:2.3.4]
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)[238:com.typesafe.akka.actor:2.3.4]
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)[235:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b]
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)[235:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b]
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)[235:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b]
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)[235:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b]

Comment by Tom Pantelis [ 07/Nov/14 ]

Ok - the underlying root cause error is "java.lang.IllegalStateException: Could not find leader so transaction cannot be created". This can happen on startup if the shard has not elected a leader yet when a transaction is created. This has been addressed by https://git.opendaylight.org/gerrit/#/c/12354/ which was merged on Oct 29th. The build that this error was seen did not have these changes (as evidenced by TransactionProxy.createTransactionIfMissing in the trace).

This issue should be alleviated now. Please retest.

Comment by Vratko Polak [ 07/Nov/14 ]

Attachment karaf_20141107.log.xz has been added with description: xz of full log of 2014-11-07 re-test

Comment by Vratko Polak [ 07/Nov/14 ]

Re-tested on this image:
https://nexus.opendaylight.org/content/groups/staging/org/opendaylight/integration/distribution-karaf/0.2.1-Helium-SR1/distribution-karaf-0.2.1-Helium-SR1.tar.gz

Titular error is still there, but now it happens before clustering finishes booting up.
I vaguely remember a gerrit change described similarly to "separate config modules from datastore" which I think may be related to this error, but my search skills are not good enough to find and check status of that gerrit.

Here is relevant piece of (compressed attached) log:

2014-11-07 15:40:22,301 | INFO | config-pusher | RootBindingAwareBroker | 218 - org.opendaylight.controller.sal-binding-broker-impl - 1.1.1.Helium-SR1 | Starting Binding Aware Broker: binding-broker-impl
2014-11-07 15:40:22,303 | ERROR | config-pusher | ctInstructionSchedulerImplModule | 284 - org.opendaylight.bgpcep.programming-impl - 0.3.2.Helium-SR1 | An error occurred while closing old instance org.opendaylight.controller.config.yang.programming.impl.InstructionSchedulerImplModule$1ProgrammingServiceImplCloseable@6f15ff44
java.lang.IllegalStateException: Transaction factory was closed. No further operations allowed.
at com.google.common.base.Preconditions.checkState(Preconditions.java:149)
at org.opendaylight.controller.md.sal.dom.broker.impl.AbstractDOMForwardedTransactionFactory.checkNotClosed(AbstractDOMForwardedTransactionFactory.java:209)
at org.opendaylight.controller.md.sal.dom.broker.impl.AbstractDOMForwardedTransactionFactory.newWriteOnlyTransaction(AbstractDOMForwardedTransactionFactory.java:125)
at org.opendaylight.controller.sal.dom.broker.osgi.DOMDataBrokerProxy.newWriteOnlyTransaction(DOMDataBrokerProxy.java:33)
at org.opendaylight.controller.md.sal.binding.impl.ForwardedBindingDataBroker.newWriteOnlyTransaction(ForwardedBindingDataBroker.java:49)
at org.opendaylight.bgpcep.programming.impl.ProgrammingServiceImpl.close(ProgrammingServiceImpl.java:345)
at org.opendaylight.controller.config.yang.programming.impl.InstructionSchedulerImplModule$1ProgrammingServiceImplCloseable.close(InstructionSchedulerImplModule.java:68)
at org.opendaylight.controller.config.yang.programming.impl.AbstractInstructionSchedulerImplModule.getInstance(AbstractInstructionSchedulerImplModule.java:91)
at sun.reflect.GeneratedMethodAccessor32.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)[:1.7.0_67]
at java.lang.reflect.Method.invoke(Unknown Source)[:1.7.0_67]
at org.opendaylight.controller.config.manager.impl.dependencyresolver.DependencyResolverManager$1.handleInvocation(DependencyResolverManager.java:152)
at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:70)
at com.sun.proxy.$Proxy67.getInstance(Unknown Source)
at org.opendaylight.controller.config.manager.impl.ConfigTransactionControllerImpl.secondPhaseCommit(ConfigTransactionControllerImpl.java:396)
at org.opendaylight.controller.config.manager.impl.ConfigRegistryImpl.secondPhaseCommit(ConfigRegistryImpl.java:282)
at org.opendaylight.controller.config.manager.impl.ConfigRegistryImpl.commitConfig(ConfigRegistryImpl.java:229)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.7.0_67]
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)[:1.7.0_67]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)[:1.7.0_67]
at java.lang.reflect.Method.invoke(Unknown Source)[:1.7.0_67]
at sun.reflect.misc.Trampoline.invoke(Unknown Source)[:1.7.0_67]
at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)[:1.7.0_67]
at java.lang.reflect.Method.invoke(Unknown Source)[:1.7.0_67]
at sun.reflect.misc.MethodUtil.invoke(Unknown Source)[:1.7.0_67]
at com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(Unknown Source)[:1.7.0_67]
at com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(Unknown Source)[:1.7.0_67]
at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(Unknown Source)[:1.7.0_67]
at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(Unknown Source)[:1.7.0_67]
at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(Unknown Source)[:1.7.0_67]
at com.sun.jmx.mbeanserver.PerInterface.invoke(Unknown Source)[:1.7.0_67]
at com.sun.jmx.mbeanserver.MBeanSupport.invoke(Unknown Source)[:1.7.0_67]
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Unknown Source)[:1.7.0_67]
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown Source)[:1.7.0_67]
at com.sun.jmx.mbeanserver.MXBeanProxy$InvokeHandler.invoke(Unknown Source)[:1.7.0_67]
at com.sun.jmx.mbeanserver.MXBeanProxy.invoke(Unknown Source)[:1.7.0_67]
at javax.management.MBeanServerInvocationHandler.invoke(Unknown Source)[:1.7.0_67]
at com.sun.proxy.$Proxy31.commitConfig(Unknown Source)[175:org.opendaylight.controller.config-api:0.2.6.Helium-SR1]
at org.opendaylight.controller.config.util.ConfigRegistryJMXClient.commitConfig(ConfigRegistryJMXClient.java:102)[176:org.opendaylight.controller.config-util:0.2.6.Helium-SR1]
at org.opendaylight.controller.netconf.confignetconfconnector.transactions.TransactionProvider.commitTransaction(TransactionProvider.java:110)[193:org.opendaylight.controller.config-netconf-connector:0.2.6.Helium-SR1]
at org.opendaylight.controller.netconf.confignetconfconnector.operations.Commit.handleWithNoSubsequentOperations(Commit.java:55)[193:org.opendaylight.controller.config-netconf-connector:0.2.6.Helium-SR1]
at org.opendaylight.controller.netconf.util.mapping.AbstractLastNetconfOperation.handle(AbstractLastNetconfOperation.java:33)[192:org.opendaylight.controller.netconf-util:0.2.6.Helium-SR1]
at org.opendaylight.controller.netconf.util.mapping.AbstractNetconfOperation.handle(AbstractNetconfOperation.java:105)[192:org.opendaylight.controller.netconf-util:0.2.6.Helium-SR1]
at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.sendRequestGetResponseCheckIsOK(ConfigPusherImpl.java:285)[196:org.opendaylight.controller.config-persister-impl:0.2.6.Helium-SR1]
at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.pushConfig(ConfigPusherImpl.java:244)[196:org.opendaylight.controller.config-persister-impl:0.2.6.Helium-SR1]
at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.pushConfigWithConflictingVersionRetries(ConfigPusherImpl.java:133)[196:org.opendaylight.controller.config-persister-impl:0.2.6.Helium-SR1]
at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.internalPushConfigs(ConfigPusherImpl.java:109)[196:org.opendaylight.controller.config-persister-impl:0.2.6.Helium-SR1]
at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.process(ConfigPusherImpl.java:76)[196:org.opendaylight.controller.config-persister-impl:0.2.6.Helium-SR1]
at org.opendaylight.controller.netconf.persist.impl.osgi.ConfigPersisterActivator$InnerCustomizer$1.run(ConfigPersisterActivator.java:178)[196:org.opendaylight.controller.config-persister-impl:0.2.6.Helium-SR1]
at java.lang.Thread.run(Unknown Source)[:1.7.0_67]

Comment by Vratko Polak [ 07/Nov/14 ]

> Ok - the underlying root cause error is "java.lang.IllegalStateException: Could not find leader so transaction cannot be created".

I would pattern-match this more to CONTROLLER-976 than to this one. And yes, that is perhaps fixed.
I have not tested that precisely, as CONTROLLER-976 has different steps to reproduce, namely there we make sure the other ODL instances are not up yet.

Comment by Tom Pantelis [ 07/Nov/14 ]

The "java.lang.IllegalStateException: Transaction factory was closed. No further operations allowed." errors occurs when the PCEP module is closed before spinning up a new instance. This emanates from the broker and not the data store itself and occurs when the clustering feature is installed as it configures a new broker instance. This was seen with the toaster and will occur for any module that tries to interact with the broker on close.

However, IMO, this is not really an issue with the clustered store but a general issue with the config system. The config system should not close dependencies (e.g. the broker) until all its dependents have been closed. This is what other dependency injection frameworks (e.g. Spring) do. Also this error can occur on controller shutdown or if the IMDS or CDS is reconfigured on the fly via restconf or netconf.

Comment by Maros Marsalek [ 10/Nov/14 ]

Taking this as it is in fact a config subsystem issue as Tom pointed out. For now it can be "fixed" in pcep by adding a try catch block when cleaning up resources from md-sal.

Comment by Dana Kutenicsova [ 14/Nov/14 ]

Will assign it back to Maros, once the workaround in pcep is pushed to stable/helium.

Comment by Moiz Raja [ 25/Nov/14 ]

bgp/pcep workaround

https://git.opendaylight.org/gerrit/#/c/12927/ - stable/helium
https://git.opendaylight.org/gerrit/#/c/12915/ - master

Comment by Maros Marsalek [ 04/Dec/14 ]

Fixed ordering in which close and createInstance methods are called by config subsystem when performing reconfiguration.

https://git.opendaylight.org/gerrit/#/c/13343/

Comment by George Zhao [ 19/Jan/15 ]

the change seems merged, could someone test and mark this as resolved?

Comment by Maros Marsalek [ 20/Jan/15 ]

This is not resolved. Only the workarounds were merged, the real fix is still on review:

https://git.opendaylight.org/gerrit/#/c/13343/

Marking this as CONFIRMED (as WAITING FOR REVIEW cannot be set from RESOLVED ??? )

Comment by Robert Varga [ 21/Jan/15 ]

The fix broke API contracts, so it cannot be applied to Helium SR2. Retargetted to Li M2.

Generated at Wed Feb 07 19:54:21 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.