https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-858 controller <p>When attempting to install via karaf odl-mdsal-all, one gets the following errors.</p> <p>Steps to reproduce:</p> <ul class="alternate" type="square"> <li>Download latest integration branch</li> <li>maven clean install</li> <li>karaf + install:feature odl-mdsal-all</li> </ul> <p>opendaylight-user@root&gt;Exception in thread "config-pusher" java.lang.IllegalStateException: Failed to send commit for configuration 04-xsql.xml(odl-mdsal-all,odl-mdsal-xsql)<br/> at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.sendRequestGetResponseCheckIsOK(ConfigPusherImpl.java:292)<br/> at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.pushConfig(ConfigPusherImpl.java:246)<br/> at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.pushConfigWithConflictingVersionRetries(ConfigPusherImpl.java:135)<br/> at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.internalPushConfigs(ConfigPusherImpl.java:111)<br/> at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.process(ConfigPusherImpl.java:78)<br/> at org.opendaylight.controller.netconf.persist.impl.osgi.ConfigPersisterActivator$InnerCustomizer$1.run(ConfigPersisterActivator.java:178)<br/> at java.lang.Thread.run(Thread.java:744)<br/> Caused by: java.lang.IllegalStateException: Error - getInstance() failed for ModuleIdentifier</p> {factoryName='XSQL', instanceName='XSQL'} <p> in transaction TransactionIdentifier</p> {name='ConfigTransaction-50-52'} <p> at org.opendaylight.controller.config.manager.impl.ConfigTransactionControllerImpl.secondPhaseCommit(ConfigTransactionControllerImpl.java:402)<br/> at org.opendaylight.controller.config.manager.impl.ConfigRegistryImpl.secondPhaseCommit(ConfigRegistryImpl.java:282)<br/> at org.opendaylight.controller.config.manager.impl.ConfigRegistryImpl.commitConfig(ConfigRegistryImpl.java:229)<br/> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)<br/> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)<br/> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)<br/> at java.lang.reflect.Method.invoke(Method.java:606)<br/> at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)<br/> at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)<br/> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)<br/> at java.lang.reflect.Method.invoke(Method.java:606)<br/> at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279)<br/> at com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(ConvertingMethod.java:193)<br/> at com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(ConvertingMethod.java:175)<br/> at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:117)<br/> at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:54)<br/> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)<br/> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)<br/> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)<br/> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)<br/> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)<br/> at com.sun.jmx.mbeanserver.MXBeanProxy$InvokeHandler.invoke(MXBeanProxy.java:150)<br/> at com.sun.jmx.mbeanserver.MXBeanProxy.invoke(MXBeanProxy.java:167)<br/> at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:252)<br/> at com.sun.proxy.$Proxy31.commitConfig(Unknown Source)<br/> at org.opendaylight.controller.config.util.ConfigRegistryJMXClient.commitConfig(ConfigRegistryJMXClient.java:102)<br/> at org.opendaylight.controller.netconf.confignetconfconnector.transactions.TransactionProvider.commitTransaction(TransactionProvider.java:110)<br/> at org.opendaylight.controller.netconf.confignetconfconnector.operations.Commit.handleWithNoSubsequentOperations(Commit.java:55)<br/> at org.opendaylight.controller.netconf.util.mapping.AbstractLastNetconfOperation.handle(AbstractLastNetconfOperation.java:33)<br/> at org.opendaylight.controller.netconf.util.mapping.AbstractNetconfOperation.handle(AbstractNetconfOperation.java:105)<br/> at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.sendRequestGetResponseCheckIsOK(ConfigPusherImpl.java:287)<br/> ... 6 more<br/> Caused by: java.lang.IllegalStateException: Failed to read pre-existing data<br/> at org.opendaylight.controller.md.sal.binding.impl.AbstractReadWriteTransaction.ensureParentsByMerge(AbstractReadWriteTransaction.java:55)<br/> at org.opendaylight.controller.md.sal.binding.impl.AbstractWriteTransaction.put(AbstractWriteTransaction.java:44)<br/> at org.opendaylight.controller.md.sal.binding.impl.ForwardedBackwardsCompatibleDataBroker$ForwardedBackwardsCompatibleTransacion.putOperationalData(ForwardedBackwardsCompatibleDataBroker.java:218)<br/> at org.opendaylight.controller.md.sal.binding.impl.ForwardedBackwardsCompatibleDataBroker$ForwardedBackwardsCompatibleTransacion.putOperationalData(ForwardedBackwardsCompatibleDataBroker.java:186)<br/> at org.opendaylight.xsql.XSQLProvider.buildXSQL(XSQLProvider.java:31)<br/> at org.opendaylight.yang.gen.v1.http.netconfcentral.org.ns.xsql.rev140626.XSQLModule.createInstance(XSQLModule.java:26)<br/> at org.opendaylight.yang.gen.v1.http.netconfcentral.org.ns.xsql.rev140626.AbstractXSQLModule.getInstance(AbstractXSQLModule.java:95)<br/> at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source)<br/> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)<br/> at java.lang.reflect.Method.invoke(Method.java:606)<br/> at org.opendaylight.controller.config.manager.impl.dependencyresolver.DependencyResolverManager$1.handleInvocation(DependencyResolverManager.java:152)<br/> at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:70)<br/> at com.sun.proxy.$Proxy70.getInstance(Unknown Source)<br/> at org.opendaylight.controller.config.manager.impl.ConfigTransactionControllerImpl.secondPhaseCommit(ConfigTransactionControllerImpl.java:396)<br/> ... 36 more<br/> Caused by: ReadFailedException</p> {message=Error checking exists for path /(http://netconfcentral.org/ns/XSQL?revision=2014-06-26)XSQL, errorList=[RpcError [message=Error checking exists for path /(http://netconfcentral.org/ns/XSQL?revision=2014-06-26)XSQL, severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=org.opendaylight.controller.cluster.datastore.exceptions.TimeoutException: Sending message class org.opendaylight.controller.protobuff.messages.transaction.ShardTransactionMessages$CreateTransaction to actor ActorSelection[Anchor(akka://opendaylight-cluster-data/), Path(/user/shardmanager-operational/member-1-shard-default-operational)] failed]]} <p> at org.opendaylight.controller.cluster.datastore.TransactionProxy$NoOpTransactionContext.dataExists(TransactionProxy.java:846)<br/> at org.opendaylight.controller.cluster.datastore.TransactionProxy.exists(TransactionProxy.java:258)<br/> at org.opendaylight.controller.md.sal.dom.broker.impl.DOMForwardedReadWriteTransaction.exists(DOMForwardedReadWriteTransaction.java:54)<br/> at org.opendaylight.controller.md.sal.binding.impl.AbstractReadWriteTransaction.ensureParentsByMerge(AbstractReadWriteTransaction.java:52)<br/> ... 49 more<br/> Caused by: org.opendaylight.controller.cluster.datastore.exceptions.TimeoutException: Sending message class org.opendaylight.controller.protobuff.messages.transaction.ShardTransactionMessages$CreateTransaction to actor ActorSelection<span class="error">&#91;Anchor(akka://opendaylight-cluster-data/), Path(/user/shardmanager-operational/member-1-shard-default-operational)&#93;</span> failed<br/> at org.opendaylight.controller.cluster.datastore.utils.ActorContext.executeRemoteOperation(ActorContext.java:189)<br/> at org.opendaylight.controller.cluster.datastore.utils.ActorContext.executeShardOperation(ActorContext.java:243)<br/> at org.opendaylight.controller.cluster.datastore.TransactionProxy.createTransactionIfMissing(TransactionProxy.java:382)<br/> at org.opendaylight.controller.cluster.datastore.TransactionProxy.exists(TransactionProxy.java:256)<br/> ... 51 more<br/> Caused by: java.lang.IllegalStateException: Could not find leader so transaction cannot be created<br/> at org.opendaylight.controller.cluster.datastore.Shard.onReceiveCommand(Shard.java:221)<br/> at akka.persistence.UntypedPersistentActor.onReceive(Eventsourced.scala:430)<br/> at org.opendaylight.controller.cluster.common.actor.MeteringBehavior.apply(MeteringBehavior.java:80)<br/> at akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:534)<br/> at akka.persistence.Recovery$State$class.process(Recovery.scala:30)<br/> at akka.persistence.ProcessorImpl$$anon$2.process(Processor.scala:103)<br/> at akka.persistence.ProcessorImpl$$anon$2.aroundReceive(Processor.scala:114)<br/> at akka.persistence.Recovery$class.aroundReceive(Recovery.scala:256)<br/> at akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(Eventsourced.scala:428)<br/> at akka.persistence.Eventsourced$$anon$2.doAroundReceive(Eventsourced.scala:82)<br/> at akka.persistence.Eventsourced$$anon$2.aroundReceive(Eventsourced.scala:78)<br/> at akka.persistence.Eventsourced$class.aroundReceive(Eventsourced.scala:369)<br/> at akka.persistence.UntypedPersistentActor.aroundReceive(Eventsourced.scala:428)<br/> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)<br/> at akka.actor.ActorCell.invoke(ActorCell.scala:487)<br/> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)<br/> at akka.dispatch.Mailbox.run(Mailbox.scala:220)<br/> at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)<br/> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)<br/> at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)<br/> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)<br/> at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)</p> <p>Operating System: All<br/> Platform: All</p> CONTROLLER-858

Clustering: Update Cluster datastore to provide notification/API to check its availability for clients

Improvement Resolved Won't Do Unassigned Wojciech Dec Thu, 18 Sep 2014 13:08:13 +0000 Thu, 19 Oct 2017 21:27:05 +0000 Wed, 5 Jul 2017 13:09:08 +0000 Helium clustering 0 10 <p>This issues happens because Xsql bundle tries to use datastore, While clustering shards are getting ready and they are not available for use. </p> <p>This is not clustering issue, should be fixed at each application level.</p> <p>Ideally, each application, if performing some datastore related operations, should check, if datastore is available or perform that operation in callback manner, once its available.</p> <p>The fix implementation should be done "underneath" the Data Broker interface so ALL the applications should not be concerned with this race condition issue. If the fix will be done in XSQL, the race condition will just move to the next loading application and thus, the bug won't be fixed, just shift to another application.</p> <p>The issue that we see can happen with any client application that is trying to use cluster data store, we should provide some notification of status of datastore that applications can listen to.</p> <p>Clustered data store should not provide its instance to clients when is not ready, since client gets "not fully working" implementation. Better solution is to wait in create instance till clustering is initialized and then return.</p> <p>Consensus</p> <p>1. Datastore should notify when it is not available<br/> 2. An external listener should shutdown the controller<br/> 3. Some external program should restart the controller (eg monit)</p> <p>There are a few things that were done to help with this,</p> <ul class="alternate" type="square"> <li>The datastore remains unavailable till the leaders for all shards are available. We wait for 90 seconds and then give up</li> <li>We retry transactions</li> <li>We throw an DataStoreUnavailableException when elections are going on</li> </ul> <p>There is no clear use case for a notification to be issued when datastore is unavailable.</p> <p>We may want to create a service which given a yanginstanceid can inform us of the availability of the underlying shard.</p> Blocks CONTROLLER-1483 Development External issue ID 1955 External issue URL Issue Type ODL SR Target Milestone Priority Rank 0|i02mzr: