https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/OVSDB-292 ovsdb <p>Southbound plugin throws DataStoreUnavailableException while read transaction when shard leader node is killed. This only happens when shard leader node is killed rather then shutting it down gracefully.</p> <p>Operating System: All<br/> Platform: All</p> OVSDB-292

Clustering: DataStoreUnavailableException when shard leader is killed

Bug Resolved Cannot Reproduce ranjithkumar_t Anil Vishnoi Mon, 8 Feb 2016 19:43:38 +0000 Thu, 2 Feb 2017 20:06:21 +0000 Thu, 2 Feb 2017 20:06:21 +0000 unspecified Clustering 0 5 <p>Lowering the priority, i am not able to recreate this issue.</p> <p>Hi Anil,</p> <p>For recreating the bug the following steps are followed</p> <p>1. Cloned OVSDB project and switched to stable/beryllium branch. <br/> 2. The above step followed for all 3 controllers and copied the cluster configuration file.<br/> 3. Installed required features for clustering and southbound plugin.<br/> 4. HA happens and initial network topology is initiated in operational side</p> <p>The following virtual machines are used to create OVSDB 3 node clustering 10.0.0.1, 10.0.0.2, 10.0.0.3</p> <p>Observation : 1 ( Down Leader node 10.0.0.1) – One Node Failure</p> <p>1. Leader node is 10.0.0.1 the remaining two nodes are followers.<br/> 2. In 3 node cluster, If leader node goes down HA happens for another two nodes and I am not getting any exception like “DataStoreUnavailableException”. <br/> 3. There is no change in operational side.</p> <p>Observation : 2 ( Down Leader node 10.0.0.2) – Two Node Failure</p> <p>1. Now the Leader node is 10.0.0.2 and the remaining one node is follower.<br/> 2. In 2 node cluster, If leader node goes down, I am getting the exception “DataStoreUnavailableException” in operational.</p> <p><a href="http://10.0.0.3:8181/restconf/operational/network-topology:network-topology/" class="external-link" target="_blank" rel="nofollow noopener">http://10.0.0.3:8181/restconf/operational/network-topology:network-topology/</a></p> <p>org.opendaylight.controller.md.sal.common.api.data.DataStoreUnavailableException: Shard member-3-shard-topology-operational currently has no leader. Try again later. at org.opendaylight.controller.cluster.datastore.NoOpTransactionContext.executeRead(NoOpTransactionContext.java:67) ... 23 more Caused by: org.opendaylight.controller.cluster.datastore.exceptions.NoShardLeaderException: Shard member-3-shard-topology-operational currently has no leader. Try again later. at org.opendaylight.controller.cluster.datastore.ShardManager.createNoShardLeaderException(ShardManager.java:744) at org.opendaylight.controller.cluster.datastore.ShardManager.onShardNotInitializedTimeout(ShardManager.java:551)</p> <p>I am not sure the singleton cluster is supported in 3 node OVSDB clustering mode.</p> <p>I have tried to reproduce this with Beryllium SR2 candidate distro <span class="error">&#91;0&#93;</span>. </p> <ul class="alternate" type="square"> <li>3 node cluster</li> <li>features: odl-jolokia, odl-ovsdb-southbound-impl-rest</li> <li>mininet topo,1,3 connected and verified in operational/network-topology:network-topology/topology/ovsdb:1/</li> <li>kill -9 on operational topology shard leader</li> </ul> <p>I tried this several times, and never saw the DataStoreUnavailableException. I did see many of<br/> these IllegalStateException.</p> <p>2016-05-03 17:43:56,597 | INFO | ult-dispatcher-5 | ShardManager | 185 - org.opendaylight.controller.sal-d<br/> istributed-datastore - 1.3.2.Beryllium-SR2 | shard-manager-operational Received follower initial sync status for member-1-shard<br/> -toaster-operational status sync done true<br/> 2016-05-03 17:43:56,600 | ERROR | ult-dispatcher-2 | SimpleShardDataTreeCohort | 185 - org.opendaylight.controller.sal-distributed-datastore - 1.3.2.Beryllium-SR2 | Transaction failed to commit<br/> java.lang.IllegalStateException: Store tree org.opendaylight.yangtools.yang.data.api.schema.tree.spi.MaterializedContainerNode@7518b3f0 and candidate base org.opendaylight.yangtools.yang.data.api.schema.tree.spi.MaterializedContainerNode@dc0cde7 differ.<br/> at org.opendaylight.yangtools.yang.data.impl.schema.tree.InMemoryDataTree.commit(InMemoryDataTree.java:119)<span class="error">&#91;100:org.opendaylight.yangtools.yang-data-impl:0.8.2.Beryllium-SR2&#93;</span><br/> at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.commit(SimpleShardDataTreeCohort.java:106)<span class="error">&#91;185:org.opendaylight.controller.sal-distributed-datastore:1.3.2.Beryllium-SR2&#93;</span><br/> at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator$CohortEntry.commit(ShardCommitCoordinator.java:670)<span class="error">&#91;185:org.opendaylight.controller.sal-distributed-datastore:1.3.2.Beryllium-SR2&#93;</span><br/> at org.opendaylight.controller.cluster.datastore.Shard.finishCommit(Shard.java:351)<span class="error">&#91;185:org.opendaylight.controller.sal-distributed-datastore:1.3.2.Beryllium-SR2&#93;</span><br/> at org.opendaylight.controller.cluster.datastore.Shard.finishCommit(Shard.java:415)<span class="error">&#91;185:org.opendaylight.controller.sal-distributed-datastore:1.3.2.Beryllium-SR2&#93;</span><br/> at org.opendaylight.controller.cluster.datastore.Shard.applyState(Shard.java:663)<span class="error">&#91;185:org.opendaylight.controller.sal-distributed-datastore:1.3.2.Beryllium-SR2&#93;</span><br/> at org.opendaylight.controller.cluster.raft.RaftActor.handleCommand(RaftActor.java:224)<span class="error">&#91;182:org.opendaylight.controller.sal-akka-raft:1.3.2.Beryllium-SR2&#93;</span><br/> at org.opendaylight.controller.cluster.common.actor.AbstractUntypedPersistentActor.onReceiveCommand(AbstractUntypedPersistentActor.java:36)<span class="error">&#91;181:org.opendaylight.controller.sal-clustering-commons:1.3.2.Beryllium-SR2&#93;</span><br/> at org.opendaylight.controller.cluster.datastore.Shard.onReceiveCommand(Shard.java:275)<span class="error">&#91;185:org.opendaylight.controller.sal-distributed-datastore:1.3.2.Beryllium-SR2&#93;</span><br/> at akka.persistence.UntypedPersistentActor.onReceive(Eventsourced.scala:430)<span class="error">&#91;175:com.typesafe.akka.persistence.experimental:2.3.14&#93;</span><br/> at org.opendaylight.controller.cluster.common.actor.MeteringBehavior.apply(MeteringBehavior.java:97)<span class="error">&#91;181:org.opendaylight.controller.sal-clustering-commons:1.3.2.Beryllium-SR2&#93;</span><br/> at akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:534)<span class="error">&#91;170:com.typesafe.akka.actor:2.3.14&#93;</span><br/> at akka.persistence.Recovery$State$class.process(Recovery.scala:30)<span class="error">&#91;175:com.typesafe.akka.persistence.experimental:2.3.14&#93;</span><br/> at akka.persistence.ProcessorImpl$$anon$2.process(Processor.scala:103)<span class="error">&#91;175:com.typesafe.akka.persistence.experimental:2.3.14&#93;</span><br/> at akka.persistence.ProcessorImpl$$anon$2.aroundReceive(Processor.scala:114)<span class="error">&#91;175:com.typesafe.akka.persistence.experimental:2.3.14&#93;</span><br/> at akka.persistence.Recovery$class.aroundReceive(Recovery.scala:265)<span class="error">&#91;175:com.typesafe.akka.persistence.experimental:2.3.14&#93;</span><br/> at akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(Eventsourced.scala:428)<span class="error">&#91;175:com.typesafe.akka.persistence.experimental:2.3.14&#93;</span><br/> at akka.persistence.Eventsourced$$anon$2.doAroundReceive(Eventsourced.scala:82)<span class="error">&#91;175:com.typesafe.akka.persistence.experimental:2.3.14&#93;</span><br/> at akka.persistence.Eventsourced$$anon$2.aroundReceive(Eventsourced.scala:78)<span class="error">&#91;175:com.typesafe.akka.persistence.experimental:2.3.14&#93;</span><br/> at akka.persistence.Eventsourced$class.aroundReceive(Eventsourced.scala:369)<span class="error">&#91;175:com.typesafe.akka.persistence.experimental:2.3.14&#93;</span><br/> at akka.persistence.UntypedPersistentActor.aroundReceive(Eventsourced.scala:428)<span class="error">&#91;175:com.typesafe.akka.persistence.experimental:2.3.14&#93;</span><br/> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)<span class="error">&#91;170:com.typesafe.akka.actor:2.3.14&#93;</span><br/> at akka.actor.ActorCell.invoke(ActorCell.scala:487)<span class="error">&#91;170:com.typesafe.akka.actor:2.3.14&#93;</span><br/> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)<span class="error">&#91;170:com.typesafe.akka.actor:2.3.14&#93;</span><br/> at akka.dispatch.Mailbox.run(Mailbox.scala:220)<span class="error">&#91;170:com.typesafe.akka.actor:2.3.14&#93;</span><br/> at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)<span class="error">&#91;170:com.typesafe.akka.actor:2.3.14&#93;</span><br/> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)<span class="error">&#91;167:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c&#93;</span><br/> at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)<span class="error">&#91;167:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c&#93;</span><br/> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)<span class="error">&#91;167:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c&#93;</span><br/> at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)<span class="error">&#91;167:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c&#93;</span></p> <p><span class="error">&#91;0&#93;</span> <a href="https://wiki.opendaylight.org/view/Simultaneous_Release:Beryllium_Release_Plan#Beryllium_SR2_Download" class="external-link" target="_blank" rel="nofollow noopener">https://wiki.opendaylight.org/view/Simultaneous_Release:Beryllium_Release_Plan#Beryllium_SR2_Download</a></p> <p>Looks like this issue is not recreatable. Please re-open the issue if you are able to recreate it.</p> Development External issue ID 5274 External issue URL Issue Type Priority Rank 0|i021tz: