[OVSDB-434] br-int not created after failing over one controller in 3 node cluster

Relates — Thu, 2 May 2019 00:01:50 +0000

3 node ODL cluster with these features:
odl-ovsdb-openstack
odl-mdsal-clustering
odl-jolokia

2 OVS nodes where they are being set to connect to each of the
3 controllers. The ovs nodes were disconnected and connected
many (~20) times in a row and it was verified that br-int was
being created.

The controller reporting leader for default config shard was
then stopped (logout command on karaf shell) and started. Once
it came back, connecting to the controllers would see that the
ovsdb manager was set, but there would be no br-int created.

this exception was coming in a controller (not the restarted controller)
when the ovs nodes were set to connect:

{targetType=interface org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.topology.Node, path=[org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.NetworkTopology, org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.Topology[key=TopologyKey [_topologyId=Uri [_value=ovsdb:1]]], org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.topology.Node[key=NodeKey [_nodeId=Uri [_value=ovsdb://uuid/f3354257-9201-4e55-bf6e-98320d6c5f94]]]]}

ReadFailedException{message=Error executeRead ReadData for path /(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)network-topology/topology/topology[

{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)topology-id=ovsdb:1}

]/node/node[

{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)node-id=ovsdb://uuid/f3354257-9201-4e55-bf6e-98320d6c5f94}

], errorList=[RpcError [message=Error executeRead ReadData for path /(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)network-topology/topology/topology[

{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)topology-id=ovsdb:1}

]/node/node[

{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)node-id=ovsdb://uuid/f3354257-9201-4e55-bf6e-98320d6c5f94}

], severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=org.opendaylight.controller.md.sal.common.api.data.DataStoreUnavailableException: Shard member-2-shard-topology-operational currently has no leader. Try again later.]]}
at org.opendaylight.controller.cluster.datastore.NoOpTransactionContext.executeRead(NoOpTransactionContext.java:71)[165:org.opendaylight.controller.sal-distributed-datastore:1.3.2.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.TransactionProxy$1.invoke(TransactionProxy.java:92)[165:org.opendaylight.controller.sal-distributed-datastore:1.3.2.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.TransactionContextWrapper.executePriorTransactionOperations(TransactionContextWrapper.java:132)[165:org.opendaylight.controller.sal-distributed-datastore:1.3.2.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.AbstractTransactionContextFactory.onFindPrimaryShardFailure(AbstractTransactionContextFactory.java:97)[165:org.opendaylight.controller.sal-distributed-datastore:1.3.2.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.AbstractTransactionContextFactory.access$100(AbstractTransactionContextFactory.java:35)[165:org.opendaylight.controller.sal-distributed-datastore:1.3.2.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.AbstractTransactionContextFactory$1.onComplete(AbstractTransactionContextFactory.java:123)[165:org.opendaylight.controller.sal-distributed-datastore:1.3.2.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.AbstractTransactionContextFactory$1.onComplete(AbstractTransactionContextFactory.java:117)[165:org.opendaylight.controller.sal-distributed-datastore:1.3.2.SNAPSHOT]
at akka.dispatch.OnComplete.internal(Future.scala:247)[150:com.typesafe.akka.actor:2.3.14]
at akka.dispatch.OnComplete.internal(Future.scala:245)[150:com.typesafe.akka.actor:2.3.14]
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:175)[150:com.typesafe.akka.actor:2.3.14]
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:172)[150:com.typesafe.akka.actor:2.3.14]
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)[147:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]
at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)[150:com.typesafe.akka.actor:2.3.14]
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)[150:com.typesafe.akka.actor:2.3.14]
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)[150:com.typesafe.akka.actor:2.3.14]
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)[150:com.typesafe.akka.actor:2.3.14]
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)[147:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]
at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)[150:com.typesafe.akka.actor:2.3.14]
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)[150:com.typesafe.akka.actor:2.3.14]
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)[150:com.typesafe.akka.actor:2.3.14]
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)[147:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)[147:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)[147:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)[147:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]
Caused by: org.opendaylight.controller.md.sal.common.api.data.DataStoreUnavailableException: Shard member-2-shard-topology-operational currently has no leader. Try again later.
at org.opendaylight.controller.cluster.datastore.NoOpTransactionContext.executeRead(NoOpTransactionContext.java:67)[165:org.opendaylight.controller.sal-distributed-datastore:1.3.2.SNAPSHOT]
... 23 more
Caused by: org.opendaylight.controller.cluster.datastore.exceptions.NoShardLeaderException: Shard member-2-shard-topology-operational currently has no leader. Try again later.
at org.opendaylight.controller.cluster.datastore.ShardManager.createNoShardLeaderException(ShardManager.java:744)[165:org.opendaylight.controller.sal-distributed-datastore:1.3.2.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.ShardManager.onShardNotInitializedTimeout(ShardManager.java:551)[165:org.opendaylight.controller.sal-distributed-datastore:1.3.2.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.ShardManager.handleCommand(ShardManager.java:222)[165:org.opendaylight.controller.sal-distributed-datastore:1.3.2.SNAPSHOT]
at org.opendaylight.controller.cluster.common.actor.AbstractUntypedPersistentActor.onReceiveCommand(AbstractUntypedPersistentActor.java:36)[161:org.opendaylight.controller.sal-clustering-commons:1.3.2.SNAPSHOT]
at akka.persistence.UntypedPersistentActor.onReceive(Eventsourced.scala:430)[155:com.typesafe.akka.persistence.experimental:2.3.14]
at org.opendaylight.controller.cluster.common.actor.MeteringBehavior.apply(MeteringBehavior.java:97)[161:org.opendaylight.controller.sal-clustering-commons:1.3.2.SNAPSHOT]
at akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:534)[150:com.typesafe.akka.actor:2.3.14]
at akka.persistence.Recovery$State$class.process(Recovery.scala:30)[155:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.persistence.ProcessorImpl$$anon$2.process(Processor.scala:103)[155:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.persistence.ProcessorImpl$$anon$2.aroundReceive(Processor.scala:114)[155:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.persistence.Recovery$class.aroundReceive(Recovery.scala:265)[155:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(Eventsourced.scala:428)[155:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.persistence.Eventsourced$$anon$2.doAroundReceive(Eventsourced.scala:82)[155:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.persistence.Eventsourced$$anon$2.aroundReceive(Eventsourced.scala:78)[155:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.persistence.Eventsourced$class.aroundReceive(Eventsourced.scala:369)[155:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.persistence.UntypedPersistentActor.aroundReceive(Eventsourced.scala:428)[155:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)[150:com.typesafe.akka.actor:2.3.14]
at akka.actor.ActorCell.invoke(ActorCell.scala:487)[150:com.typesafe.akka.actor:2.3.14]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)[150:com.typesafe.akka.actor:2.3.14]
at akka.dispatch.Mailbox.run(Mailbox.scala:220)[150:com.typesafe.akka.actor:2.3.14]
... 5 more
2016-04-13 21:19:58,665 | WARN | n-invoker-impl-0 | SouthboundUtil | 251 - org.opendaylight.ovsdb.southbound-impl - 1.2.3.SNAPSHOT | Read Operational/DS for Node failed! KeyedInstanceIdentifier

ReadFailedException{message=Error executeRead ReadData for path /(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)network-topology/topology/topology[

{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)topology-id=ovsdb:1}

]/node/node[

{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)node-id=ovsdb://uuid/f3354257-9201-4e55-bf6e-98320d6c5f94}

], errorList=[RpcError [message=Error executeRead ReadData for path /(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)network-topology/topology/topology[

{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)topology-id=ovsdb:1}

]/node/node[

{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)node-id=ovsdb://uuid/f3354257-9201-4e55-bf6e-98320d6c5f94}

], severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=org.opendaylight.controller.md.sal.common.api.data.DataStoreUnavailableException: Shard member-2-shard-topology-operational currently has no leader. Try again later.]]}
at org.opendaylight.controller.cluster.datastore.NoOpTransactionContext.executeRead(NoOpTransactionContext.java:71)[165:org.opendaylight.controller.sal-distributed-datastore:1.3.2.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.TransactionProxy$1.invoke(TransactionProxy.java:92)[165:org.opendaylight.controller.sal-distributed-datastore:1.3.2.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.TransactionContextWrapper.maybeExecuteTransactionOperation(TransactionContextWrapper.java:92)[165:org.opendaylight.controller.sal-distributed-datastore:1.3.2.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.TransactionProxy.executeRead(TransactionProxy.java:89)[165:org.opendaylight.controller.sal-distributed-datastore:1.3.2.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.TransactionProxy.singleShardRead(TransactionProxy.java:114)[165:org.opendaylight.controller.sal-distributed-datastore:1.3.2.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.TransactionProxy.read(TransactionProxy.java:108)[165:org.opendaylight.controller.sal-distributed-datastore:1.3.2.SNAPSHOT]
at org.opendaylight.controller.cluster.databroker.DOMBrokerReadWriteTransaction.read(DOMBrokerReadWriteTransaction.java:37)[165:org.opendaylight.controller.sal-distributed-datastore:1.3.2.SNAPSHOT]
at org.opendaylight.controller.md.sal.binding.impl.AbstractForwardedTransaction.doRead(AbstractForwardedTransaction.java:63)[140:org.opendaylight.controller.sal-binding-broker-impl:1.3.2.SNAPSHOT]
at org.opendaylight.controller.md.sal.binding.impl.BindingDOMReadWriteTransactionAdapter.read(BindingDOMReadWriteTransactionAdapter.java:31)[140:org.opendaylight.controller.sal-binding-broker-impl:1.3.2.SNAPSHOT]
at org.opendaylight.ovsdb.southbound.SouthboundUtil.readNode(SouthboundUtil.java:112)[251:org.opendaylight.ovsdb.southbound-impl:1.2.3.SNAPSHOT]
at org.opendaylight.ovsdb.southbound.transactions.md.OvsdbQosRemovedCommand.execute(OvsdbQosRemovedCommand.java:54)[251:org.opendaylight.ovsdb.southbound-impl:1.2.3.SNAPSHOT]
at org.opendaylight.ovsdb.southbound.transactions.md.OvsdbOperationalCommandAggregator.execute(OvsdbOperationalCommandAggregator.java:46)[251:org.opendaylight.ovsdb.southbound-impl:1.2.3.SNAPSHOT]
at org.opendaylight.ovsdb.southbound.transactions.md.TransactionInvokerImpl.run(TransactionInvokerImpl.java:88)[251:org.opendaylight.ovsdb.southbound-impl:1.2.3.SNAPSHOT]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)[:1.8.0_77]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_77]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_77]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_77]
at java.lang.Thread.run(Thread.java:745)[:1.8.0_77]
Caused by: org.opendaylight.controller.md.sal.common.api.data.DataStoreUnavailableException: Shard member-2-shard-topology-operational currently has no leader. Try again later.
at org.opendaylight.controller.cluster.datastore.NoOpTransactionContext.executeRead(NoOpTransactionContext.java:67)[165:org.opendaylight.controller.sal-distributed-datastore:1.3.2.SNAPSHOT]
... 17 more
Caused by: org.opendaylight.controller.cluster.datastore.exceptions.NoShardLeaderException: Shard member-2-shard-topology-operational currently has no leader. Try again later.
at org.opendaylight.controller.cluster.datastore.ShardManager.createNoShardLeaderException(ShardManager.java:744)[165:org.opendaylight.controller.sal-distributed-datastore:1.3.2.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.ShardManager.onShardNotInitializedTimeout(ShardManager.java:551)[165:org.opendaylight.controller.sal-distributed-datastore:1.3.2.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.ShardManager.handleCommand(ShardManager.java:222)[165:org.opendaylight.controller.sal-distributed-datastore:1.3.2.SNAPSHOT]
at org.opendaylight.controller.cluster.common.actor.AbstractUntypedPersistentActor.onReceiveCommand(AbstractUntypedPersistentActor.java:36)[161:org.opendaylight.controller.sal-clustering-commons:1.3.2.SNAPSHOT]
at akka.persistence.UntypedPersistentActor.onReceive(Eventsourced.scala:430)[155:com.typesafe.akka.persistence.experimental:2.3.14]
at org.opendaylight.controller.cluster.common.actor.MeteringBehavior.apply(MeteringBehavior.java:97)[161:org.opendaylight.controller.sal-clustering-commons:1.3.2.SNAPSHOT]
at akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:534)[150:com.typesafe.akka.actor:2.3.14]
at akka.persistence.Recovery$State$class.process(Recovery.scala:30)[155:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.persistence.ProcessorImpl$$anon$2.process(Processor.scala:103)[155:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.persistence.ProcessorImpl$$anon$2.aroundReceive(Processor.scala:114)[155:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.persistence.Recovery$class.aroundReceive(Recovery.scala:265)[155:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(Eventsourced.scala:428)[155:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.persistence.Eventsourced$$anon$2.doAroundReceive(Eventsourced.scala:82)[155:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.persistence.Eventsourced$$anon$2.aroundReceive(Eventsourced.scala:78)[155:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.persistence.Eventsourced$class.aroundReceive(Eventsourced.scala:369)[155:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.persistence.UntypedPersistentActor.aroundReceive(Eventsourced.scala:428)[155:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)[150:com.typesafe.akka.actor:2.3.14]
at akka.actor.ActorCell.invoke(ActorCell.scala:487)[150:com.typesafe.akka.actor:2.3.14]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)[150:com.typesafe.akka.actor:2.3.14]
at akka.dispatch.Mailbox.run(Mailbox.scala:220)[150:com.typesafe.akka.actor:2.3.14]
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)[150:com.typesafe.akka.actor:2.3.14]
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)[147:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)[147:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)[147:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)[147:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]

OpenDaylight JIRA

[OVSDB-434] br-int not created after failing over one controller in 3 node cluster