[CONTROLLER-1552] odl-mdsal-benchmark does not wait for Leaders to appear

Duplicate — Tue, 25 Jul 2023 08:24:13 +0000

This happens especially in cluster testing when odl-mdsal-benchmark is listed in featuresBoot.
Feature starts creating its initial data, but "default" shards may not yet be ready. This leads to "server is unhealthy". Note that this happens even before users starts any test.

To fix, either wait in activator, or create initial structure (each time) when user asks for a test.

Parts of karaf.log:

2016-09-23 08:48:39,386 | INFO | config-pusher | ConfigPusherImpl | 136 - org.opend
aylight.controller.config-persister-impl - 0.5.0.Boron | Pushing configuration snapshot dsbenchmark-imp
l-default-config.xml(odl-mdsal-benchmark,odl-mdsal-benchmark)
2016-09-23 08:48:39,390 | INFO | er [_value=8185] | WebSocketServer | 265 - org.opend
aylight.netconf.sal-rest-connector - 1.4.0.Boron | Web socket server started at port 8185.
2016-09-23 08:48:39,504 | INFO | config-pusher | DsbenchmarkListenerProvider | 269 - org.opend
aylight.controller.dsbenchmark - 1.2.0.Boron | DsbenchmarkListenerProvider created
2016-09-23 08:48:43,524 | INFO | lt-dispatcher-17 | EntityOwnershipShard | 193 - org.opend
aylight.controller.sal-akka-raft - 1.4.0.Boron | member-1-shard-entity-ownership-operational (Follower)
:- Switching from behavior Follower to Candidate
2016-09-23 08:48:43,525 | INFO | ult-dispatcher-2 | RoleChangeNotifier | 192 - org.opend
aylight.controller.sal-clustering-commons - 1.4.0.Boron | RoleChangeNotifier for member-1-shard-entity-
ownership-operational , received role change from Follower to Candidate
2016-09-23 08:48:43,525 | INFO | ult-dispatcher-2 | ShardManager | 197 - org.opend
aylight.controller.sal-distributed-datastore - 1.4.0.Boron | shard-manager-operational: Received role c
hanged for member-1-shard-entity-ownership-operational from Follower to Candidate
2016-09-23 08:48:49,463 | WARN | saction-71-73'}} | DeadlockMonitor | 132 - org.opend
aylight.controller.config-manager - 0.5.0.Boron | ModuleIdentifier

{factoryName='dsbenchmark-impl', inst anceName='dsbenchmark-default'} did not finish after 9983 ms
2016-09-23 08:48:54,463 | WARN | saction-71-73'}} | DeadlockMonitor | 132 - org.opend
aylight.controller.config-manager - 0.5.0.Boron | ModuleIdentifier{factoryName='dsbenchmark-impl', instanceName='dsbenchmark-default'}

did not finish after 14983 ms
2016-09-23 08:48:59,464 | WARN | saction-71-73'}} | DeadlockMonitor | 132 - org.opend
aylight.controller.config-manager - 0.5.0.Boron | ModuleIdentifier

{factoryName='dsbenchmark-impl', inst anceName='dsbenchmark-default'} did not finish after 19984 ms
2016-09-23 08:48:59,667 | WARN | lt-dispatcher-18 | ConcurrentDOMDataBroker | 197 - org.opend
aylight.controller.sal-distributed-datastore - 1.4.0.Boron | Tx: DOM-0 Error during phase CAN_COMMIT, s
tarting Abort
org.opendaylight.controller.cluster.datastore.exceptions.NoShardLeaderException: Shard member-1-shard-d
efault-operational currently has no leader. Try again later.
at org.opendaylight.controller.cluster.datastore.shardmanager.ShardManager.createNoShardLeaderE
xception(ShardManager.java:723)[197:org.opendaylight.controller.sal-distributed-datastore:1.4.0.Boron]
at org.opendaylight.controller.cluster.datastore.shardmanager.ShardManager.onShardNotInitialize
dTimeout(ShardManager.java:537)[197:org.opendaylight.controller.sal-distributed-datastore:1.4.0.Boron]
at org.opendaylight.controller.cluster.datastore.shardmanager.ShardManager.handleCommand(ShardM
anager.java:216)[197:org.opendaylight.controller.sal-distributed-datastore:1.4.0.Boron]
at org.opendaylight.controller.cluster.common.actor.AbstractUntypedPersistentActor.onReceiveCom
mand(AbstractUntypedPersistentActor.java:29)[192:org.opendaylight.controller.sal-clustering-commons:1.4
.0.Boron]
at akka.persistence.UntypedPersistentActor.onReceive(PersistentActor.scala:170)[186:com.typesaf
e.akka.persistence:2.4.7]
at org.opendaylight.controller.cluster.common.actor.MeteringBehavior.apply(MeteringBehavior.jav
a:97)[192:org.opendaylight.controller.sal-clustering-commons:1.4.0.Boron]
at akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:544)[180:com.typesafe.akk
a.actor:2.4.7]
at akka.actor.Actor$class.aroundReceive(Actor.scala:484)[180:com.typesafe.akka.actor:2.4.7]
at akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(PersistentActor.scala:168)[186:com.typesafe.akka.persistence:2.4.7]
at akka.persistence.Eventsourced$$anon$1.stateReceive(Eventsourced.scala:633)[186:com.typesafe.akka.persistence:2.4.7]
at akka.persistence.Eventsourced$class.aroundReceive(Eventsourced.scala:179)[186:com.typesafe.akka.persistence:2.4.7]
at akka.persistence.UntypedPersistentActor.aroundReceive(PersistentActor.scala:168)[186:com.typesafe.akka.persistence:2.4.7]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)[180:com.typesafe.akka.actor:2.4.7]
at akka.actor.ActorCell.invoke(ActorCell.scala:495)[180:com.typesafe.akka.actor:2.4.7]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)[180:com.typesafe.akka.actor:2.4.7]
at akka.dispatch.Mailbox.run(Mailbox.scala:224)[180:com.typesafe.akka.actor:2.4.7]
at akka.dispatch.Mailbox.exec(Mailbox.scala:234)[180:com.typesafe.akka.actor:2.4.7]
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)[176:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)[176:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)[176:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)[176:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
2016-09-23 08:48:59,678 | ERROR | config-pusher | ConfigTransactionControllerImpl | 132 - org.opend
aylight.controller.config-manager - 0.5.0.Boron | Commit failed on ModuleIdentifier{factoryName='dsbenchmark-impl', instanceName='dsbenchmark-default'}

in transaction TransactionIdentifier

{name='ConfigTrans action-71-73'}
java.lang.IllegalStateException: TransactionCommitFailedException{message=canCommit encountered an unex pected failure, errorList=[RpcError [message=canCommit encountered an unexpected failure, severity=ERRO R, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=org.opendaylight. controller.md.sal.common.api.data.DataStoreUnavailableException: Shard member-1-shard-default-operation al currently has no leader. Try again later.]]}
at org.opendaylight.dsbenchmark.DsbenchmarkProvider.setTestOperData(DsbenchmarkProvider.java:17
5)
at org.opendaylight.dsbenchmark.DsbenchmarkProvider.onSessionInitiated(DsbenchmarkProvider.java
:81)
at org.opendaylight.controller.md.sal.binding.util.BindingContextUtils.createProviderContextAnd
Initialize(BindingContextUtils.java:56)
at org.opendaylight.controller.sal.binding.impl.RootBindingAwareBroker.registerProvider(RootBin
dingAwareBroker.java:187)
at org.opendaylight.yang.gen.v1.urn.opendaylight.params.xml.ns.yang.dsbenchmark.impl.rev141210.
DsbenchmarkImplModule.createInstance(DsbenchmarkImplModule.java:30)
at org.opendaylight.controller.config.spi.AbstractModule.getInstance(AbstractModule.java:87)
at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0
_91]
at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_91]
at org.opendaylight.controller.config.manager.impl.dependencyresolver.DependencyResolverManager
$ModuleInvocationHandler.handleInvocation(DependencyResolverManager.java:149)
at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87
)
at com.sun.proxy.$Proxy34.getInstance(Unknown Source)
at org.opendaylight.controller.config.manager.impl.ConfigTransactionControllerImpl.secondPhaseC
ommit(ConfigTransactionControllerImpl.java:437)
at org.opendaylight.controller.config.manager.impl.ConfigRegistryImpl.secondPhaseCommit(ConfigR
egistryImpl.java:346)
at org.opendaylight.controller.config.manager.impl.ConfigRegistryImpl.commitConfigSafe(ConfigRe
gistryImpl.java:285)
at org.opendaylight.controller.config.manager.impl.ConfigRegistryImpl.commitConfig(ConfigRegist
ryImpl.java:250)

...

2016-09-23 08:48:59,706 | ERROR | config-pusher | ConfigRegistryImpl | 132 - org.opendaylight.controller.config-manager - 0.5.0.Boron | Configuration Transaction failed on 2PC, server is unhealthy
java.lang.IllegalStateException: Error - getInstance() failed for ModuleIdentifier{factoryName='dsbenchmark-impl', instanceName='dsbenchmark-default'} in transaction TransactionIdentifier{name='ConfigTransaction-71-73'}

at org.opendaylight.controller.config.manager.impl.ConfigTransactionControllerImpl.secondPhaseCommit(ConfigTransactionControllerImpl.java:444)
at org.opendaylight.controller.config.manager.impl.ConfigRegistryImpl.secondPhaseCommit(ConfigRegistryImpl.java:346)
at org.opendaylight.controller.config.manager.impl.ConfigRegistryImpl.commitConfigSafe(ConfigRegistryImpl.java:285)
at org.opendaylight.controller.config.manager.impl.ConfigRegistryImpl.commitConfig(ConfigRegistryImpl.java:250)

OpenDaylight JIRA

[CONTROLLER-1552] odl-mdsal-benchmark does not wait for Leaders to appear