[CONTROLLER-1552] odl-mdsal-benchmark does not wait for Leaders to appear Created: 23/Sep/16  Updated: 25/Jul/23  Resolved: 10/Jul/17

Status: Resolved
Project: controller
Component/s: clustering
Affects Version/s: None
Fix Version/s: None

Type: Bug
Reporter: Vratko Polak Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: All
Platform: All

Issue Links:
Duplicate
duplicates CONTROLLER-1739 Dsbenchmark: Conflicting modification... Open
External issue ID: 6793

 Description   

This happens especially in cluster testing when odl-mdsal-benchmark is listed in featuresBoot.
Feature starts creating its initial data, but "default" shards may not yet be ready. This leads to "server is unhealthy". Note that this happens even before users starts any test.

To fix, either wait in activator, or create initial structure (each time) when user asks for a test.

Parts of karaf.log:

2016-09-23 08:48:39,386 | INFO | config-pusher | ConfigPusherImpl | 136 - org.opend
aylight.controller.config-persister-impl - 0.5.0.Boron | Pushing configuration snapshot dsbenchmark-imp
l-default-config.xml(odl-mdsal-benchmark,odl-mdsal-benchmark)
2016-09-23 08:48:39,390 | INFO | er [_value=8185] | WebSocketServer | 265 - org.opend
aylight.netconf.sal-rest-connector - 1.4.0.Boron | Web socket server started at port 8185.
2016-09-23 08:48:39,504 | INFO | config-pusher | DsbenchmarkListenerProvider | 269 - org.opend
aylight.controller.dsbenchmark - 1.2.0.Boron | DsbenchmarkListenerProvider created
2016-09-23 08:48:43,524 | INFO | lt-dispatcher-17 | EntityOwnershipShard | 193 - org.opend
aylight.controller.sal-akka-raft - 1.4.0.Boron | member-1-shard-entity-ownership-operational (Follower)
:- Switching from behavior Follower to Candidate
2016-09-23 08:48:43,525 | INFO | ult-dispatcher-2 | RoleChangeNotifier | 192 - org.opend
aylight.controller.sal-clustering-commons - 1.4.0.Boron | RoleChangeNotifier for member-1-shard-entity-
ownership-operational , received role change from Follower to Candidate
2016-09-23 08:48:43,525 | INFO | ult-dispatcher-2 | ShardManager | 197 - org.opend
aylight.controller.sal-distributed-datastore - 1.4.0.Boron | shard-manager-operational: Received role c
hanged for member-1-shard-entity-ownership-operational from Follower to Candidate
2016-09-23 08:48:49,463 | WARN | saction-71-73'}} | DeadlockMonitor | 132 - org.opend
aylight.controller.config-manager - 0.5.0.Boron | ModuleIdentifier

{factoryName='dsbenchmark-impl', inst anceName='dsbenchmark-default'} did not finish after 9983 ms
2016-09-23 08:48:54,463 | WARN | saction-71-73'}} | DeadlockMonitor | 132 - org.opend
aylight.controller.config-manager - 0.5.0.Boron | ModuleIdentifier{factoryName='dsbenchmark-impl', instanceName='dsbenchmark-default'}

did not finish after 14983 ms
2016-09-23 08:48:59,464 | WARN | saction-71-73'}} | DeadlockMonitor | 132 - org.opend
aylight.controller.config-manager - 0.5.0.Boron | ModuleIdentifier

{factoryName='dsbenchmark-impl', inst anceName='dsbenchmark-default'} did not finish after 19984 ms
2016-09-23 08:48:59,667 | WARN | lt-dispatcher-18 | ConcurrentDOMDataBroker | 197 - org.opend
aylight.controller.sal-distributed-datastore - 1.4.0.Boron | Tx: DOM-0 Error during phase CAN_COMMIT, s
tarting Abort
org.opendaylight.controller.cluster.datastore.exceptions.NoShardLeaderException: Shard member-1-shard-d
efault-operational currently has no leader. Try again later.
at org.opendaylight.controller.cluster.datastore.shardmanager.ShardManager.createNoShardLeaderE
xception(ShardManager.java:723)[197:org.opendaylight.controller.sal-distributed-datastore:1.4.0.Boron]
at org.opendaylight.controller.cluster.datastore.shardmanager.ShardManager.onShardNotInitialize
dTimeout(ShardManager.java:537)[197:org.opendaylight.controller.sal-distributed-datastore:1.4.0.Boron]
at org.opendaylight.controller.cluster.datastore.shardmanager.ShardManager.handleCommand(ShardM
anager.java:216)[197:org.opendaylight.controller.sal-distributed-datastore:1.4.0.Boron]
at org.opendaylight.controller.cluster.common.actor.AbstractUntypedPersistentActor.onReceiveCom
mand(AbstractUntypedPersistentActor.java:29)[192:org.opendaylight.controller.sal-clustering-commons:1.4
.0.Boron]
at akka.persistence.UntypedPersistentActor.onReceive(PersistentActor.scala:170)[186:com.typesaf
e.akka.persistence:2.4.7]
at org.opendaylight.controller.cluster.common.actor.MeteringBehavior.apply(MeteringBehavior.jav
a:97)[192:org.opendaylight.controller.sal-clustering-commons:1.4.0.Boron]
at akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:544)[180:com.typesafe.akk
a.actor:2.4.7]
at akka.actor.Actor$class.aroundReceive(Actor.scala:484)[180:com.typesafe.akka.actor:2.4.7]
at akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(PersistentActor.scala:168)[186:com.typesafe.akka.persistence:2.4.7]
at akka.persistence.Eventsourced$$anon$1.stateReceive(Eventsourced.scala:633)[186:com.typesafe.akka.persistence:2.4.7]
at akka.persistence.Eventsourced$class.aroundReceive(Eventsourced.scala:179)[186:com.typesafe.akka.persistence:2.4.7]
at akka.persistence.UntypedPersistentActor.aroundReceive(PersistentActor.scala:168)[186:com.typesafe.akka.persistence:2.4.7]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)[180:com.typesafe.akka.actor:2.4.7]
at akka.actor.ActorCell.invoke(ActorCell.scala:495)[180:com.typesafe.akka.actor:2.4.7]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)[180:com.typesafe.akka.actor:2.4.7]
at akka.dispatch.Mailbox.run(Mailbox.scala:224)[180:com.typesafe.akka.actor:2.4.7]
at akka.dispatch.Mailbox.exec(Mailbox.scala:234)[180:com.typesafe.akka.actor:2.4.7]
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)[176:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)[176:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)[176:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)[176:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
2016-09-23 08:48:59,678 | ERROR | config-pusher | ConfigTransactionControllerImpl | 132 - org.opend
aylight.controller.config-manager - 0.5.0.Boron | Commit failed on ModuleIdentifier{factoryName='dsbenchmark-impl', instanceName='dsbenchmark-default'}

in transaction TransactionIdentifier

{name='ConfigTrans action-71-73'}
java.lang.IllegalStateException: TransactionCommitFailedException{message=canCommit encountered an unex pected failure, errorList=[RpcError [message=canCommit encountered an unexpected failure, severity=ERRO R, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=org.opendaylight. controller.md.sal.common.api.data.DataStoreUnavailableException: Shard member-1-shard-default-operation al currently has no leader. Try again later.]]}
at org.opendaylight.dsbenchmark.DsbenchmarkProvider.setTestOperData(DsbenchmarkProvider.java:17
5)
at org.opendaylight.dsbenchmark.DsbenchmarkProvider.onSessionInitiated(DsbenchmarkProvider.java
:81)
at org.opendaylight.controller.md.sal.binding.util.BindingContextUtils.createProviderContextAnd
Initialize(BindingContextUtils.java:56)
at org.opendaylight.controller.sal.binding.impl.RootBindingAwareBroker.registerProvider(RootBin
dingAwareBroker.java:187)
at org.opendaylight.yang.gen.v1.urn.opendaylight.params.xml.ns.yang.dsbenchmark.impl.rev141210.
DsbenchmarkImplModule.createInstance(DsbenchmarkImplModule.java:30)
at org.opendaylight.controller.config.spi.AbstractModule.getInstance(AbstractModule.java:87)
at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0
_91]
at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_91]
at org.opendaylight.controller.config.manager.impl.dependencyresolver.DependencyResolverManager
$ModuleInvocationHandler.handleInvocation(DependencyResolverManager.java:149)
at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87
)
at com.sun.proxy.$Proxy34.getInstance(Unknown Source)
at org.opendaylight.controller.config.manager.impl.ConfigTransactionControllerImpl.secondPhaseC
ommit(ConfigTransactionControllerImpl.java:437)
at org.opendaylight.controller.config.manager.impl.ConfigRegistryImpl.secondPhaseCommit(ConfigR
egistryImpl.java:346)
at org.opendaylight.controller.config.manager.impl.ConfigRegistryImpl.commitConfigSafe(ConfigRe
gistryImpl.java:285)
at org.opendaylight.controller.config.manager.impl.ConfigRegistryImpl.commitConfig(ConfigRegist
ryImpl.java:250)

...

2016-09-23 08:48:59,706 | ERROR | config-pusher | ConfigRegistryImpl | 132 - org.opendaylight.controller.config-manager - 0.5.0.Boron | Configuration Transaction failed on 2PC, server is unhealthy
java.lang.IllegalStateException: Error - getInstance() failed for ModuleIdentifier{factoryName='dsbenchmark-impl', instanceName='dsbenchmark-default'} in transaction TransactionIdentifier{name='ConfigTransaction-71-73'}

at org.opendaylight.controller.config.manager.impl.ConfigTransactionControllerImpl.secondPhaseCommit(ConfigTransactionControllerImpl.java:444)
at org.opendaylight.controller.config.manager.impl.ConfigRegistryImpl.secondPhaseCommit(ConfigRegistryImpl.java:346)
at org.opendaylight.controller.config.manager.impl.ConfigRegistryImpl.commitConfigSafe(ConfigRegistryImpl.java:285)
at org.opendaylight.controller.config.manager.impl.ConfigRegistryImpl.commitConfig(ConfigRegistryImpl.java:250)

 Comments   
Comment by Vratko Polak [ 28/Feb/17 ]

An attempt to fix:
https://git.opendaylight.org/gerrit/#/c/52317/

(Un)fortunately, this bug is currently not visible, so the fix has not been tested.

Un-assigning for now.

Comment by Vratko Polak [ 10/Jul/17 ]

This should get fixed together with newer duplicate CONTROLLER-1739.

Generated at Wed Feb 07 19:55:50 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.