Uploaded image for project: 'controller'
  1. controller
  2. CONTROLLER-1552

odl-mdsal-benchmark does not wait for Leaders to appear

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Duplicate
    • None
    • None
    • clustering
    • None
    • Operating System: All
      Platform: All

    • 6793

      This happens especially in cluster testing when odl-mdsal-benchmark is listed in featuresBoot.
      Feature starts creating its initial data, but "default" shards may not yet be ready. This leads to "server is unhealthy". Note that this happens even before users starts any test.

      To fix, either wait in activator, or create initial structure (each time) when user asks for a test.

      Parts of karaf.log:

      2016-09-23 08:48:39,386 | INFO | config-pusher | ConfigPusherImpl | 136 - org.opend
      aylight.controller.config-persister-impl - 0.5.0.Boron | Pushing configuration snapshot dsbenchmark-imp
      l-default-config.xml(odl-mdsal-benchmark,odl-mdsal-benchmark)
      2016-09-23 08:48:39,390 | INFO | er [_value=8185] | WebSocketServer | 265 - org.opend
      aylight.netconf.sal-rest-connector - 1.4.0.Boron | Web socket server started at port 8185.
      2016-09-23 08:48:39,504 | INFO | config-pusher | DsbenchmarkListenerProvider | 269 - org.opend
      aylight.controller.dsbenchmark - 1.2.0.Boron | DsbenchmarkListenerProvider created
      2016-09-23 08:48:43,524 | INFO | lt-dispatcher-17 | EntityOwnershipShard | 193 - org.opend
      aylight.controller.sal-akka-raft - 1.4.0.Boron | member-1-shard-entity-ownership-operational (Follower)
      :- Switching from behavior Follower to Candidate
      2016-09-23 08:48:43,525 | INFO | ult-dispatcher-2 | RoleChangeNotifier | 192 - org.opend
      aylight.controller.sal-clustering-commons - 1.4.0.Boron | RoleChangeNotifier for member-1-shard-entity-
      ownership-operational , received role change from Follower to Candidate
      2016-09-23 08:48:43,525 | INFO | ult-dispatcher-2 | ShardManager | 197 - org.opend
      aylight.controller.sal-distributed-datastore - 1.4.0.Boron | shard-manager-operational: Received role c
      hanged for member-1-shard-entity-ownership-operational from Follower to Candidate
      2016-09-23 08:48:49,463 | WARN | saction-71-73'}} | DeadlockMonitor | 132 - org.opend
      aylight.controller.config-manager - 0.5.0.Boron | ModuleIdentifier

      {factoryName='dsbenchmark-impl', inst anceName='dsbenchmark-default'} did not finish after 9983 ms
      2016-09-23 08:48:54,463 | WARN | saction-71-73'}} | DeadlockMonitor | 132 - org.opend
      aylight.controller.config-manager - 0.5.0.Boron | ModuleIdentifier{factoryName='dsbenchmark-impl', instanceName='dsbenchmark-default'}

      did not finish after 14983 ms
      2016-09-23 08:48:59,464 | WARN | saction-71-73'}} | DeadlockMonitor | 132 - org.opend
      aylight.controller.config-manager - 0.5.0.Boron | ModuleIdentifier

      {factoryName='dsbenchmark-impl', inst anceName='dsbenchmark-default'} did not finish after 19984 ms
      2016-09-23 08:48:59,667 | WARN | lt-dispatcher-18 | ConcurrentDOMDataBroker | 197 - org.opend
      aylight.controller.sal-distributed-datastore - 1.4.0.Boron | Tx: DOM-0 Error during phase CAN_COMMIT, s
      tarting Abort
      org.opendaylight.controller.cluster.datastore.exceptions.NoShardLeaderException: Shard member-1-shard-d
      efault-operational currently has no leader. Try again later.
      at org.opendaylight.controller.cluster.datastore.shardmanager.ShardManager.createNoShardLeaderE
      xception(ShardManager.java:723)[197:org.opendaylight.controller.sal-distributed-datastore:1.4.0.Boron]
      at org.opendaylight.controller.cluster.datastore.shardmanager.ShardManager.onShardNotInitialize
      dTimeout(ShardManager.java:537)[197:org.opendaylight.controller.sal-distributed-datastore:1.4.0.Boron]
      at org.opendaylight.controller.cluster.datastore.shardmanager.ShardManager.handleCommand(ShardM
      anager.java:216)[197:org.opendaylight.controller.sal-distributed-datastore:1.4.0.Boron]
      at org.opendaylight.controller.cluster.common.actor.AbstractUntypedPersistentActor.onReceiveCom
      mand(AbstractUntypedPersistentActor.java:29)[192:org.opendaylight.controller.sal-clustering-commons:1.4
      .0.Boron]
      at akka.persistence.UntypedPersistentActor.onReceive(PersistentActor.scala:170)[186:com.typesaf
      e.akka.persistence:2.4.7]
      at org.opendaylight.controller.cluster.common.actor.MeteringBehavior.apply(MeteringBehavior.jav
      a:97)[192:org.opendaylight.controller.sal-clustering-commons:1.4.0.Boron]
      at akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:544)[180:com.typesafe.akk
      a.actor:2.4.7]
      at akka.actor.Actor$class.aroundReceive(Actor.scala:484)[180:com.typesafe.akka.actor:2.4.7]
      at akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(PersistentActor.scala:168)[186:com.typesafe.akka.persistence:2.4.7]
      at akka.persistence.Eventsourced$$anon$1.stateReceive(Eventsourced.scala:633)[186:com.typesafe.akka.persistence:2.4.7]
      at akka.persistence.Eventsourced$class.aroundReceive(Eventsourced.scala:179)[186:com.typesafe.akka.persistence:2.4.7]
      at akka.persistence.UntypedPersistentActor.aroundReceive(PersistentActor.scala:168)[186:com.typesafe.akka.persistence:2.4.7]
      at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)[180:com.typesafe.akka.actor:2.4.7]
      at akka.actor.ActorCell.invoke(ActorCell.scala:495)[180:com.typesafe.akka.actor:2.4.7]
      at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)[180:com.typesafe.akka.actor:2.4.7]
      at akka.dispatch.Mailbox.run(Mailbox.scala:224)[180:com.typesafe.akka.actor:2.4.7]
      at akka.dispatch.Mailbox.exec(Mailbox.scala:234)[180:com.typesafe.akka.actor:2.4.7]
      at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)[176:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
      at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)[176:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
      at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)[176:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
      at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)[176:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
      2016-09-23 08:48:59,678 | ERROR | config-pusher | ConfigTransactionControllerImpl | 132 - org.opend
      aylight.controller.config-manager - 0.5.0.Boron | Commit failed on ModuleIdentifier{factoryName='dsbenchmark-impl', instanceName='dsbenchmark-default'}

      in transaction TransactionIdentifier

      {name='ConfigTrans action-71-73'}
      java.lang.IllegalStateException: TransactionCommitFailedException{message=canCommit encountered an unex pected failure, errorList=[RpcError [message=canCommit encountered an unexpected failure, severity=ERRO R, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=org.opendaylight. controller.md.sal.common.api.data.DataStoreUnavailableException: Shard member-1-shard-default-operation al currently has no leader. Try again later.]]}
      at org.opendaylight.dsbenchmark.DsbenchmarkProvider.setTestOperData(DsbenchmarkProvider.java:17
      5)
      at org.opendaylight.dsbenchmark.DsbenchmarkProvider.onSessionInitiated(DsbenchmarkProvider.java
      :81)
      at org.opendaylight.controller.md.sal.binding.util.BindingContextUtils.createProviderContextAnd
      Initialize(BindingContextUtils.java:56)
      at org.opendaylight.controller.sal.binding.impl.RootBindingAwareBroker.registerProvider(RootBin
      dingAwareBroker.java:187)
      at org.opendaylight.yang.gen.v1.urn.opendaylight.params.xml.ns.yang.dsbenchmark.impl.rev141210.
      DsbenchmarkImplModule.createInstance(DsbenchmarkImplModule.java:30)
      at org.opendaylight.controller.config.spi.AbstractModule.getInstance(AbstractModule.java:87)
      at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0
      _91]
      at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_91]
      at org.opendaylight.controller.config.manager.impl.dependencyresolver.DependencyResolverManager
      $ModuleInvocationHandler.handleInvocation(DependencyResolverManager.java:149)
      at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87
      )
      at com.sun.proxy.$Proxy34.getInstance(Unknown Source)
      at org.opendaylight.controller.config.manager.impl.ConfigTransactionControllerImpl.secondPhaseC
      ommit(ConfigTransactionControllerImpl.java:437)
      at org.opendaylight.controller.config.manager.impl.ConfigRegistryImpl.secondPhaseCommit(ConfigR
      egistryImpl.java:346)
      at org.opendaylight.controller.config.manager.impl.ConfigRegistryImpl.commitConfigSafe(ConfigRe
      gistryImpl.java:285)
      at org.opendaylight.controller.config.manager.impl.ConfigRegistryImpl.commitConfig(ConfigRegist
      ryImpl.java:250)

      ...

      2016-09-23 08:48:59,706 | ERROR | config-pusher | ConfigRegistryImpl | 132 - org.opendaylight.controller.config-manager - 0.5.0.Boron | Configuration Transaction failed on 2PC, server is unhealthy
      java.lang.IllegalStateException: Error - getInstance() failed for ModuleIdentifier{factoryName='dsbenchmark-impl', instanceName='dsbenchmark-default'} in transaction TransactionIdentifier{name='ConfigTransaction-71-73'}

      at org.opendaylight.controller.config.manager.impl.ConfigTransactionControllerImpl.secondPhaseCommit(ConfigTransactionControllerImpl.java:444)
      at org.opendaylight.controller.config.manager.impl.ConfigRegistryImpl.secondPhaseCommit(ConfigRegistryImpl.java:346)
      at org.opendaylight.controller.config.manager.impl.ConfigRegistryImpl.commitConfigSafe(ConfigRegistryImpl.java:285)
      at org.opendaylight.controller.config.manager.impl.ConfigRegistryImpl.commitConfig(ConfigRegistryImpl.java:250)

            Unassigned Unassigned
            vrpolak Vratko Polak
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: