Uploaded image for project: 'controller'
  1. controller
  2. CONTROLLER-1678

Futures timed out after short isolation

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved
    • Resolution: Done
    • None
    • None
    • clustering
    • None
    • Operating System: All
      Platform: All

    • 8450

    Description

      This is basically CONTROLLER-1660 but this time tell-based protocol (and prefix-based shard) is used.

      Less than 12 seconds after the isolation, transaction writer still fails [0].
      Errors (without stack traces) are:
      Problem accessing /restconf/operations/odl-mdsal-lowlevel-control:produce-transactions. Reason:
      <pre> Server Error</pre></p><h3>Caused by:</h3><pre>org.opendaylight.controller.cluster.datastore.exceptions.TimeoutException: Sending message class org.opendaylight.controller.cluster.sharding.messages.ProducerCreated to actor Actorakka://opendaylight-cluster-data/user/ShardedDOMDataTreeFrontend#-1001725550 failed. Try again later.
      Caused by: java.util.concurrent.TimeoutException: Futures timed out after [5000 milliseconds]

      Looking at karaf.log [1], the only suspicious message is AskTimeoutExcepton on a prefix-based shard, perhaps it was caused by Jolokia request?
      2017-05-14 06:29:28,171 | WARN | monPool-worker-2 | AbstractShardBackendResolver | 199 - org.opendaylight.controller.sal-distributed-datastore - 1.5.0.Carbon | Failed to resolve shard
      akka.pattern.AskTimeoutException: Ask timed out on [ActorSelection[Anchor(akka.tcp://opendaylight-cluster-data@10.29.14.191:2550/), Path(/user/shardmanager-config/member-2-shard-id-ints!-config)]] after [5000 ms]. Sender[null] sent message of type "org.opendaylight.controller.cluster.access.commands.ConnectClientRequest".
      at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604)[173:com.typesafe.akka.actor:2.4.17]
      at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126)[173:com.typesafe.akka.actor:2.4.17]
      at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)[169:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
      at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)[169:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
      at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)[169:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
      at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:329)[173:com.typesafe.akka.actor:2.4.17]
      at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:280)[173:com.typesafe.akka.actor:2.4.17]
      at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:284)[173:com.typesafe.akka.actor:2.4.17]
      at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:236)[173:com.typesafe.akka.actor:2.4.17]
      at java.lang.Thread.run(Thread.java:745)[:1.8.0_121]

      [0] https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/684/archives/log.html.gz#s1-s35-t1-k2-k15
      [1] https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/684/archives/odl3_karaf.log.gz

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            tcere Tomas Cere
            vrpolak Vratko Polak
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: