Uploaded image for project: 'controller'
  1. controller
  2. CONTROLLER-1689

stopping resolution of shard 0 on stale connection: AskTimeoutException

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved
    • Resolution: Done
    • None
    • None
    • clustering
    • None
    • Operating System: All
      Platform: All

    • 8511

    Description

      This is the karaf.log [0] symptom for robot failure [1] already reported as [2].

      Scenario: Module-based shard with tell-based protocol, single writer active on the leader member. Then remove-shard-replica is called, the writer (30s duration) should report success within 180 seconds.

      The suspicious log segment:
      017-05-19 05:05:34,352 | INFO | lt-dispatcher-31 | EmptyLocalActorRef | 174 - com.typesafe.akka.slf4j - 2.4.17 | Message [org.opendaylight.controller.cluster.access.commands.ConnectClientRequest] from Actor[akka://opendaylight-cluster-data/temp/$g] to Actorakka://opendaylight-cluster-data/user/shardmanager-config/member-2-shard-default-config#-863014827 was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
      2017-05-19 05:05:39,370 | INFO | lt-dispatcher-21 | ClientActorBehavior | 197 - org.opendaylight.controller.cds-access-client - 1.1.0.Carbon | member-2-frontend-datastore-config: stopping resolution of shard 0 on stale connection ReconnectingClientConnection{client=ClientIdentifier

      {frontend=member-2-frontend-datastore-config, generation=0}

      , cookie=0, backend=ShardBackendInfo{actor=Actorakka://opendaylight-cluster-data/user/shardmanager-config/member-2-shard-default-config#-863014827, sessionId=0, version=BORON, maxMessages=1000, cookie=0, shard=default, dataTree present=true}}
      akka.pattern.AskTimeoutException: Ask timed out on ActorSelection[Anchor(akka://opendaylight-cluster-data/), Path(/user/shardmanager-config/member-2-shard-default-config#-863014827)] after [5000 ms]. Sender[null] sent message of type "org.opendaylight.controller.cluster.access.commands.ConnectClientRequest".
      at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604)[173:com.typesafe.akka.actor:2.4.17]
      at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126)[173:com.typesafe.akka.actor:2.4.17]
      at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)[169:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
      at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)[169:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
      at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)[169:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
      at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:329)[173:com.typesafe.akka.actor:2.4.17]
      at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:280)[173:com.typesafe.akka.actor:2.4.17]
      at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:284)[173:com.typesafe.akka.actor:2.4.17]
      at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:236)[173:com.typesafe.akka.actor:2.4.17]
      at java.lang.Thread.run(Thread.java:745)[:1.8.0_121]
      2017-05-19 05:05:39,371 | WARN | monPool-worker-3 | AbstractShardBackendResolver | 199 - org.opendaylight.controller.sal-distributed-datastore - 1.5.0.Carbon | Failed to resolve shard
      akka.pattern.AskTimeoutException: Ask timed out on ActorSelection[Anchor(akka://opendaylight-cluster-data/), Path(/user/shardmanager-config/member-2-shard-default-config#-863014827)] after [5000 ms]. Sender[null] sent message of type "org.opendaylight.controller.cluster.access.commands.ConnectClientRequest".
      at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604)[173:com.typesafe.akka.actor:2.4.17]
      at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126)[173:com.typesafe.akka.actor:2.4.17]
      at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)[169:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
      at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)[169:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
      at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)[169:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
      at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:329)[173:com.typesafe.akka.actor:2.4.17]
      at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:280)[173:com.typesafe.akka.actor:2.4.17]
      at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:284)[173:com.typesafe.akka.actor:2.4.17]
      at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:236)[173:com.typesafe.akka.actor:2.4.17]
      at java.lang.Thread.run(Thread.java:745)[:1.8.0_121]

      [0] https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/697/archives/odl2_karaf.log.gz
      [1] https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/697/archives/log.html.gz#s1-s20-t1-k2-k9
      [2] https://bugs.opendaylight.org/show_bug.cgi?id=8403#c3

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              rovarga Robert Varga
              vrpolak Vratko Polak
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: