Uploaded image for project: 'controller'
  1. controller
  2. CONTROLLER-1753

AskTimeoutException in dsbenchmark nitrogen -all- job

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • None
    • Nitrogen
    • clustering
    • None
    • Operating System: All
      Platform: All

    • 9021

      This occasionally affects CSIT results: [0].

      Here is a copy of the restconf error message:
      {"errors":{"error":[

      {"error-type":"application","error-tag":"operation-failed","error-message":"The operation encountered an unexpected error while executing.","error-info":"akka.pattern.AskTimeoutException: Ask timed out on [Actor[akka.tcp://opendaylight-cluster-data@10.29.15.213:2550/user/rpc/broker#1754800661]] after [15000 ms]. Sender[null] sent message of type \"org.opendaylight.controller.remote.rpc.messages.ExecuteRpc\".\n\tat akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604)\n\tat akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126)\n\tat scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)\n\tat scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)\n\tat scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)\n\tat akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:329)\n\tat akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:280)\n\tat akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:284)\n\tat akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:236)\n\tat java.lang.Thread.run(Thread.java:748)\n"}

      ]}}

      Karaf.log does not contain any exception. Moreover, even though the test is aiming at current leader [1] (member-3), the INFO messages in karaf.log are present on member-2 [2] instead.

      As the title suggests, this does not happen in only jobs nor in Carbon jobs.
      Apparently, some other ODL feature is slowing datastore down, but even then, AskTimeoutException usually happens only for long pauses (the job does not start ODL with Garbage Collection logging). To be clear, no leader movement or UnreachableMember happens either.

      [0] https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-periodic-benchmark-all-nitrogen/116/log.html.gz#s1-t2-k2-k6-k2
      [1] https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-periodic-benchmark-all-nitrogen/116/log.html.gz#s1-t2-k2-k1-k3-k5
      [2] https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-periodic-benchmark-all-nitrogen/116/odl2_karaf.log.gz

            Unassigned Unassigned
            vrpolak Vratko Polak
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: