Uploaded image for project: 'controller'
  1. controller
  2. CONTROLLER-1700

Timeout waiting for task from writer started after heal after long isolation

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved
    • Resolution: Done
    • None
    • None
    • clustering
    • None
    • Operating System: All
      Platform: All

    • 8562

    Description

      This symptom is very similar to CONTROLLER-1674 but happens in a different scenario.

      Still three writers (module-based shards, tell-based protocol) and leader is isolated, for more than request timeout. The isolated writer fails (with TimeoutException instead of RequestTimeoutException, that might be a separate bug or a cause of this one). Then the member is rejoined and it is verified each shard has a Leader and two Followers.

      At the end of the scenario, we start a writer on the rejoined node, and we expect it to finish writing without errors. Instead finishing after 67 seconds, TimeoutException is seen. The other two writers finish correctly.

      In a recent Sandbox test [0] (which failed to upload archive with logs) the response starts with:
      {"errors":{"error":[{"error-type":"application","error-tag":"operation-failed","error-message":"Unexpected-exception","error-info":"java.util.concurrent.TimeoutException: Timeout waiting for task.\n\tat com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:269)\n\tat com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:96)\n\tat com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:69)\n\tat org.opendaylight.yangtools.util.concurrent.MappingCheckedFuture.get(MappingCheckedFuture.java:85)\n\tat com.google.common.util.concurrent.AbstractCheckedFuture.checkedGet(AbstractCheckedFuture.java:107)\n\tat org.opendaylight.controller.clustering.it.provider.impl.WriteTransactionsHandler.ensureListExists(WriteTransactionsHandler.java:146)\n\tat org.opendaylight.controller.clustering.it.provider.impl.WriteTransactionsHandler.start(WriteTransactionsHandler.java:126)\n\tat org.opendaylight.controller.clustering.it.provider.MdsalLowLevelTestProvider.writeTransactions(MdsalLowLevelTestProvider.java:250)\n\tat org.opendaylight.yangtools.yang.binding.util.RpcMethodInvokerWithInput.invokeOn(RpcMethodInvokerWithInput.java:31)\n\tat

      [0] https://jenkins.opendaylight.org/sandbox/job/controller-csit-3node-clustering-only-carbon/6/robot/report/log.html#s1-s10-t3-k2-k25-k1-k8

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              Unassigned Unassigned
              vrpolak Vratko Polak
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: