Uploaded image for project: 'controller'
  1. controller
  2. CONTROLLER-1712

RequestTimeoutException happens few seconds before it should (120s)

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved
    • Resolution: Done
    • None
    • None
    • clustering
    • None
    • Operating System: All
      Platform: All

    • 8636

    Description

      This is probably duplicate of CONTROLLER-1679 but that was opened as a blocker, and the current response is different.

      This affects test cases where prefix-based shard leader is isolated while single transaction producer is on a different node (if it is on the same node, CONTROLLER-1687 happens instead).

      Response from transaction producer [0] starts with:
      {"errors":{"error":[{"error-type":"application","error-tag":"operation-failed","error-message":"Unexpected-exception","error-info":"TransactionCommitFailedException

      {message=submit execution failed, errorList=[RpcError [message=submit execution failed, severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=org.opendaylight.controller.cluster.access.client.RequestTimeoutException: Timed out after 120027148727ns]]}

      \n\tat org.opendaylight.mdsal.dom.broker.TransactionCommitFailedExceptionMapper.newWithCause(TransactionCommitFailedExceptionMapper.java:37)\n\tat

      This may happen if there was a transaction opened ~3 seconds before isolation, but the backend took more than that to process it, so the final confirmation was blocked by the isolation.

      The rate is 1000 transaction per second (implemented by producer waiting 1 millisecond after each submit) and warmup period is 5 seconds. We may need to lower the transaction rate (considering those are writes to config datastore) and subtract few seconds from the period where failures are not tolerated.

      [0] https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/736/log.html.gz#s1-s34-t5-k2-k15-k1-k1-k1-k1-k1-k1-k2-k1-k1-k1

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            Unassigned Unassigned
            vrpolak Vratko Polak
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: