[CONTROLLER-1712] RequestTimeoutException happens few seconds before it should (120s) Created: 08/Jun/17  Updated: 25/Jul/23  Resolved: 14/Jun/17

Status: Resolved
Project: controller
Component/s: clustering
Affects Version/s: None
Fix Version/s: None

Type: Bug
Reporter: Vratko Polak Assignee: Unassigned
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: All
Platform: All


External issue ID: 8636

 Description   

This is probably duplicate of CONTROLLER-1679 but that was opened as a blocker, and the current response is different.

This affects test cases where prefix-based shard leader is isolated while single transaction producer is on a different node (if it is on the same node, CONTROLLER-1687 happens instead).

Response from transaction producer [0] starts with:
{"errors":{"error":[{"error-type":"application","error-tag":"operation-failed","error-message":"Unexpected-exception","error-info":"TransactionCommitFailedException

{message=submit execution failed, errorList=[RpcError [message=submit execution failed, severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=org.opendaylight.controller.cluster.access.client.RequestTimeoutException: Timed out after 120027148727ns]]}

\n\tat org.opendaylight.mdsal.dom.broker.TransactionCommitFailedExceptionMapper.newWithCause(TransactionCommitFailedExceptionMapper.java:37)\n\tat

This may happen if there was a transaction opened ~3 seconds before isolation, but the backend took more than that to process it, so the final confirmation was blocked by the isolation.

The rate is 1000 transaction per second (implemented by producer waiting 1 millisecond after each submit) and warmup period is 5 seconds. We may need to lower the transaction rate (considering those are writes to config datastore) and subtract few seconds from the period where failures are not tolerated.

[0] https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/736/log.html.gz#s1-s34-t5-k2-k15-k1-k1-k1-k1-k1-k1-k2-k1-k1-k1



 Comments   
Comment by Vratko Polak [ 14/Jun/17 ]

Fixed on suite [1] side.

[1] https://git.opendaylight.org/gerrit/58598

Generated at Wed Feb 07 19:56:15 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.