https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-1631 controller <p>The test scenario follows the steps of DOMDataBroker testing: Clean Leader Shutdown: local leader shurdown<br/> 1) producer started with write-transactions rpc on the config default leader node<br/> 2) the default config shard leader was removed via remove shard replica rpc<br/> 3) producer aborted and returned http status 500</p> <p>{"errors":{"error":[{"error-type":"application","error-tag":"operation-failed","error-message":"Unexpected-exception","error-info":"TransactionCommitFailedException</p> {message=canCommit encountered an unexpected failure, errorList=[RpcError [message=canCommit encountered an unexpected failure, severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=akka.pattern.AskTimeoutException: Ask timed out on [ActorSelection[Anchor(akka://opendaylight-cluster-data/), Path(/user/shardmanager-config/member-1-shard-default-config#-1992099962)]] after [30000 ms]. Sender[null] sent message of type "org.opendaylight.controller.cluster.datastore.messages.ReadyLocalTransaction".]]} <p> at org.opendaylight.controller.md.sal.dom.broker.impl.TransactionCommitFailedExceptionMapper.newWithCause(TransactionCommitFailedExceptionMapper.java:37)<br/> at org.opendaylight.controller.md.sal.dom.broker.impl.TransactionCommitFailedExceptionMapper.newWithCause(TransactionCommitFailedExceptionMapper.java:18)<br/> at org.opendaylight.yangtools.util.concurrent.ExceptionMapper.apply(ExceptionMapper.java:102)<br/> at org.opendaylight.controller.cluster.databroker.ConcurrentDOMDataBroker.handleException(ConcurrentDOMDataBroker.java:222)<br/> at org.opendaylight.controller.cluster.databroker.ConcurrentDOMDataBroker.access$000(ConcurrentDOMDataBroker.java:46)<br/> at org.opendaylight.controller.cluster.databroker.ConcurrentDOMDataBroker$1.onFailure(ConcurrentDOMDataBroker.java:126)<br/> at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)<br/> at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)<br/> at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)<br/> at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)<br/> at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)<br/> at com.google.common.util.concurrent.SettableFuture.setException(SettableFuture.java:68)<br/> at org.opendaylight.controller.cluster.datastore.SingleCommitCohortProxy$1.onComplete(SingleCommitCohortProxy.java:58)<br/> at akka.dispatch.OnComplete.internal(Future.scala:258)<br/> at akka.dispatch.OnComplete.internal(Future.scala:256)<br/> at akka.dispatch.japi$CallbackBridge.apply(Future.scala:186)<br/> at akka.dispatch.japi$CallbackBridge.apply(Future.scala:183)<br/> at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)<br/> at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)<br/> at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)<br/> at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)<br/> at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)<br/> at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)<br/> at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)<br/> at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)<br/> at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:415)<br/> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)<br/> at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)<br/> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)<br/> at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)<br/> Caused by: akka.pattern.AskTimeoutException: Ask timed out on <a href="#-1992099962)]" target="_blank" rel="noopener">ActorSelection[Anchor(akka://opendaylight-cluster-data/), Path(/user/shardmanager-config/member-1-shard-default-config#-1992099962)]</a> after <span class="error">&#91;30000 ms&#93;</span>. Sender<span class="error">&#91;null&#93;</span> sent message of type "org.opendaylight.controller.cluster.datastore.messages.ReadyLocalTransaction".<br/> at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604)<br/> at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126)<br/> at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)<br/> at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)<br/> at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)<br/> at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:329)<br/> at akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:280)<br/> at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:284)<br/> at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:236)<br/> at java.lang.Thread.run(Thread.java:745)<br/> "}]}}</p> <p>Operating System: All<br/> Platform: All</p> CONTROLLER-1631

Transaction producer aborted when local shard leader removed

Bug Resolved Done Robert Varga Peter Gubka Tue, 11 Apr 2017 09:53:32 +0000 Tue, 25 Jul 2023 08:24:29 +0000 Sat, 15 Apr 2017 22:15:32 +0000 clustering 0 2 <p>Attachment log.html has been added with description: robot log</p> <p>Attachment karaf_log_1.tar.gz has been added with description: node1 log</p> <p>Attachment karaf_log_2.tar.gz has been added with description: node2 log</p> <p>Attachment karaf_log_3.tar.gz has been added with description: node3 log</p> <p>What's the problem here? Can you provide more details as to the purpose of the test and what was expected? From my understanding of the description, you had a component committing transactions on the shard leader then removed the shard replica on the leader and a transaction failed after (timed out). I would expect that to occur.</p> <p>(In reply to Tom Pantelis from comment #4)<br/> &gt; What's the problem here? Can you provide more details as to the purpose of<br/> &gt; the test and what was expected? From my understanding of the description,<br/> &gt; you had a component committing transactions on the shard leader then removed<br/> &gt; the shard replica on the leader and a transaction failed after (timed out).<br/> &gt; I would expect that to occur.</p> <p>The odl clustering test plan document describes the following:</p> <p>Clean Leader Shutdown<br/> ---------------------<br/> The goal is to ensure that applications do not observe disruption when a shard leader is shut down cleanly. This is performed by having a steady-stream producer execute operations against the shard and then initiate leader shard shutdown, then the producer is shut down cleanly.</p> <p>This test is executed in two scenarios:</p> <ul class="alternate" type="square"> <li>Three-node, with shard leader being local</li> <li>Three-node, with shard leader being remote</li> </ul> <p>Success criteria are:</p> <ul class="alternate" type="square"> <li>No transaction errors occur</li> <li>Producer shuts down cleanly (i.e. all transactions complete successfully)</li> </ul> <p>Test tool: test-transaction-producer, running at 1K tps</p> <ul class="alternate" type="square"> <li>Steady, configurable producer started with:<br/> o A transaction chain<br/> o Single transactions (note: these cannot overlap)</li> <li>Configurable transaction rate (i.e. transactions-per-second)</li> <li>Single-operation transactions</li> <li>Random mix across 1M entries</li> </ul> <p>My test case start transaction producer with write-transaction rpc and the use remove-shard-replica rpc to "shutdown" the leader.<br/> Rpc wite-transaction then return 500, but my understanding is that new leader will be elected and transactions should finish well and http status 200 is expected from write-transactions rpc.</p> <p>I don't know who wrote the test plan but there's no guarantee that all transactions will succeed across leader changes. It does retry transactions if there's no current leader but a transaction already in flight may time out and fail. Now that's the behavior with the current implementation. With Robert's new front-end stuff tracked by <a href="https://jira.opendaylight.org/browse/CONTROLLER-1483" title="akka.pattern.AskTimeoutException on follower while BGP peer introduces 1M prefixes" class="issue-link" data-issue-key="CONTROLLER-1483"><del>CONTROLLER-1483</del></a> the intention is to retry transactions in all cases to also cover ones in flight. So I think the expectations in the test plan <b>should</b> be consistent with 5280. I believe you can enable the new behavior via the cfg file but it isn't on by default yet. Is that correct Robert?</p> <p>Peter, is this with tell-based-protocol=true? The AskTimeoutException seems to be indicating otherwise.</p> <p>(In reply to Robert Varga from comment #7)<br/> &gt; Peter, is this with tell-based-protocol=true? The AskTimeoutException seems<br/> &gt; to be indicating otherwise.</p> <p>No. tell-based-protocol was not enabled.<br/> Is this the right way to enable it? <a href="https://git.opendaylight.org/gerrit/#/c/54340/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/54340/</a></p> <p>Because i used it here <a href="https://jenkins.opendaylight.org/sandbox/job/bgpcep-csit-3node-periodic-bgpclustering-only-carbon/1/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/sandbox/job/bgpcep-csit-3node-periodic-bgpclustering-only-carbon/1/</a> and the particular log message is not in the logs.</p> <p>This scenario assumes tell-based protocol, <a href="https://git.opendaylight.org/gerrit/54848" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/54848</a> should fix its activation.</p> <p>master: <a href="https://git.opendaylight.org/gerrit/54956" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/54956</a></p> Development External issue ID 8205 External issue URL Issue Type ODL SR Target Milestone Rank 0|i02rrj: