https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-1674 controller <p>This happened <span class="error">&#91;0&#93;</span> in a suite for module-based shards with tell-based protocol.</p> <p>Karaf.log <span class="error">&#91;1&#93;</span> shows an unexpected UnreachableMember at 06:17:00,808, followed by many warning as this:<br/> 2017-05-14 06:17:16,519 | WARN | ult-dispatcher-2 | FrontendClientMetadataBuilder | 199 - org.opendaylight.controller.sal-distributed-datastore - 1.5.0.Carbon | member-2-shard-default-config: Unknown history for commited transaction member-2-datastore-config-fe-0-chn-2-txn-2-0, ignoring</p> <p><span class="error">&#91;0&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/684/archives/log.html.gz#s1-s29-t1-k2-k9-k1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/684/archives/log.html.gz#s1-s29-t1-k2-k9-k1</a><br/> <span class="error">&#91;1&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/684/archives/odl2_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/684/archives/odl2_karaf.log.gz</a></p> <p>Operating System: All<br/> Platform: All</p> CONTROLLER-1674

Timeout waiting for task in producer when moving shard leader

Bug Resolved Done Robert Varga Vratko Polak Mon, 15 May 2017 10:38:13 +0000 Tue, 25 Jul 2023 08:24:37 +0000 Mon, 3 Jul 2017 12:09:51 +0000 clustering 0 2 <p>Today, the writer has timed out <span class="error">&#91;2&#93;</span> on restconf level, without returning a timeout error on its own. Karaf.log <span class="error">&#91;3&#93;</span> shows no suspicious message within the test case duration (between 03:01:22,254 and 03:02:52,753).</p> <p>Note that the run was still without a potential fix <span class="error">&#91;4&#93;</span>.</p> <p><span class="error">&#91;2&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/692/archives/log.html.gz#s1-s24-t1-k2-k8" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/692/archives/log.html.gz#s1-s24-t1-k2-k8</a><br/> <span class="error">&#91;3&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/692/archives/odl2_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/692/archives/odl2_karaf.log.gz</a><br/> <span class="error">&#91;4&#93;</span> <a href="https://git.opendaylight.org/gerrit/56874" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/56874</a></p> <p>This is still happening. The good news is that now only one writer reports this error <span class="error">&#91;5&#93;</span>, others pass (albeit with lower performance as they wait for submits).</p> <p><span class="error">&#91;5&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/694/archives/log.html.gz#s1-s36-t3-k2-k13" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/694/archives/log.html.gz#s1-s36-t3-k2-k13</a></p> <p>In another scenario (the previous one has listener and moves leader gracefully, this one has no listener but isolates the leader for a brief time, both scenarios have 3 transaction writers and 2 of them finish correctly now; module-based shards with tell-based protocol) something similar <span class="error">&#91;6&#93;</span> happened.</p> <p>But the exceptions are slightly different. The graceful movement response starts with:<br/> {"errors":{"error":[{"error-type":"application","error-tag":"operation-failed","error-message":"Unexpected-exception","error-info":"java.util.concurrent.TimeoutException: Timeout waiting for task.\n\tat com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:269)\n\tat com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:96)\n\tat com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:69)\n\tat org.opendaylight.yangtools.util.concurrent.MappingCheckedFuture.get(MappingCheckedFuture.java:85)\n\tat </p> <p>The brief isolation response starts with:<br/> {"errors":{"error":[{"error-type":"application","error-tag":"operation-failed","error-message":"Unexpected-exception","error-info":"java.util.concurrent.ExecutionException: TransactionCommitFailedException</p> {message=canCommit encountered an unexpected failure, errorList=[RpcError [message=canCommit encountered an unexpected failure, severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=org.opendaylight.controller.cluster.access.client.RequestTimeoutException: Timed out after 144970935433ns]]} <p>\n\tat com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)\n\tat com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:272)\n\tat com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:96)\n\tat org.opendaylight.controller.clustering.it.provider.impl.WriteTransactionsHandler.maybeFinish(WriteTransactionsHandler.java:238)\n\tat org.opendaylight.controller.clustering.it.provider.impl.WriteTransactionsHandler.run(WriteTransactionsHandler.java:112)\n\tat </p> <p><span class="error">&#91;6&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/694/archives/log.html.gz#s1-s28-t1-k2-k25-k1-k2-k3-k1-k1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/694/archives/log.html.gz#s1-s28-t1-k2-k25-k1-k2-k3-k1-k1</a></p> <p>Vratko, please add comment when you have new run, thank you</p> <p>This is still happening. In a recent Sandbox test <span class="error">&#91;7&#93;</span> (which failed to upload archive with logs) one writer failed with "TimeoutException: Timeout waiting for task" (as opposed to RequestTimeoutException), one finished successfully, and one overstepped the generous (440s) timeout for Restconf response.</p> <p>This was the brief isolation scenario (module-based shards, tell-based protocol), with <span class="error">&#91;8&#93;</span> codebase.</p> <p><span class="error">&#91;7&#93;</span> <a href="https://jenkins.opendaylight.org/sandbox/job/controller-csit-3node-clustering-only-carbon/6/robot/report/log.html#s1-s10-t1-k2-k25-k1-k1" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/sandbox/job/controller-csit-3node-clustering-only-carbon/6/robot/report/log.html#s1-s10-t1-k2-k25-k1-k1</a><br/> <span class="error">&#91;8&#93;</span> <a href="https://git.opendaylight.org/gerrit/#/c/57698/3" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/57698/3</a></p> <p>2017-06-01 23:37:21,958 | INFO | h for user karaf | command | 265 - org.apache.karaf.log.command - 3.0.8 | ROBOT MESSAGE: Starting test Local_Leader_Shutdown<br/> 2017-06-01 23:37:27,553 | INFO | qtp10706451-296 | ClusterAdminRpcService | 201 - org.opendaylight.controller.sal-cluster-admin-impl - 1.5.1.SNAPSHOT | Removing replica for shard default memberName member-2, datastoreType Config<br/> 2017-06-01 23:37:27,569 | INFO | lt-dispatcher-19 | ClusterAdminRpcService | 201 - org.opendaylight.controller.sal-cluster-admin-impl - 1.5.1.SNAPSHOT | Successfully removed replica for shard default<br/> 2017-06-01 23:37:36,439 | INFO | ult-dispatcher-4 | tActorServerConfigurationSupport | 193 - org.opendaylight.controller.sal-akka-raft - 1.5.1.SNAPSHOT | member-2-shard-default-config: ServerConfigurationPayload [serverConfig=[ServerInfo <span class="error">&#91;id=member-1-shard-default-config, isVoting=true&#93;</span>, ServerInfo <span class="error">&#91;id=member-3-shard-default-config, isVoting=true&#93;</span>]] has been successfully replicated to a majority of followers<br/> 2017-06-01 23:37:36,445 | INFO | ult-dispatcher-4 | ShardManager | 199 - org.opendaylight.controller.sal-distributed-datastore - 1.5.1.SNAPSHOT | shard-manager-config: Received LeaderStateChanged message: LeaderStateChanged <span class="error">&#91;memberId=member-2-shard-default-config, leaderId=null, leaderPayloadVersion=5&#93;</span><br/> 2017-06-01 23:37:38,659 | INFO | lt-dispatcher-32 | ClientActorBehavior | 197 - org.opendaylight.controller.cds-access-client - 1.1.1.SNAPSHOT | member-2-frontend-datastore-config: connection ConnectedClientConnection{client=ClientIdentifier</p> {frontend=member-2-frontend-datastore-config, generation=0}, cookie=0, backend=ShardBackendInfo{actor=Actor<a href="#-579249557" target="_blank" rel="noopener">akka://opendaylight-cluster-data/user/shardmanager-config/member-2-shard-default-config#-579249557</a>, sessionId=0, version=BORON, maxMessages=1000, cookie=0, shard=default, dataTree present=true}} indicated no leadership, reconnecting it<br/> org.opendaylight.controller.cluster.access.commands.NotLeaderException: Actor Actor<a href="#-579249557" target="_blank" rel="noopener">akka://opendaylight-cluster-data/user/shardmanager-config/member-2-shard-default-config#-579249557</a> is not the current leader<br/> at org.opendaylight.controller.cluster.datastore.Shard.handleRequest(Shard.java:455)<span class="error">&#91;199:org.opendaylight.controller.sal-distributed-datastore:1.5.1.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.cluster.datastore.Shard.handleNonRaftCommand(Shard.java:293)<span class="error">&#91;199:org.opendaylight.controller.sal-distributed-datastore:1.5.1.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.cluster.raft.RaftActor.handleCommand(RaftActor.java:270)<span class="error">&#91;193:org.opendaylight.controller.sal-akka-raft:1.5.1.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.cluster.common.actor.AbstractUntypedPersistentActor.onReceiveCommand(AbstractUntypedPersistentActor.java:31)<span class="error">&#91;192:org.opendaylight.controller.sal-clustering-commons:1.5.1.SNAPSHOT&#93;</span><br/> at akka.persistence.UntypedPersistentActor.onReceive(PersistentActor.scala:170)<span class="error">&#91;180:com.typesafe.akka.persistence:2.4.18&#93;</span><br/> at org.opendaylight.controller.cluster.common.actor.MeteringBehavior.apply(MeteringBehavior.java:104)<span class="error">&#91;192:org.opendaylight.controller.sal-clustering-commons:1.5.1.SNAPSHOT&#93;</span><br/> at akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:544)<span class="error">&#91;173:com.typesafe.akka.actor:2.4.18&#93;</span><br/> at akka.actor.Actor$class.aroundReceive(Actor.scala:502)<span class="error">&#91;173:com.typesafe.akka.actor:2.4.18&#93;</span><br/> at akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(PersistentActor.scala:168)<span class="error">&#91;180:com.typesafe.akka.persistence:2.4.18&#93;</span><br/> at akka.persistence.Eventsourced$$anon$1.stateReceive(Eventsourced.scala:727)<span class="error">&#91;180:com.typesafe.akka.persistence:2.4.18&#93;</span><br/> at akka.persistence.Eventsourced$class.aroundReceive(Eventsourced.scala:183)<span class="error">&#91;180:com.typesafe.akka.persistence:2.4.18&#93;</span><br/> at akka.persistence.UntypedPersistentActor.aroundReceive(PersistentActor.scala:168)<span class="error">&#91;180:com.typesafe.akka.persistence:2.4.18&#93;</span><br/> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)<span class="error">&#91;173:com.typesafe.akka.actor:2.4.18&#93;</span><br/> at akka.actor.ActorCell.invoke(ActorCell.scala:495)<span class="error">&#91;173:com.typesafe.akka.actor:2.4.18&#93;</span><br/> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)<span class="error">&#91;173:com.typesafe.akka.actor:2.4.18&#93;</span><br/> at akka.dispatch.Mailbox.run(Mailbox.scala:224)<span class="error">&#91;173:com.typesafe.akka.actor:2.4.18&#93;</span><br/> at akka.dispatch.Mailbox.exec(Mailbox.scala:234)<span class="error">&#91;173:com.typesafe.akka.actor:2.4.18&#93;</span><br/> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)<span class="error">&#91;169:org.scala-lang.scala-library:2.11.11.v20170413-090219-8a413ba7cc&#93;</span><br/> at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)<span class="error">&#91;169:org.scala-lang.scala-library:2.11.11.v20170413-090219-8a413ba7cc&#93;</span><br/> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)<span class="error">&#91;169:org.scala-lang.scala-library:2.11.11.v20170413-090219-8a413ba7cc&#93;</span><br/> at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)<span class="error">&#91;169:org.scala-lang.scala-library:2.11.11.v20170413-090219-8a413ba7cc&#93;</span><br/> 2017-06-01 23:37:38,662 | INFO | lt-dispatcher-32 | ClientActorBehavior | 197 - org.opendaylight.controller.cds-access-client - 1.1.1.SNAPSHOT | member-2-frontend-datastore-config: connection ConnectedClientConnection{client=ClientIdentifier{frontend=member-2-frontend-datastore-config, generation=0} <p>, cookie=0, backend=ShardBackendInfo{actor=Actor<a href="#-579249557" target="_blank" rel="noopener">akka://opendaylight-cluster-data/user/shardmanager-config/member-2-shard-default-config#-579249557</a>, sessionId=0, version=BORON, maxMessages=1000, cookie=0, shard=default, dataTree present=true}} reconnecting as ReconnectingClientConnection{client=ClientIdentifier</p> {frontend=member-2-frontend-datastore-config, generation=0}, cookie=0, backend=ShardBackendInfo{actor=Actor<a href="#-579249557" target="_blank" rel="noopener">akka://opendaylight-cluster-data/user/shardmanager-config/member-2-shard-default-config#-579249557</a>, sessionId=0, version=BORON, maxMessages=1000, cookie=0, shard=default, dataTree present=true}}<br/> 2017-06-01 23:37:38,662 | INFO | lt-dispatcher-32 | ClientActorBehavior | 197 - org.opendaylight.controller.cds-access-client - 1.1.1.SNAPSHOT | member-2-frontend-datastore-config: refreshing backend for shard 0<br/> 2017-06-01 23:37:38,703 | WARN | monPool-worker-2 | AbstractShardBackendResolver | 199 - org.opendaylight.controller.sal-distributed-datastore - 1.5.1.SNAPSHOT | Failed to resolve shard<br/> java.util.concurrent.TimeoutException: Connection attempt failed<br/> at org.opendaylight.controller.cluster.databroker.actors.dds.AbstractShardBackendResolver.wrap(AbstractShardBackendResolver.java:129)<span class="error">&#91;199:org.opendaylight.controller.sal-distributed-datastore:1.5.1.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.cluster.databroker.actors.dds.AbstractShardBackendResolver.lambda$connectShard$2(AbstractShardBackendResolver.java:142)<span class="error">&#91;199:org.opendaylight.controller.sal-distributed-datastore:1.5.1.SNAPSHOT&#93;</span><br/> at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)<span class="error">&#91;:1.8.0_121&#93;</span><br/> at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)<span class="error">&#91;:1.8.0_121&#93;</span><br/> at java.util.concurrent.CompletableFuture$Completion.exec(CompletableFuture.java:443)<span class="error">&#91;:1.8.0_121&#93;</span><br/> at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)<span class="error">&#91;:1.8.0_121&#93;</span><br/> at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)<span class="error">&#91;:1.8.0_121&#93;</span><br/> at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)<span class="error">&#91;:1.8.0_121&#93;</span><br/> at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)<span class="error">&#91;:1.8.0_121&#93;</span><br/> Caused by: org.opendaylight.controller.cluster.access.commands.NotLeaderException: Actor Actor<a href="#-579249557" target="_blank" rel="noopener">akka://opendaylight-cluster-data/user/shardmanager-config/member-2-shard-default-config#-579249557</a> is not the current leader<br/> at org.opendaylight.controller.cluster.datastore.Shard.handleConnectClient(Shard.java:436)<span class="error">&#91;199:org.opendaylight.controller.sal-distributed-datastore:1.5.1.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.cluster.datastore.Shard.handleNonRaftCommand(Shard.java:306)<span class="error">&#91;199:org.opendaylight.controller.sal-distributed-datastore:1.5.1.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.cluster.raft.RaftActor.handleCommand(RaftActor.java:270)<span class="error">&#91;193:org.opendaylight.controller.sal-akka-raft:1.5.1.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.cluster.common.actor.AbstractUntypedPersistentActor.onReceiveCommand(AbstractUntypedPersistentActor.java:31)<span class="error">&#91;192:org.opendaylight.controller.sal-clustering-commons:1.5.1.SNAPSHOT&#93;</span><br/> at akka.persistence.UntypedPersistentActor.onReceive(PersistentActor.scala:170)<span class="error">&#91;180:com.typesafe.akka.persistence:2.4.18&#93;</span><br/> at org.opendaylight.controller.cluster.common.actor.MeteringBehavior.apply(MeteringBehavior.java:104)<span class="error">&#91;192:org.opendaylight.controller.sal-clustering-commons:1.5.1.SNAPSHOT&#93;</span><br/> at akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:544)<span class="error">&#91;173:com.typesafe.akka.actor:2.4.18&#93;</span><br/> at akka.actor.Actor$class.aroundReceive(Actor.scala:502)<span class="error">&#91;173:com.typesafe.akka.actor:2.4.18&#93;</span><br/> at akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(PersistentActor.scala:168)<span class="error">&#91;180:com.typesafe.akka.persistence:2.4.18&#93;</span><br/> at akka.persistence.Eventsourced$$anon$1.stateReceive(Eventsourced.scala:727)<span class="error">&#91;180:com.typesafe.akka.persistence:2.4.18&#93;</span><br/> at akka.persistence.Eventsourced$class.aroundReceive(Eventsourced.scala:183)<span class="error">&#91;180:com.typesafe.akka.persistence:2.4.18&#93;</span><br/> at akka.persistence.UntypedPersistentActor.aroundReceive(PersistentActor.scala:168)<span class="error">&#91;180:com.typesafe.akka.persistence:2.4.18&#93;</span><br/> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)<span class="error">&#91;173:com.typesafe.akka.actor:2.4.18&#93;</span><br/> at akka.actor.ActorCell.invoke(ActorCell.scala:495)<span class="error">&#91;173:com.typesafe.akka.actor:2.4.18&#93;</span><br/> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)<span class="error">&#91;173:com.typesafe.akka.actor:2.4.18&#93;</span><br/> at akka.dispatch.Mailbox.run(Mailbox.scala:224)<span class="error">&#91;173:com.typesafe.akka.actor:2.4.18&#93;</span><br/> at akka.dispatch.Mailbox.exec(Mailbox.scala:234)<span class="error">&#91;173:com.typesafe.akka.actor:2.4.18&#93;</span><br/> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)<span class="error">&#91;169:org.scala-lang.scala-library:2.11.11.v20170413-090219-8a413ba7cc&#93;</span><br/> at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)<span class="error">&#91;169:org.scala-lang.scala-library:2.11.11.v20170413-090219-8a413ba7cc&#93;</span><br/> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)<span class="error">&#91;169:org.scala-lang.scala-library:2.11.11.v20170413-090219-8a413ba7cc&#93;</span><br/> at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)<span class="error">&#91;169:org.scala-lang.scala-library:2.11.11.v20170413-090219-8a413ba7cc&#93;</span><br/> 2017-06-01 23:37:39,688 | INFO | lt-dispatcher-23 | Shard | 192 - org.opendaylight.controller.sal-clustering-commons - 1.5.1.SNAPSHOT | member-2-shard-default-config (Leader): Term 3 in "AppendEntriesReply <span class="error">&#91;term=3, success=false, followerId=member-3-shard-default-config, logLastIndex=1047, logLastTerm=2, forceInstallSnapshot=false, payloadVersion=5, raftVersion=3&#93;</span>" message is greater than leader's term 2 - switching to Follower<br/> 2017-06-01 23:37:39,689 | INFO | lt-dispatcher-23 | Shard | 192 - org.opendaylight.controller.sal-clustering-commons - 1.5.1.SNAPSHOT | member-2-shard-default-config (Leader) :- Switching from behavior Leader to Follower, election term: 3<br/> 2017-06-01 23:37:39,689 | INFO | lt-dispatcher-20 | ShardManager | 199 - org.opendaylight.controller.sal-distributed-datastore - 1.5.1.SNAPSHOT | shard-manager-config: Received LeaderStateChanged message: LeaderStateChanged <span class="error">&#91;memberId=member-2-shard-default-config, leaderId=null, leaderPayloadVersion=-1&#93;</span><br/> 2017-06-01 23:37:39,689 | INFO | ult-dispatcher-4 | RoleChangeNotifier | 192 - org.opendaylight.controller.sal-clustering-commons - 1.5.1.SNAPSHOT | RoleChangeNotifier for member-2-shard-default-config , received role change from Leader to Follower<br/> 2017-06-01 23:37:39,689 | INFO | ult-dispatcher-4 | ShardManager | 199 - org.opendaylight.controller.sal-distributed-datastore - 1.5.1.SNAPSHOT | shard-manager-config: Received role changed for member-2-shard-default-config from Leader to Follower<br/> 2017-06-01 23:37:40,765 | INFO | lt-dispatcher-22 | aftActorLeadershipTransferCohort | 193 - org.opendaylight.controller.sal-akka-raft - 1.5.1.SNAPSHOT | member-2-shard-default-config: Successfully transferred leadership to null in 4.320 s<br/> 2017-06-01 23:37:40,768 | INFO | ult-dispatcher-3 | Shard | 192 - org.opendaylight.controller.sal-clustering-commons - 1.5.1.SNAPSHOT | Stopping Shard member-2-shard-default-config<br/> 2017-06-01 23:37:50,032 | INFO | lt-dispatcher-32 | AbstractClientConnection | 197 - org.opendaylight.controller.cds-access-client - 1.1.1.SNAPSHOT | Capping ReconnectingClientConnection{client=ClientIdentifier{frontend=member-2-frontend-datastore-config, generation=0} <p>, cookie=0, backend=ShardBackendInfo{actor=Actor<a href="#-579249557" target="_blank" rel="noopener">akka://opendaylight-cluster-data/user/shardmanager-config/member-2-shard-default-config#-579249557</a>, sessionId=0, version=BORON, maxMessages=1000, cookie=0, shard=default, dataTree present=true}} throttle delay from 6 to 5 seconds<br/> 2017-06-01 23:40:15,167 | ERROR | pool-30-thread-1 | WriteTransactionsHandler | 256 - org.opendaylight.controller.samples.clustering-it-provider - 1.5.1.SNAPSHOT | Write transactions failed.<br/> java.util.concurrent.TimeoutException: Timeout waiting for task.<br/> at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:269)<span class="error">&#91;57:com.google.guava:18.0.0&#93;</span><br/> at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:96)<span class="error">&#91;57:com.google.guava:18.0.0&#93;</span><br/> at org.opendaylight.controller.clustering.it.provider.impl.WriteTransactionsHandler.maybeFinish(WriteTransactionsHandler.java:238)<span class="error">&#91;256:org.opendaylight.controller.samples.clustering-it-provider:1.5.1.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.clustering.it.provider.impl.WriteTransactionsHandler.run(WriteTransactionsHandler.java:112)<span class="error">&#91;256:org.opendaylight.controller.samples.clustering-it-provider:1.5.1.SNAPSHOT&#93;</span><br/> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)<span class="error">&#91;:1.8.0_121&#93;</span><br/> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)<span class="error">&#91;:1.8.0_121&#93;</span><br/> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)<span class="error">&#91;:1.8.0_121&#93;</span><br/> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)<span class="error">&#91;:1.8.0_121&#93;</span><br/> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)<span class="error">&#91;:1.8.0_121&#93;</span><br/> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)<span class="error">&#91;:1.8.0_121&#93;</span><br/> at java.lang.Thread.run(Thread.java:745)<span class="error">&#91;:1.8.0_121&#93;</span><br/> 2017-06-01 23:40:15,367 | INFO | qtp10706451-308 | ClusterAdminRpcService | 201 - org.opendaylight.controller.sal-cluster-admin-impl - 1.5.1.SNAPSHOT | Adding replica for shard default</p> <p>The karaf log segment above is from Sandbox running against 57996/5 with cherry-picked 57699/5.</p> <p>The scenario uses a module-based shard (default-config) with tell-based protocol. The transaction writer is started on the leader member, then remove-shard-replica is called. A similar test case (writer on a follower, still calling remove-shard-replica on the leader) passed (the writer finished without reporting any failure).</p> <p>Similar failure happened on recent RelEng run <span class="error">&#91;9&#93;</span>, but karaf.log <span class="error">&#91;10&#93;</span> segment between 13:53:54,938 and 13:56:47,140 shows UnreachableMember happened.</p> <p><span class="error">&#91;9&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/733/log.html.gz#s1-s20-t1-k2-k10-k1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/733/log.html.gz#s1-s20-t1-k2-k10-k1</a><br/> <span class="error">&#91;10&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/733/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/733/odl1_karaf.log.gz</a></p> <p>This symptom re-appeared, it now happens <span class="error">&#91;11&#93;</span> consistently but only in one test case. It tests module-based shards (tell based protocol) with single writer started on a follower, but then cluster-admin:make-leader-local attempts to move the leader to the writer member. The test expects writer to finish writing successfully.</p> <p>The full writer response:<br/> {"errors":{"error":[</p> {"error-type":"application","error-tag":"operation-failed","error-message":"Unexpected-exception","error-info":"java.util.concurrent.TimeoutException: Timeout waiting for task.\n\tat com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:269)\n\tat com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:96)\n\tat org.opendaylight.controller.clustering.it.provider.impl.WriteTransactionsHandler.maybeFinish(WriteTransactionsHandler.java:234)\n\tat org.opendaylight.controller.clustering.it.provider.impl.WriteTransactionsHandler.run(WriteTransactionsHandler.java:112)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n\tat java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\tat java.lang.Thread.run(Thread.java:748)\n"} <p>]}}</p> <p><span class="error">&#91;11&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/750/log.html.gz#s1-s24-t5-k2-k10" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/750/log.html.gz#s1-s24-t5-k2-k10</a></p> <p>&gt; consistently but only in one test case</p> <p>Now I would say 50% of the time, still in only one <span class="error">&#91;12&#93;</span> test case.<br/> The response message is now more specific: "Final submit was timed out by the test provider or was interrupted".</p> <p>The writer took 160 seconds to respond.</p> <p>Few suspicious lines from writer member karaf.log <span class="error">&#91;13&#93;</span> since 07:45:22,131 follow. The first one is especially confusing, since the connection is now reconnected to the new (local, member-2) leader, but it is still reporting NotLeaderException from the previous (remote, member-1) leader.</p> <p>2017-06-21 07:45:38,519 | INFO | lt-dispatcher-20 | ClientActorBehavior | 197 - org.opendaylight.controller.cds-access-client - 1.1.1.SNAPSHOT | member-2-frontend-datastore-config: connection ConnectedClientConnection{client=ClientIdentifier</p> {frontend=member-2-frontend-datastore-config, generation=0} <p>, cookie=0, backend=ShardBackendInfo{actor=Actor<a href="#494504499" target="_blank" rel="noopener">akka://opendaylight-cluster-data/user/shardmanager-config/member-2-shard-default-config#494504499</a>, sessionId=1, version=BORON, maxMessages=1000, cookie=0, shard=default, dataTree present=true}} indicated no leadership, reconnecting it<br/> org.opendaylight.controller.cluster.access.commands.NotLeaderException: Actor Actor<a href="#-1667137708" target="_blank" rel="noopener">akka://opendaylight-cluster-data/user/shardmanager-config/member-1-shard-default-config#-1667137708</a> is not the current leader</p> <p>2017-06-21 07:45:38,624 | ERROR | lt-dispatcher-22 | OneForOneStrategy | 174 - com.typesafe.akka.slf4j - 2.4.18 | Transaction member-2-datastore-config-fe-0-chn-4-txn-5365-0 has already been sealed<br/> java.lang.IllegalStateException: Transaction member-2-datastore-config-fe-0-chn-4-txn-5365-0 has already been sealed<br/> at com.google.common.base.Preconditions.checkState(Preconditions.java:197)<span class="error">&#91;57:com.google.guava:18.0.0&#93;</span><br/> at org.opendaylight.controller.cluster.databroker.actors.dds.AbstractProxyTransaction.checkNotSealed(AbstractProxyTransaction.java:363)<span class="error">&#91;199:org.opendaylight.controller.sal-distributed-datastore:1.5.1.SNAPSHOT&#93;</span></p> <p>2017-06-21 07:45:38,657 | INFO | ult-dispatcher-5 | ClientActorBehavior | 197 - org.opendaylight.controller.cds-access-client - 1.1.1.SNAPSHOT | member-2-frontend-datastore-config: Ignoring unknown response Envelope{sessionId=1, txSequence=1, message=TransactionPurgeResponse{target=member-2-datastore-config-fe-0-chn-4-txn-5346-0, sequence=0}}</p> <p>2017-06-21 07:48:18,136 | WARN | pool-30-thread-1 | WriteTransactionsHandler | 257 - org.opendaylight.controller.samples.clustering-it-provider - 1.5.1.SNAPSHOT | Future #5362/7971 not completed yet</p> <p><span class="error">&#91;12&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/756/log.html.gz#s1-s24-t5-k2-k10" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/756/log.html.gz#s1-s24-t5-k2-k10</a><br/> <span class="error">&#91;13&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/756/odl2_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/756/odl2_karaf.log.gz</a></p> <p>&gt; the connection is now reconnected to the new (local, member-2) leader,<br/> &gt; but it is still reporting NotLeaderException from the previous (remote, member-1) leader.</p> <p>Comparing log messages to code, I think that is a case of multiple FailureEnvelope from the same old backend.</p> <p>Specifically, this <span class="error">&#91;14&#93;</span> line confirms the connection to the new backend is established, but then this <span class="error">&#91;15&#93;</span> line says the repeated message from the old backend is causing reconnect on the new connection.</p> <p>In karaf.log <span class="error">&#91;16&#93;</span> both log messages occur at 09:14:37,757.</p> <p>In theory, spurious reconnects should not cause system test failures, so there is something to fix in other parts of the code as well.<br/> But limiting reconnects (for example by having sessionId in FailureEnvelope and comparing it to the current value) would be helpful.</p> <p>A cosmetic remark: ReconnectingClientConnection probably only needs to know maxMessages value; printing the whole ShardBackendInfo pointing to the old backend is slightly confusing, as that is not the backend ODL is likely to connect to.</p> <p><span class="error">&#91;14&#93;</span> <a href="https://git.opendaylight.org/gerrit/gitweb?p=controller.git;a=blob;f=opendaylight/md-sal/cds-access-client/src/main/java/org/opendaylight/controller/cluster/access/client/ClientActorBehavior.java;h=554ffe97c77ae7c3011daca2a28c22f660a20e78;hb=refs/heads/stable/carbon#l307" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/gitweb?p=controller.git;a=blob;f=opendaylight/md-sal/cds-access-client/src/main/java/org/opendaylight/controller/cluster/access/client/ClientActorBehavior.java;h=554ffe97c77ae7c3011daca2a28c22f660a20e78;hb=refs/heads/stable/carbon#l307</a><br/> <span class="error">&#91;15&#93;</span> <a href="https://git.opendaylight.org/gerrit/gitweb?p=controller.git;a=blob;f=opendaylight/md-sal/cds-access-client/src/main/java/org/opendaylight/controller/cluster/access/client/ClientActorBehavior.java;h=554ffe97c77ae7c3011daca2a28c22f660a20e78;hb=refs/heads/stable/carbon#l177" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/gitweb?p=controller.git;a=blob;f=opendaylight/md-sal/cds-access-client/src/main/java/org/opendaylight/controller/cluster/access/client/ClientActorBehavior.java;h=554ffe97c77ae7c3011daca2a28c22f660a20e78;hb=refs/heads/stable/carbon#l177</a><br/> <span class="error">&#91;16&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/761/odl2_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/761/odl2_karaf.log.gz</a></p> <p><a href="https://git.opendaylight.org/gerrit/59527" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/59527</a></p> <p>After rework of transaction writer, the error message <span class="error">&#91;17&#93;</span> is different, but I believe it is the same bug. Now it happened on releng when the old leader got remove-shard-replica, and the single writer was on the member which stayed follower.</p> <p>The new writer response:<br/> {"errors":{"error":[</p> {"error-type":"application","error-tag":"operation-failed","error-message":"Final submit was timed out by the test provider or was interrupted","error-info":"java.util.concurrent.TimeoutException: Collection did not finish in 300 seconds\n\tat org.opendaylight.controller.clustering.it.provider.impl.AbstractTransactionHandler.checkComplete(AbstractTransactionHandler.java:187)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\tat java.lang.Thread.run(Thread.java:748)\n"} <p>]}}</p> <p>The karaf.log <span class="error">&#91;18&#93;</span> reports 2011 futures as not completed, nothing else is visible on INFO level.</p> <p><span class="error">&#91;17&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/765/log.html.gz#s1-s20-t3-k2-k8" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/765/log.html.gz#s1-s20-t3-k2-k8</a><br/> <span class="error">&#91;18&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/765/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-clustering-only-carbon/765/odl1_karaf.log.gz</a></p> Blocks CONTROLLER-1668 Development External issue ID 8445 External issue URL Issue Type ODL SR Target Milestone Rank 0|i02s13: