https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-1333 controller <p>After connecting BGP to a feed, it timeouts after about 3 minutes with ask.timeout exceptions. Build is: distribution-karaf-0.3.0-20150522.085838-1738. I'll send steps to reproduce in an email. Here's the exception:</p> <p>2015-05-22 07:48:52,740 | INFO | oupCloseable-3-1 | AbstractBGPSessionNegotiator | 227 - org.opendaylight.bgpcep.bgp-rib-impl - 0.4.0.SNAPSHOT | BGP Session established successfully.<br/> 2015-05-22 07:53:15,620 | ERROR | lt-dispatcher-39 | LocalThreePhaseCommitCohort | 169 - org.opendaylight.controller.sal-distributed-datastore - 1.2.0.SNAPSHOT | Failed to prepare transaction member-1-txn-6043 on backend<br/> akka.pattern.AskTimeoutException: Ask timed out on <a href="#-1012054239)]" target="_blank" rel="noopener">ActorSelection[Anchor(akka://opendaylight-cluster-data/), Path(/user/shardmanager-operational/member-1-shard-default-operational#-1012054239)]</a> after <span class="error">&#91;30000 ms&#93;</span><br/> at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:334)<span class="error">&#91;154:com.typesafe.akka.actor:2.3.10&#93;</span><br/> at akka.actor.Scheduler$$anon$7.run(Scheduler.scala:117)<span class="error">&#91;154:com.typesafe.akka.actor:2.3.10&#93;</span><br/> at scala.concurrent.Future$InternalCallbackExecutor$.scala$concurrent$Future$InternalCallbackExecutor$$unbatchedExecute(Future.scala:694)<span class="error">&#91;151:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b&#93;</span><br/> at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:691)<span class="error">&#91;151:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b&#93;</span><br/> at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(Scheduler.scala:467)<span class="error">&#91;154:com.typesafe.akka.actor:2.3.10&#93;</span><br/> at akka.actor.LightArrayRevolverScheduler$$anon$8.executeBucket$1(Scheduler.scala:419)<span class="error">&#91;154:com.typesafe.akka.actor:2.3.10&#93;</span><br/> at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:423)<span class="error">&#91;154:com.typesafe.akka.actor:2.3.10&#93;</span><br/> at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)<span class="error">&#91;154:com.typesafe.akka.actor:2.3.10&#93;</span><br/> at java.lang.Thread.run(Thread.java:745)<span class="error">&#91;:1.7.0_79&#93;</span><br/> 2015-05-22 07:53:15,621 | ERROR | lt-dispatcher-38 | LocalThreePhaseCommitCohort | 169 - org.opendaylight.controller.sal-distributed-datastore - 1.2.0.SNAPSHOT | Failed to prepare transaction member-1-txn-4520 on backend<br/> akka.pattern.AskTimeoutException: Ask timed out on <a href="#-1012054239)]" target="_blank" rel="noopener">ActorSelection[Anchor(akka://opendaylight-cluster-data/), Path(/user/shardmanager-operational/member-1-shard-default-operational#-1012054239)]</a> after <span class="error">&#91;30000 ms&#93;</span><br/> at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:334)<span class="error">&#91;154:com.typesafe.akka.actor:2.3.10&#93;</span><br/> at akka.actor.Scheduler$$anon$7.run(Scheduler.scala:117)<span class="error">&#91;154:com.typesafe.akka.actor:2.3.10&#93;</span><br/> at scala.concurrent.Future$InternalCallbackExecutor$.scala$concurrent$Future$InternalCallbackExecutor$$unbatchedExecute(Future.scala:694)<span class="error">&#91;151:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b&#93;</span><br/> at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:691)<span class="error">&#91;151:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b&#93;</span><br/> at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(Scheduler.scala:467)<span class="error">&#91;154:com.typesafe.akka.actor:2.3.10&#93;</span><br/> at akka.actor.LightArrayRevolverScheduler$$anon$8.executeBucket$1(Scheduler.scala:419)<span class="error">&#91;154:com.typesafe.akka.actor:2.3.10&#93;</span><br/> at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:423)<span class="error">&#91;154:com.typesafe.akka.actor:2.3.10&#93;</span><br/> at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)<span class="error">&#91;154:com.typesafe.akka.actor:2.3.10&#93;</span><br/> at java.lang.Thread.run(Thread.java:745)<span class="error">&#91;:1.7.0_79&#93;</span><br/> 2015-05-22 07:53:15,622 | WARN | lt-dispatcher-23 | ConcurrentDOMDataBroker | 169 - org.opendaylight.controller.sal-distributed-datastore - 1.2.0.SNAPSHOT | Tx: DOM-CHAIN-6-2369 Error during phase CAN_COMMIT, starting Abort<br/> akka.pattern.AskTimeoutException: Ask timed out on <a href="#-1012054239)]" target="_blank" rel="noopener">ActorSelection[Anchor(akka://opendaylight-cluster-data/), Path(/user/shardmanager-operational/member-1-shard-default-operational#-1012054239)]</a> after <span class="error">&#91;30000 ms&#93;</span><br/> at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:334)<span class="error">&#91;154:com.typesafe.akka.actor:2.3.10&#93;</span><br/> at akka.actor.Scheduler$$anon$7.run(Scheduler.scala:117)<span class="error">&#91;154:com.typesafe.akka.actor:2.3.10&#93;</span><br/> at scala.concurrent.Future$InternalCallbackExecutor$.scala$concurrent$Future$InternalCallbackExecutor$$unbatchedExecute(Future.scala:694)<span class="error">&#91;151:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b&#93;</span><br/> at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:691)<span class="error">&#91;151:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b&#93;</span><br/> at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(Scheduler.scala:467)<span class="error">&#91;154:com.typesafe.akka.actor:2.3.10&#93;</span><br/> at akka.actor.LightArrayRevolverScheduler$$anon$8.executeBucket$1(Scheduler.scala:419)<span class="error">&#91;154:com.typesafe.akka.actor:2.3.10&#93;</span><br/> at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:423)<span class="error">&#91;154:com.typesafe.akka.actor:2.3.10&#93;</span><br/> at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)<span class="error">&#91;154:com.typesafe.akka.actor:2.3.10&#93;</span><br/> at java.lang.Thread.run(Thread.java:745)<span class="error">&#91;:1.7.0_79&#93;</span></p> <p>Operating System: All<br/> Platform: All</p> CONTROLLER-1333

Clustering : Performance issues in BGP scale test

Bug Resolved Done Moiz Raja Dana Kutenicsova Fri, 22 May 2015 12:28:31 +0000 Wed, 10 Jun 2015 14:50:08 +0000 Wed, 10 Jun 2015 14:50:08 +0000 Post-Helium clustering 0 4 <p>I enabled debug logging, grepped for YimeoutException and then grepped on some of the transactions that were failing. I see a pattern (which I cannot explain yet). Most of the transactions that fail with the timeout happen because "FinfPrimaryShard" fails with a timeout.</p> <p>See this snippet -&gt; <a href="http://pastebin.com/yBh2nrSs" class="external-link" target="_blank" rel="nofollow noopener">http://pastebin.com/yBh2nrSs</a></p> <p>A couple of curious things,</p> <p>a. The AkkaTimeout shows the actor to be the shard<br/> b. The timeout is 30 seconds which is the same as the commit timeout</p> <p>This does not make sense because the find primary request is sent to the ShardManager which is the parent of Shard.</p> <p>I reran the test and now I see the timeouts to be 5 seconds with the actor still being the shard - puzzling - I'm wondering if there is some mixup happening here and the timeout is being delivered to the wrong handler.</p> <p>Based on what I am seeing I think the following patch (which was merged May 26th) will help because it avoids looking up the primary shard unless ShardManager actually removes it from the PrimaryShardInfo cache.</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/20571/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/20571/</a></p> <p>Dana, if you could try this out it would be good. I'm hitting some trouble building an integration distribution otherwise would have tried it myself.</p> <p>That is odd. What's even odder is that the ActorContext uses the shard-initialization-timeout for FindPrimary which is set to 5 min. </p> <p>Did you have ActorContext and ShardManager debug on as well? If the PrimaryShardInfo future wasn't cached yet then we should've seen see the "Sending message FindPrimary..." output from ActorContext. It seems the PrimaryShardInfo future was not yet cached b/c the "Find primary for shard default failed" output was logged from an akka dispatcher thread which indicates it was invoked via the OnComplete callback. The AbstractTransactionContextFactory first checks Future#isCompleted to elide the async OnComplete wait on the Future so if it was already completed then the output would've been logged on the "oupCloseable-3-1" thread. </p> <p>Getting back to the AskTimeoutException indicating the "default" shard actor path, a possible explanation is that the call to actorSelection in ActorSelection#onPrimaryShardFound is what threw the AskTimeoutException. However, where did the 30000 ms timeout some from? Also the "findPrimaryShardAsync received ..." debug output should've been seen (assuming you had ActorContext debug on).</p> <p>In the paste bin, the NoOpTransactionContext didn't print the AskTimeoutException stack trace - possibly b/c it was already printed earlier too many times. Hopefully that would give an indication as to the source of the ask failure.</p> <p>Since bugzilla won't let me I've uploaded the whole debug enabled log file here <a href="https://cisco.box.com/s/eesdiopbocetz2qkv4orjy3inknde7v1" class="external-link" target="_blank" rel="nofollow noopener">https://cisco.box.com/s/eesdiopbocetz2qkv4orjy3inknde7v1</a>. </p> <p>Note that in this log you will see Akka timeouts happening at 5 seconds but the actor is still the Shard.</p> <p>Notice the time interval between these 2 log entries for tx member-1-txn-4668:</p> <p>2015-05-26 21:11:39,003 | DEBUG | oupCloseable-3-1 | SingleCommitCohortProxy | 169 - org.opendaylight.controller.sal-distributed-datastore - 1.2.0.SNAPSHOT | Tx member-1-txn-4668 canCommit<br/> 2015-05-26 21:18:51,324 | DEBUG | ult-dispatcher-3 | bstractTransactionContextFactory | 169 - org.opendaylight.controller.sal-distributed-datastore - 1.2.0.SNAPSHOT | Tx member-1-txn-4668: Find primary for shard default failed</p> <p>Over 7 minutes passed between the time it was created and readied by the caller and the time it failed to find the primary shard.</p> <p>The ActorContext sent out 2 FindPrimary messages ~9 sec after the tx was created:</p> <p>2015-05-26 21:11:48,516 | DEBUG | lt-dispatcher-39 | ActorContext | 169 - org.opendaylight.controller.sal-distributed-datastore - 1.2.0.SNAPSHOT | Sending message class org.opendaylight.controller.cluster.datastore.messages.FindPrimary to Actor<a href="#967672581" target="_blank" rel="noopener">akka://opendaylight-cluster-data/user/shardmanager-operational#967672581</a><br/> 2015-05-26 21:11:48,517 | DEBUG | ult-dispatcher-5 | ActorContext | 169 - org.opendaylight.controller.sal-distributed-datastore - 1.2.0.SNAPSHOT | Sending message class org.opendaylight.controller.cluster.datastore.messages.FindPrimary to Actor<a href="#967672581" target="_blank" rel="noopener">akka://opendaylight-cluster-data/user/shardmanager-operational#967672581</a></p> <p>I assume 1 of them was for this tx. It appears the 9 sec gap is primarily due to waiting for the previous ready Future to complete. The requests were processed by the ShardManager and the responses came back quickly to the ActorContext:</p> <p>2015-05-26 21:11:48,517 | DEBUG | lt-dispatcher-39 | ShardManager | 169 - org.opendaylight.controller.sal-distributed-datastore - 1.2.0.SNAPSHOT | shard-manager-operational: Found primary for default: LocalPrimaryShardFound <span class="error">&#91;primaryPath=akka.tcp://opendaylight-cluster-data@127.0.0.1:2550/user/shardmanager-operational/member-1-shard-default-operational#-701003399, localShardDataTree=org.opendaylight.yangtools.yang.data.impl.schema.tree.InMemoryDataTree@58bde7ca&#93;</span><br/> 2015-05-26 21:11:48,517 | DEBUG | lt-dispatcher-39 | ShardManager | 169 - org.opendaylight.controller.sal-distributed-datastore - 1.2.0.SNAPSHOT | shard-manager-operational: Found primary for default: LocalPrimaryShardFound <span class="error">&#91;primaryPath=akka.tcp://opendaylight-cluster-data@127.0.0.1:2550/user/shardmanager-operational/member-1-shard-default-operational#-701003399, localShardDataTree=org.opendaylight.yangtools.yang.data.impl.schema.tree.InMemoryDataTree@58bde7ca&#93;</span></p> <p>2015-05-26 21:11:48,517 | DEBUG | ult-dispatcher-5 | ActorContext | 169 - org.opendaylight.controller.sal-distributed-datastore - 1.2.0.SNAPSHOT | findPrimaryShardAsync received: LocalPrimaryShardFound <span class="error">&#91;primaryPath=akka.tcp://opendaylight-cluster-data@127.0.0.1:2550/user/shardmanager-operational/member-1-shard-default-operational#-701003399, localShardDataTree=org.opendaylight.yangtools.yang.data.impl.schema.tree.InMemoryDataTree@58bde7ca&#93;</span><br/> 2015-05-26 21:11:48,517 | DEBUG | lt-dispatcher-33 | ActorContext | 169 - org.opendaylight.controller.sal-distributed-datastore - 1.2.0.SNAPSHOT | findPrimaryShardAsync received: LocalPrimaryShardFound <span class="error">&#91;primaryPath=akka.tcp://opendaylight-cluster-data@127.0.0.1:2550/user/shardmanager-operational/member-1-shard-default-operational#-701003399, localShardDataTree=org.opendaylight.yangtools.yang.data.impl.schema.tree.InMemoryDataTree@58bde7ca&#93;</span></p> <p>So the FindPrimary message itself was quick. Also it succeeded so the AskTimeoutEx didn't come from that message. So what accounts for the 7 min gap between "findPrimaryShardAsync received" and the AbstractTransactionContextFactory receiving the Future callback? There could've been a long GC pause (for that long?). Or it could've taken that long for a thread to become available and process the OnComplete callback (not likely). But neither of those account for the AskTimeoutEx. </p> <p>I suspect it was onFindPrimaryShardFound that took 7 min. The only code I see that could be the culprit are the call to actorSelection and the call to onShardInfoUpdated while holding the lock. The latter shouldn't take that long even if the lock was contended. The call to actorSelection looks suspicious. I'm not sure what it does internally - maybe it tries to "ping" the shard actor but times out and throws the AskTimeoutEx - again that would explain the presence of the shard actor path in the AskTimeoutEx. It would be interesting to put a try/catch around actorSelection.</p> <p>I agree that <a href="https://git.opendaylight.org/gerrit/#/c/20571/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/20571/</a> should help here. It should alleviate the excessive FindPrimary messages.</p> <p>Another curious thing about most of the transactions failing with AskTimeout - there were at most 1 or 2 modifications in those transactions. Why would that be when most transactions seem to have thousands?</p> <p>Tom, I don't think that actorSelection taking a long time is the root cause. Creating an actorSelection is supposed to be cheap. Using an actor selection is more expensive than an ActorRef. If we go with your theory then actorSelection would have to do an ask to the actor and wait for a response. </p> <p>Note that the timeouts we are seeing 5s/30s are the kind of timeouts we have set. </p> <p>I think there is something else going on here.</p> <p>Also note a bunch of consecutive transactions failed.</p> <p>550777:2015-05-26 21:18:51,324 | DEBUG | ult-dispatcher-3 | bstractTransactionContextFactory | 169 - org.opendaylight.controller.sal-distributed-datastore - 1.2.0.SNAPSHOT | Tx member-1-txn-4668: Find primary for shard default failed<br/> 550905:2015-05-26 21:19:01,529 | DEBUG | ult-dispatcher-3 | bstractTransactionContextFactory | 169 - org.opendaylight.controller.sal-distributed-datastore - 1.2.0.SNAPSHOT | Tx member-1-txn-4669: Find primary for shard default failed<br/> 551017:2015-05-26 21:19:01,531 | DEBUG | ult-dispatcher-3 | bstractTransactionContextFactory | 169 - org.opendaylight.controller.sal-distributed-datastore - 1.2.0.SNAPSHOT | Tx member-1-txn-4670: Find primary for shard default failed<br/> 551092:2015-05-26 21:19:04,493 | DEBUG | ult-dispatcher-3 | bstractTransactionContextFactory | 169 - org.opendaylight.controller.sal-distributed-datastore - 1.2.0.SNAPSHOT | Tx member-1-txn-4671: Find primary for shard default failed<br/> 551165:2015-05-26 21:19:04,495 | DEBUG | ult-dispatcher-3 | bstractTransactionContextFactory | 169 - org.opendaylight.controller.sal-distributed-datastore - 1.2.0.SNAPSHOT | Tx member-1-txn-4672: Find primary for shard default failed<br/> 551288:2015-05-26 21:19:09,490 | DEBUG | ult-dispatcher-3 | bstractTransactionContextFactory | 169 - org.opendaylight.controller.sal-distributed-datastore - 1.2.0.SNAPSHOT | Tx member-1-txn-4673: Find primary for shard default failed<br/> 551348:2015-05-26 21:19:09,493 | DEBUG | ult-dispatcher-3 | bstractTransactionContextFactory | 169 - org.opendaylight.controller.sal-distributed-datastore - 1.2.0.SNAPSHOT | Tx member-1-txn-4674: Find primary for shard default failed</p> <p>I tried build <a href="https://nexus.opendaylight.org/content/repositories/opendaylight.snapshot/org/opendaylight/integration/distribution-karaf/0.3.0-SNAPSHOT/distribution-karaf-0.3.0-20150527.075221-1862.tar.gz" class="external-link" target="_blank" rel="nofollow noopener">https://nexus.opendaylight.org/content/repositories/opendaylight.snapshot/org/opendaylight/integration/distribution-karaf/0.3.0-SNAPSHOT/distribution-karaf-0.3.0-20150527.075221-1862.tar.gz</a> , which should contain the change 20571. From my point of view it didn't help, the logs are in ~/27-05-fail-karaf.log</p> <p>An actorSelecton shouldn't take that long but then again nothing should <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/> But something took 7 min and the logs indicate it wasn't FindPrimary...</p> <p>I think the bunch of consecutive failed transactions is b/c they were all backed up then timed out similarly.</p> <p>I agree the 5s/30s timeouts look like ours - that part is odd. The 5s is more odd b/c that would correspond to the operation timeout but we only use that now for remote tx operations - all these txs are local.</p> <p>(In reply to Moiz Raja from comment #8)<br/> &gt; Tom, I don't think that actorSelection taking a long time is the root cause.<br/> &gt; Creating an actorSelection is supposed to be cheap. Using an actor selection<br/> &gt; is more expensive than an ActorRef. If we go with your theory then<br/> &gt; actorSelection would have to do an ask to the actor and wait for a response. <br/> &gt; <br/> &gt; Note that the timeouts we are seeing 5s/30s are the kind of timeouts we have<br/> &gt; set. <br/> &gt; <br/> &gt; I think there is something else going on here.<br/> &gt; <br/> &gt; Also note a bunch of consecutive transactions failed.<br/> &gt; <br/> &gt; 550777:2015-05-26 21:18:51,324 | DEBUG | ult-dispatcher-3 |<br/> &gt; bstractTransactionContextFactory | 169 -<br/> &gt; org.opendaylight.controller.sal-distributed-datastore - 1.2.0.SNAPSHOT | Tx<br/> &gt; member-1-txn-4668: Find primary for shard default failed<br/> &gt; 550905:2015-05-26 21:19:01,529 | DEBUG | ult-dispatcher-3 |<br/> &gt; bstractTransactionContextFactory | 169 -<br/> &gt; org.opendaylight.controller.sal-distributed-datastore - 1.2.0.SNAPSHOT | Tx<br/> &gt; member-1-txn-4669: Find primary for shard default failed<br/> &gt; 551017:2015-05-26 21:19:01,531 | DEBUG | ult-dispatcher-3 |<br/> &gt; bstractTransactionContextFactory | 169 -<br/> &gt; org.opendaylight.controller.sal-distributed-datastore - 1.2.0.SNAPSHOT | Tx<br/> &gt; member-1-txn-4670: Find primary for shard default failed<br/> &gt; 551092:2015-05-26 21:19:04,493 | DEBUG | ult-dispatcher-3 |<br/> &gt; bstractTransactionContextFactory | 169 -<br/> &gt; org.opendaylight.controller.sal-distributed-datastore - 1.2.0.SNAPSHOT | Tx<br/> &gt; member-1-txn-4671: Find primary for shard default failed<br/> &gt; 551165:2015-05-26 21:19:04,495 | DEBUG | ult-dispatcher-3 |<br/> &gt; bstractTransactionContextFactory | 169 -<br/> &gt; org.opendaylight.controller.sal-distributed-datastore - 1.2.0.SNAPSHOT | Tx<br/> &gt; member-1-txn-4672: Find primary for shard default failed<br/> &gt; 551288:2015-05-26 21:19:09,490 | DEBUG | ult-dispatcher-3 |<br/> &gt; bstractTransactionContextFactory | 169 -<br/> &gt; org.opendaylight.controller.sal-distributed-datastore - 1.2.0.SNAPSHOT | Tx<br/> &gt; member-1-txn-4673: Find primary for shard default failed<br/> &gt; 551348:2015-05-26 21:19:09,493 | DEBUG | ult-dispatcher-3 |<br/> &gt; bstractTransactionContextFactory | 169 -<br/> &gt; org.opendaylight.controller.sal-distributed-datastore - 1.2.0.SNAPSHOT | Tx<br/> &gt; member-1-txn-4674: Find primary for shard default failed</p> <p>I can't get to ~/27-05-fail-karaf.log <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/sad.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>(In reply to Dana Kutenicsova from comment #9)<br/> &gt; I tried build<br/> &gt; <a href="https://nexus.opendaylight.org/content/repositories/opendaylight.snapshot/" class="external-link" target="_blank" rel="nofollow noopener">https://nexus.opendaylight.org/content/repositories/opendaylight.snapshot/</a><br/> &gt; org/opendaylight/integration/distribution-karaf/0.3.0-SNAPSHOT/distribution-<br/> &gt; karaf-0.3.0-20150527.075221-1862.tar.gz , which should contain the change<br/> &gt; 20571. From my point of view it didn't help, the logs are in<br/> &gt; ~/27-05-fail-karaf.log</p> <p>Here is 27-05-karaf.log <a href="https://cisco.box.com/s/m5o616g8qsye2urcvr844zfim2l5ikt5" class="external-link" target="_blank" rel="nofollow noopener">https://cisco.box.com/s/m5o616g8qsye2urcvr844zfim2l5ikt5</a></p> <p>But it does not have debug logging. I'll enable that again and run the test then post the log after.</p> <p>Debug logs <a href="https://cisco.box.com/s/qbw7pexrgkesgbuqpjgtij2p7kojf324" class="external-link" target="_blank" rel="nofollow noopener">https://cisco.box.com/s/qbw7pexrgkesgbuqpjgtij2p7kojf324</a></p> <p>The default-operational shard sent the LeaderStateChanged event at :</p> <p>2015-05-27 12:20:34,187 | INFO | er-data-akka.actor.default-dispatcher-34 | ShardManager | 169 - org.opendaylight.controller.sal-distributed-datastore - 1.2.0.SNAPSHOT | shard-manager-operational: Received LeaderStateChanged message: LeaderStateChanged <span class="error">&#91;memberId=member-1-shard-default-operational, leaderId=member-1-shard-default-operational&#93;</span></p> <p>The FindPrimary message was sent and received 5 sec later</p> <p>2015-05-27 12:20:39,113 | DEBUG | er-data-akka.actor.default-dispatcher-16 | ActorContext | 169 - org.opendaylight.controller.sal-distributed-datastore - 1.2.0.SNAPSHOT | findPrimaryShardAsync received: LocalPrimaryShardFound <span class="error">&#91;primaryPath=akka.tcp://opendaylight-cluster-data@127.0.0.1:2550/user/shardmanager-operational/member-1-shard-default-operational#-1521885687, localShardDataTree=org.opendaylight.yangtools.yang.data.impl.schema.tree.InMemoryDataTree@3eb14c15&#93;</span></p> <p>In fact this is the only such log entry for default-operational, which makes sense b/c it should've been cached thereafter and there were no more LeaderStateChanged events to clear it.</p> <p>Also we see that the local data tree was resolved so it must have gotten put in the cache:</p> <p>2015-05-27 12:20:39,114 | DEBUG | er-data-akka.actor.default-dispatcher-16 | bstractTransactionContextFactory | 169 - org.opendaylight.controller.sal-distributed-datastore - 1.2.0.SNAPSHOT | Shard default resolved to local data tree - adding local factory</p> <p>But yet a tx ~35 sec later failed to find the primary shard. Notice the 11 sec gap. </p> <p>2015-05-27 12:21:14,949 | DEBUG | er-data-akka.actor.default-dispatcher-19 | SingleCommitCohortProxy | 169 - org.opendaylight.controller.sal-distributed-datastore - 1.2.0.SNAPSHOT | Tx member-1-txn-7907 canCommit<br/> 2015-05-27 12:32:15,902 | DEBUG | er-data-akka.actor.default-dispatcher-32 | bstractTransactionContextFactory | 169 - org.opendaylight.controller.sal-distributed-datastore - 1.2.0.SNAPSHOT | Tx member-1-txn-7907: Find primary for shard default failed</p> <p>But I see no FindPrimary message going out between 12:21:14 and 12:32:15 which indicates it got the Future from the cache. But how could the Future indicate failure when we only put immediate, successful Futures in the cache? Plus other transactions at 12:20:39+ succeeded b/c I see ReadyLocalTransaction messages going out.</p> <p>It sort of seems like there's something funky going on with akka/scala Futures - almost like it somehow loses its completed state and gets some unrelated failure associated to it.</p> <p>Maybe we should try switching to use ListenableFutures for the cache...</p> <p>Could the future returned to us be mutable? Could it be reused by akka and set to something else? What if we cached the PrimaryShardInfo instead of the future?</p> <p>Mutable futures is making less sense to me now. In PrimaryShardInfoFutureCache#putSuccessful we create a new future instead of storing the one from akka#ask. </p> <p>Now it only looks like this may be a case of a confused future...</p> <p>Yeah - it doesn't make sense. We need more debug logging - I'll push a patch for that. I want to try to prove that the scala/akka Future got messed up somehow or maybe I did something wrong in the isCompleted code block.</p> <p>We really don't need to use a scala Future here anyway - ListenableFutures are more of a known commodity - I feel more comfortable with them - I've read the source code and fully understand how they work. I don't really know enough scala to be able to understand their code at this point.</p> <p>Tom, hold off on the ListenableFuture changes till we put in more diagnostics. I'll try to push a diagnostic patch as well.</p> <p>This appears to have been fixed by <a href="https://git.opendaylight.org/gerrit/#/c/21749/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/21749/</a></p> <p>On my machine running a test with 100K routes takes 15 mins for CDS vs 4.5 mins for IMDS.</p> <p>Performance is acceptable now : 1M routes ingested in 5m</p> Duplicate CONTROLLER-1072 Development External issue ID 3340 External issue URL Issue Type ODL SR Target Milestone Rank 0|i02pxb: