https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-1311 controller <p>BGP scale tests 100k routes failed on following exception:</p> <p>2015-05-11 12:57:40,915 | ERROR | ult-dispatcher-2 | Shard | 205 - org.opendaylight.controller.sal-akka-raft - 1.2.0.SNAPSHOT | member-1-shard-default-operational: Error handling BatchedModifications for Tx member-1-txn-313834<br/> java.lang.IllegalStateException: Transaction ReadWriteShardDataTreeTransaction{id=member-1-txn-313834, closed=false, snapshot=MutableDataTree [modification=NodeModification [identifier=(urn:ietf:params:xml:ns:netconf:base:1.0)data, modificationType=TOUCH, childModification={....very long body...}} is open<br/> at com.google.common.base.Preconditions.checkState(Preconditions.java:197)<span class="error">&#91;94:com.google.guava:18.0.0&#93;</span><br/> at org.opendaylight.controller.cluster.datastore.ShardDataTreeTransactionChain.getSnapshot(ShardDataTreeTransactionChain.java:37)<span class="error">&#91;212:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.cluster.datastore.ShardDataTreeTransactionChain.newReadWriteTransaction(ShardDataTreeTransactionChain.java:54)<span class="error">&#91;212:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.cluster.datastore.ShardDataTree.newReadWriteTransaction(ShardDataTree.java:92)<span class="error">&#91;212:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.handleBatchedModifications(ShardCommitCoordinator.java:163)<span class="error">&#91;212:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.cluster.datastore.Shard.handleBatchedModifications(Shard.java:423)<span class="error">&#91;212:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.cluster.datastore.Shard.onReceiveCommand(Shard.java:228)<span class="error">&#91;212:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT&#93;</span><br/> at akka.persistence.UntypedPersistentActor.onReceive(Eventsourced.scala:430)<span class="error">&#91;202:com.typesafe.akka.persistence.experimental:2.3.10&#93;</span><br/> at org.opendaylight.controller.cluster.common.actor.MeteringBehavior.apply(MeteringBehavior.java:97)<span class="error">&#91;204:org.opendaylight.controller.sal-clustering-commons:1.2.0.SNAPSHOT&#93;</span><br/> at akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:534)<span class="error">&#91;197:com.typesafe.akka.actor:2.3.10&#93;</span><br/> at akka.persistence.Recovery$State$class.process(Recovery.scala:30)<span class="error">&#91;202:com.typesafe.akka.persistence.experimental:2.3.10&#93;</span><br/> at akka.persistence.ProcessorImpl$$anon$2.process(Processor.scala:103)<span class="error">&#91;202:com.typesafe.akka.persistence.experimental:2.3.10&#93;</span><br/> at akka.persistence.ProcessorImpl$$anon$2.aroundReceive(Processor.scala:114)<span class="error">&#91;202:com.typesafe.akka.persistence.experimental:2.3.10&#93;</span><br/> at akka.persistence.Recovery$class.aroundReceive(Recovery.scala:265)<span class="error">&#91;202:com.typesafe.akka.persistence.experimental:2.3.10&#93;</span><br/> at akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(Eventsourced.scala:428)<span class="error">&#91;202:com.typesafe.akka.persistence.experimental:2.3.10&#93;</span><br/> at akka.persistence.Eventsourced$$anon$2.doAroundReceive(Eventsourced.scala:82)<span class="error">&#91;202:com.typesafe.akka.persistence.experimental:2.3.10&#93;</span><br/> at akka.persistence.Eventsourced$$anon$2.aroundReceive(Eventsourced.scala:78)<span class="error">&#91;202:com.typesafe.akka.persistence.experimental:2.3.10&#93;</span><br/> at akka.persistence.Eventsourced$class.aroundReceive(Eventsourced.scala:369)<span class="error">&#91;202:com.typesafe.akka.persistence.experimental:2.3.10&#93;</span><br/> at akka.persistence.UntypedPersistentActor.aroundReceive(Eventsourced.scala:428)<span class="error">&#91;202:com.typesafe.akka.persistence.experimental:2.3.10&#93;</span><br/> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)<span class="error">&#91;197:com.typesafe.akka.actor:2.3.10&#93;</span><br/> at akka.actor.ActorCell.invoke(ActorCell.scala:487)<span class="error">&#91;197:com.typesafe.akka.actor:2.3.10&#93;</span><br/> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)<span class="error">&#91;197:com.typesafe.akka.actor:2.3.10&#93;</span><br/> at akka.dispatch.Mailbox.run(Mailbox.scala:221)<span class="error">&#91;197:com.typesafe.akka.actor:2.3.10&#93;</span><br/> at akka.dispatch.Mailbox.exec(Mailbox.scala:231)<span class="error">&#91;197:com.typesafe.akka.actor:2.3.10&#93;</span><br/> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)<span class="error">&#91;194:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b&#93;</span><br/> at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:1253)<span class="error">&#91;194:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b&#93;</span><br/> at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1346)<span class="error">&#91;194:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b&#93;</span><br/> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)<span class="error">&#91;194:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b&#93;</span><br/> at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)<span class="error">&#91;194:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b&#93;</span><br/> 2015-05-11 12:57:42,092 | WARN | lt-dispatcher-16 | Shard | 205 - org.opendaylight.controller.sal-akka-raft - 1.2.0.SNAPSHOT | member-1-shard-default-operational: Transaction member-1-txn-313834 was timed out of the cache<br/> 2015-05-11 12:58:14,430 | WARN | lt-dispatcher-14 | TransactionProxy | 212 - org.opendaylight.controller.sal-distributed-datastore - 1.2.0.SNAPSHOT | Failed to acquire operation permit for transaction member-1-txn-313834</p> <p>Operating System: All<br/> Platform: All</p> CONTROLLER-1311

Clustering : Exception in BGP scale test

Bug Resolved Done Tom Pantelis Dana Kutenicsova Wed, 13 May 2015 08:27:52 +0000 Tue, 30 Jun 2015 09:47:08 +0000 Tue, 30 Jun 2015 09:47:08 +0000 Post-Helium clustering 0 5 <p>Currently masked by <a href="https://jira.opendaylight.org/browse/BGPCEP-221" title="&quot;IllegalStateException: MpReachNlri codec not available&quot; when pushing 10k routes or more" class="issue-link" data-issue-key="BGPCEP-221"><del>BGPCEP-221</del></a> but a possible fix/workaround was suggested for that bug. When that workaround will be confirmed to be working, then this bug can be exposed and hunted down.</p> <p>Does this exception still happen? I've not seen it happen in the bgp scale test.</p> <p>Because of changes in CDS code - this particular situation with the BatchedModifications being sent to the Shard can only happen in the remote case - the local case is covered differently.</p> <p>This exception means that it attempted to create a new read/write tx on the chain before the previous one was readied. However notice that the tx ID (member-1-txn-313834) associated with the BatchedModifications message is the same as the tx ID reported as still being open. Ruling out a tx ID being erroneously reused, that means there was a previous BatchedModifications message from the front-end for the same tx ID that created a read/write tx on the chain and put an entry in the cohortCache. A subsequent BatcheModifications message should find an entry already in the cohortCache and not create a new read/write tx. But in this case it found no entry in the cache and tried to create a new chained tx which indicates there was sufficient time in between the BatchedModifications messages for the entry to be timed out of the cache. Unfortunately we don't have the whole log to confirm this but it seems like that's what happened.</p> <p>The code has since been refactored a bit so entries will no longer age out of the cache in between BatchedModifications message - they can only be aged out once they're enqueued on ready. So it doesn't appear this issue can occur now.</p> <p>However, looking at the code more, I think there's some places with incomplete cleanup in some error code paths.</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/22656/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/22656/</a></p> <p>Not observed during the last 2 weeks of testing.</p> Development External issue ID 3195 External issue URL Issue Type ODL SR Target Milestone Priority Rank 0|i02psf: