https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-996 controller <p>2014-10-31 11:29:44,814 | WARN | lt-dispatcher-42 | ShardManager | 248 - com.typesafe.akka.slf4j - 2.3.4 | Supervisor Strategy of resume applied<br/> at com.google.common.base.Preconditions.checkState(Preconditions.java:176)<br/> at org.opendaylight.controller.md.sal.dom.store.impl.DOMStoreTransactionChainImpl$Allocated.getSnapshot(DOMStoreTransactionChainImpl.java:68)<br/> at org.opendaylight.controller.md.sal.dom.store.impl.DOMStoreTransactionChainImpl.getSnapshot(DOMStoreTransactionChainImpl.java:111)<br/> at org.opendaylight.controller.md.sal.dom.store.impl.DOMStoreTransactionChainImpl.newReadWriteTransaction(DOMStoreTransactionChainImpl.java:131)<br/> at org.opendaylight.controller.cluster.datastore.Shard.createTypedTransactionActor(Shard.java:508)<br/> at org.opendaylight.controller.cluster.datastore.Shard.createTransaction(Shard.java:543)<br/> at org.opendaylight.controller.cluster.datastore.Shard.createTransaction(Shard.java:530)<br/> at org.opendaylight.controller.cluster.datastore.Shard.handleCreateTransaction(Shard.java:441)<br/> at org.opendaylight.controller.cluster.datastore.Shard.onReceiveCommand(Shard.java:240)<br/> at akka.persistence.UntypedPersistentActor.onReceive(Eventsourced.scala:430)<br/> at org.opendaylight.controller.cluster.common.actor.MeteringBehavior.apply(MeteringBehavior.java:96)<br/> at akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:534)<br/> at akka.persistence.Recovery$State$class.process(Recovery.scala:30)<br/> at akka.persistence.ProcessorImpl$$anon$2.process(Processor.scala:103)<br/> at akka.persistence.ProcessorImpl$$anon$2.aroundReceive(Processor.scala:114)<br/> at akka.persistence.Recovery$class.aroundReceive(Recovery.scala:256)<br/> at akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(Eventsourced.scala:428)<br/> at akka.persistence.Eventsourced$$anon$2.doAroundReceive(Eventsourced.scala:82)<br/> at akka.persistence.Eventsourced$$anon$2.aroundReceive(Eventsourced.scala:78)<br/> at akka.persistence.Eventsourced$class.aroundReceive(Eventsourced.scala:369)<br/> at akka.persistence.UntypedPersistentActor.aroundReceive(Eventsourced.scala:428)<br/> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)<br/> at akka.actor.ActorCell.invoke(ActorCell.scala:487)<br/> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)<br/> at akka.dispatch.Mailbox.run(Mailbox.scala:220)<br/> at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)<br/> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)<br/> at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)<br/> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)<br/> at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)</p> <p>2014-10-31 11:29:44,815 | WARN | lt-dispatcher-42 | OneForOneStrategy | 248 - com.typesafe.akka.slf4j - 2.3.4 | Previous transaction member-1-shard-inventory-operational-40 is not ready yet</p> <p>Operating System: All<br/> Platform: All</p> CONTROLLER-996

Clustering : Exception thrown in Shard because a transaction was created on a chain when previous transaction was not yet ready

Bug Resolved Done Tom Pantelis Moiz Raja Wed, 5 Nov 2014 01:35:34 +0000 Tue, 25 Jul 2023 08:24:18 +0000 Sat, 15 Nov 2014 15:45:01 +0000 mdsal 0 4 <p>I think the Tx chain is coming from the FlowCapableInventoryProvider and this error is related to <a href="https://jira.opendaylight.org/browse/CONTROLLER-997" title="Clustering : Occasional failure to create transactions during single node integration test" class="issue-link" data-issue-key="CONTROLLER-997"><del>CONTROLLER-997</del></a>.</p> <p>The FlowCapableInventoryProvider uses a Tx chain to continuously batch and submit modification operations on a separate thread. After it submits a Tx batch, it creates a new read-write Tx from the chain and starts a new batch. It does not wait for the Future from the the previous Tx to complete. This is valid. The semantics of a Tx chain are such that the modifications from a previously submitted Tx in the chain are visible to the next Tx without the client having to wait for the previous Tx to be committed. So as soon as the previous Tx is readied, the next Tx can be created and its snapshot will contain the modifications made by the previous Tx.</p> <p>With the IMDS, i.e. w/o clustering, Tx's are readied synchronously and then submitted on a thread to be committed. However with the CDS, Tx's are readied async, i.e. the TransactionProxy sends a message to the ShardTransaction actor. So I think it's this timing difference that can cause issues with Tx chains and break the semantics.</p> <p>Here's a scenario that can break with the CDS:</p> <p>ReadWriteTransaction tx1 = txChain.newReadWriteTransaction()<br/> tx1.write(id1, path1);<br/> tx1.submit();</p> <p>ReadWriteTransaction tx2 = txChain.newReadWriteTransaction()<br/> tx2.write(id2, path2);</p> <p>On tx.submit(), the ready operation is done async and may not complete before tx2 is created. If so, tx2 creation fails. With the IMDS, tx1 is readied when submit returns to the caller so this issue does not occur.</p> <p>In the CDS, we need to ensure the previous Tx in a chain completes its ready operation before it attempts to create the next Tx.</p> <p>I think this is the root cause of <a href="https://jira.opendaylight.org/browse/CONTROLLER-997" title="Clustering : Occasional failure to create transactions during single node integration test" class="issue-link" data-issue-key="CONTROLLER-997"><del>CONTROLLER-997</del></a>. In the ex trace in 2319, the Tx create timed out. This ex appears to be the root cause - it was thrown back to akka which doesn't propagate it. Also, this ex incurred at 11:29:44 and the ex in 2319 occurred at 11:29:49, the latter being 5 seconds after which is the ask time out.</p> <p>This issue may also be the cause of <a href="https://jira.opendaylight.org/browse/CONTROLLER-998" title="OptimisticLockFailed exception when writing inventory and topology data" class="issue-link" data-issue-key="CONTROLLER-998"><del>CONTROLLER-998</del></a> as well.</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/12535/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/12535/</a> for master</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/12537/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/12537/</a> for Helium.</p> <p>Submitted follow-up patch <a href="https://git.opendaylight.org/gerrit/#/c/12582/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/12582/</a> to master.</p> <p>Need to cherry pick to stable/helium so keeping this bug open for now.</p> <p>Merged follow-up patch <a href="https://git.opendaylight.org/gerrit/#/c/12878/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/12878/</a> to helium</p> Blocks CONTROLLER-1016 CONTROLLER-1006 CONTROLLER-1007 Development External issue ID 2318 External issue URL ODL SR Target Milestone Rank 0|i02nuf: