https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/OVSDB-324 ovsdb <p>The following message keeps coming in the plugin logs.<br/> Because of this operational ds is not updated with latest info from the switch.</p> <p>2016-03-18 11:39:06,279 | WARN | n-invoker-impl-0 | TransactionInvokerImpl | 228 - org.opendaylight.ovsdb.hwvtepsouthbound-impl - 1.2.2.SNAPSHOT | Exception invoking Transaction:<br/> java.lang.IllegalStateException: Previous transaction member-1-chn-9-txn-1605-1458297541317 is not ready yet<br/> at org.opendaylight.controller.cluster.datastore.TransactionChainProxy$Allocated.checkReady(TransactionChainProxy.java:85)<span class="error">&#91;146:org.opendaylight.controller.sal-distributed-datastore:1.3.1.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.cluster.datastore.TransactionChainProxy.allocateWriteTransaction(TransactionChainProxy.java:192)<span class="error">&#91;146:org.opendaylight.controller.sal-distributed-datastore:1.3.1.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.cluster.datastore.TransactionChainProxy.newReadWriteTransaction(TransactionChainProxy.java:173)<span class="error">&#91;146:org.opendaylight.controller.sal-distributed-datastore:1.3.1.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.cluster.databroker.DOMBrokerReadWriteTransaction.createTransaction(DOMBrokerReadWriteTransaction.java:49)<span class="error">&#91;146:org.opendaylight.controller.sal-distributed-datastore:1.3.1.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.cluster.databroker.DOMBrokerReadWriteTransaction.createTransaction(DOMBrokerReadWriteTransaction.java:22)<span class="error">&#91;146:org.opendaylight.controller.sal-distributed-datastore:1.3.1.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.cluster.databroker.AbstractDOMBrokerTransaction.getSubtransaction(AbstractDOMBrokerTransaction.java:57)<span class="error">&#91;146:org.opendaylight.controller.sal-distributed-datastore:1.3.1.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.cluster.databroker.DOMBrokerReadWriteTransaction.read(DOMBrokerReadWriteTransaction.java:37)<span class="error">&#91;146:org.opendaylight.controller.sal-distributed-datastore:1.3.1.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.md.sal.binding.impl.AbstractForwardedTransaction.doRead(AbstractForwardedTransaction.java:63)<span class="error">&#91;121:org.opendaylight.controller.sal-binding-broker-impl:1.3.1.SNAPSHOT&#93;</span><br/> at org.opendaylight.controller.md.sal.binding.impl.BindingDOMReadWriteTransactionAdapter.read(BindingDOMReadWriteTransactionAdapter.java:31)<span class="error">&#91;121:org.opendaylight.controller.sal-binding-broker-impl:1.3.1.SNAPSHOT&#93;</span><br/> at org.opendaylight.ovsdb.hwvtepsouthbound.HwvtepSouthboundUtil.readNode(HwvtepSouthboundUtil.java:68)<span class="error">&#91;228:org.opendaylight.ovsdb.hwvtepsouthbound-impl:1.2.2.SNAPSHOT&#93;</span><br/> at org.opendaylight.ovsdb.hwvtepsouthbound.transactions.md.HwvtepManagerUpdateCommand.updateManager(HwvtepManagerUpdateCommand.java:60)<span class="error">&#91;228:org.opendaylight.ovsdb.hwvtepsouthbound-impl:1.2.2.SNAPSHOT&#93;</span><br/> at org.opendaylight.ovsdb.hwvtepsouthbound.transactions.md.HwvtepManagerUpdateCommand.execute(HwvtepManagerUpdateCommand.java:54)<span class="error">&#91;228:org.opendaylight.ovsdb.hwvtepsouthbound-impl:1.2.2.SNAPSHOT&#93;</span><br/> at org.opendaylight.ovsdb.hwvtepsouthbound.transactions.md.HwvtepOperationalCommandAggregator.execute(HwvtepOperationalCommandAggregator.java:46)<span class="error">&#91;228:org.opendaylight.ovsdb.hwvtepsouthbound-impl:1.2.2.SNAPSHOT&#93;</span><br/> at org.opendaylight.ovsdb.hwvtepsouthbound.transactions.md.TransactionInvokerImpl.run(TransactionInvokerImpl.java:88)<span class="error">&#91;228:org.opendaylight.ovsdb.hwvtepsouthbound-impl:1.2.2.SNAPSHOT&#93;</span><br/> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)<span class="error">&#91;:1.7.0_85&#93;</span><br/> at java.util.concurrent.FutureTask.run(FutureTask.java:262)<span class="error">&#91;:1.7.0_85&#93;</span><br/> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)<span class="error">&#91;:1.7.0_85&#93;</span><br/> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)<span class="error">&#91;:1.7.0_85&#93;</span><br/> at java.lang.Thread.run(Thread.java:745)<span class="error">&#91;:1.7.0_85&#93;</span></p> <p>Operating System: All<br/> Platform: All</p> OVSDB-324

operational datastore is not getting updated after the previous transaction failed.

Bug Resolved Done Unassigned suneel verma Fri, 18 Mar 2016 11:14:59 +0000 Mon, 30 Oct 2017 19:52:45 +0000 Wed, 31 Aug 2016 21:34:04 +0000 unspecified Southbound.hw_vtep 0 4 <p>Hi Vishal,</p> <p>Any update on this bug ?</p> <p>Thanks<br/> Anil</p> <p>I have a workaround for it already merged in hwvtep plugin. This issue happens when there is a bug in hwvtep plugin code <span class="error">&#91;or could be in OVSDB code too&#93;</span>, resulting in exception. TransactionInvoker retries failed transactions but if there is bug it will fail everytime. My workaround catches exceptions. A proper fix could be to catch non-DataStore exceptions in TransactionInvoker and not not retry in that case. Not sure if it is good idea or not.</p> <p>We've faced a similar issue twice yesterday. It seems to me that the issue halted the thread of the run() method of the TransactionInvokerImpl class of org.opendaylight.ovsdb.southbound.transactions.md package. After the issue happened, the OVSDB Southbound plugin didn't process any update notifications.</p> <p>That's really critical issue. So I've changed the "Importance" of the bug into "blocker".</p> <p>We've seen this issue when we tested the legacy NetVirt with 3-ODL-node cluster on the stable/beryllium branch. We've faced this issue twice in three test runs. Therefore, I think it is not a rare event. It would happen often.</p> <p>Just before the thread was halted, ODL output the following WARN messages. I don't think it shows the direct cause of the issue. However, I can guess something bad happens around ShardDataTreeTransactionChain.</p> <p>2016-08-23 22:10:57,909 | WARN | lt-dispatcher-15 | ConcurrentDOMDataBroker | 166 - org.opendaylight.controller.sal-distributed-datastore - 1.3.4.SNAPSHOT | Tx: DOM-CHAIN-0-58 Error during phase CAN_COMMIT, starting Abort<br/> java.lang.Exception: Error creating READ_WRITE transaction on shard topology<br/> at org.opendaylight.controller.cluster.datastore.RemoteTransactionContextSupport.createTransactionContext(RemoteTransactionContextSupport.java:222)<br/> at org.opendaylight.controller.cluster.datastore.RemoteTransactionContextSupport.onCreateTransactionComplete(RemoteTransactionContextSupport.java:200)<br/> at org.opendaylight.controller.cluster.datastore.RemoteTransactionContextSupport.access$000(RemoteTransactionContextSupport.java:40)<br/> at org.opendaylight.controller.cluster.datastore.RemoteTransactionContextSupport$1.onComplete(RemoteTransactionContextSupport.java:135)<br/> at akka.dispatch.OnComplete.internal(Future.scala:247)<br/> at akka.dispatch.OnComplete.internal(Future.scala:245)<br/> at akka.dispatch.japi$CallbackBridge.apply(Future.scala:175)<br/> at akka.dispatch.japi$CallbackBridge.apply(Future.scala:172)<br/> at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)<br/> at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)<br/> at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)<br/> at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)<br/> at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)<br/> at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)<br/> at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)<br/> at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)<br/> at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)<br/> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)<br/> at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)<br/> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)<br/> at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)<br/> Caused by: java.lang.IllegalStateException: Transaction ReadWriteShardDataTreeTransaction{id=shard-member-2-chn-1-txn-59-1472004651115, closed=false, snapshot=MutableDataTree [modification=NodeModification <span class="error">&#91;identifier=(urn:ietf:params:xml:ns:netconf:base:1.0)data, modificationType=NONE, childModification={}&#93;</span>]} is open<br/> at com.google.common.base.Preconditions.checkState(Preconditions.java:197)<br/> at org.opendaylight.controller.cluster.datastore.ShardDataTreeTransactionChain.getSnapshot(ShardDataTreeTransactionChain.java:37)<br/> at org.opendaylight.controller.cluster.datastore.ShardDataTreeTransactionChain.newReadWriteTransaction(ShardDataTreeTransactionChain.java:54)<br/> at org.opendaylight.controller.cluster.datastore.ShardDataTree.newReadWriteTransaction(ShardDataTree.java:118)<br/> at org.opendaylight.controller.cluster.datastore.ShardTransactionActorFactory.newShardTransaction(ShardTransactionFactory.java:49)<br/> at org.opendaylight.controller.cluster.datastore.Shard.createTypedTransactionActor(Shard.java:560)<br/> at org.opendaylight.controller.cluster.datastore.Shard.createTransaction(Shard.java:592)<br/> at org.opendaylight.controller.cluster.datastore.Shard.createTransaction(Shard.java:571)<br/> at org.opendaylight.controller.cluster.datastore.Shard.handleCreateTransaction(Shard.java:543)<br/> at org.opendaylight.controller.cluster.datastore.Shard.onReceiveCommand(Shard.java:233)<br/> at akka.persistence.UntypedPersistentActor.onReceive(Eventsourced.scala:430)<br/> at org.opendaylight.controller.cluster.common.actor.MeteringBehavior.apply(MeteringBehavior.java:97)<br/> at akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:534)<br/> at akka.persistence.Recovery$State$class.process(Recovery.scala:30)<br/> at akka.persistence.ProcessorImpl$$anon$2.process(Processor.scala:103)<br/> at akka.persistence.ProcessorImpl$$anon$2.aroundReceive(Processor.scala:114)<br/> at akka.persistence.Recovery$class.aroundReceive(Recovery.scala:265)<br/> at akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(Eventsourced.scala:428)<br/> at akka.persistence.Eventsourced$$anon$2.doAroundReceive(Eventsourced.scala:82)<br/> at akka.persistence.Eventsourced$$anon$2.aroundReceive(Eventsourced.scala:78)<br/> at akka.persistence.Eventsourced$class.aroundReceive(Eventsourced.scala:369)<br/> at akka.persistence.UntypedPersistentActor.aroundReceive(Eventsourced.scala:428)<br/> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)<br/> at akka.actor.ActorCell.invoke(ActorCell.scala:487)<br/> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)<br/> at akka.dispatch.Mailbox.run(Mailbox.scala:220)<br/> ... 5 more</p> <p>2016-08-23 22:10:57,932 | WARN | n-invoker-impl-0 | SouthboundUtil | 253 - org.opendaylight.ovsdb.southbound-impl - 1.2.5.SNAPSHOT | Read Operational/DS for Node failed! KeyedInstanceIdentifier</p> {targetType=interface org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.topology.Node, path=[org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.NetworkTopology, org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.Topology[key=TopologyKey [_topologyId=Uri [_value=ovsdb:1]]], org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.topology.Node[key=NodeKey [_nodeId=Uri [_value=ovsdb://uuid/69d5898c-a35e-4639-9df7-7fbda31d5c48]]]]} <p>ReadFailedException{message=Error executeRead ReadData for path /(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)network-topology/topology/topology[</p> {(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)topology-id=ovsdb:1} <p>]/node/node[</p> {(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)node-id=ovsdb://uuid/69d5898c-a35e-4639-9df7-7fbda31d5c48} <p>], errorList=[RpcError [message=Error executeRead ReadData for path /(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)network-topology/topology/topology[</p> {(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)topology-id=ovsdb:1} <p>]/node/node[</p> {(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)node-id=ovsdb://uuid/69d5898c-a35e-4639-9df7-7fbda31d5c48} <p>], severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=java.lang.Exception: Error creating READ_WRITE transaction on shard topology]]}<br/> at org.opendaylight.controller.cluster.datastore.NoOpTransactionContext.executeRead(NoOpTransactionContext.java:71)<br/> at org.opendaylight.controller.cluster.datastore.TransactionProxy$1.invoke(TransactionProxy.java:92)<br/> at org.opendaylight.controller.cluster.datastore.TransactionContextWrapper.executePriorTransactionOperations(TransactionContextWrapper.java:132)<br/> at org.opendaylight.controller.cluster.datastore.AbstractTransactionContextFactory.onFindPrimaryShardFailure(AbstractTransactionContextFactory.java:97)<br/> at org.opendaylight.controller.cluster.datastore.AbstractTransactionContextFactory.access$100(AbstractTransactionContextFactory.java:35)<br/> at org.opendaylight.controller.cluster.datastore.AbstractTransactionContextFactory$1.onComplete(AbstractTransactionContextFactory.java:123)<br/> at org.opendaylight.controller.cluster.datastore.AbstractTransactionContextFactory$1.onComplete(AbstractTransactionContextFactory.java:117)<br/> at akka.dispatch.OnComplete.internal(Future.scala:247)<br/> at akka.dispatch.OnComplete.internal(Future.scala:245)<br/> at akka.dispatch.japi$CallbackBridge.apply(Future.scala:175)<br/> at akka.dispatch.japi$CallbackBridge.apply(Future.scala:172)<br/> at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)<br/> at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)<br/> at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)<br/> at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)<br/> at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)<br/> at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)<br/> at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)<br/> at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)<br/> at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)<br/> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)<br/> at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)<br/> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)<br/> at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)<br/> Caused by: java.lang.Exception: Error creating READ_WRITE transaction on shard topology<br/> at org.opendaylight.controller.cluster.datastore.RemoteTransactionContextSupport.createTransactionContext(RemoteTransactionContextSupport.java:222)<br/> at org.opendaylight.controller.cluster.datastore.RemoteTransactionContextSupport.onCreateTransactionComplete(RemoteTransactionContextSupport.java:200)<br/> at org.opendaylight.controller.cluster.datastore.RemoteTransactionContextSupport.access$000(RemoteTransactionContextSupport.java:40)<br/> at org.opendaylight.controller.cluster.datastore.RemoteTransactionContextSupport$1.onComplete(RemoteTransactionContextSupport.java:135)<br/> ... 17 more<br/> Caused by: java.lang.IllegalStateException: Transaction ReadWriteShardDataTreeTransaction{id=shard-member-2-chn-1-txn-59-1472004651115, closed=false, snapshot=MutableDataTree [modification=NodeModification <span class="error">&#91;identifier=(urn:ietf:params:xml:ns:netconf:base:1.0)data, modificationType=NONE, childModification={}&#93;</span>]} is open<br/> at com.google.common.base.Preconditions.checkState(Preconditions.java:197)<br/> at org.opendaylight.controller.cluster.datastore.ShardDataTreeTransactionChain.getSnapshot(ShardDataTreeTransactionChain.java:37)<br/> at org.opendaylight.controller.cluster.datastore.ShardDataTreeTransactionChain.newReadWriteTransaction(ShardDataTreeTransactionChain.java:54)<br/> at org.opendaylight.controller.cluster.datastore.ShardDataTree.newReadWriteTransaction(ShardDataTree.java:118)<br/> at org.opendaylight.controller.cluster.datastore.ShardTransactionActorFactory.newShardTransaction(ShardTransactionFactory.java:49)<br/> at org.opendaylight.controller.cluster.datastore.Shard.createTypedTransactionActor(Shard.java:560)<br/> at org.opendaylight.controller.cluster.datastore.Shard.createTransaction(Shard.java:592)<br/> at org.opendaylight.controller.cluster.datastore.Shard.createTransaction(Shard.java:571)<br/> at org.opendaylight.controller.cluster.datastore.Shard.handleCreateTransaction(Shard.java:543)<br/> at org.opendaylight.controller.cluster.datastore.Shard.onReceiveCommand(Shard.java:233)<br/> at akka.persistence.UntypedPersistentActor.onReceive(Eventsourced.scala:430)<br/> at org.opendaylight.controller.cluster.common.actor.MeteringBehavior.apply(MeteringBehavior.java:97)<br/> at akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:534)<br/> at akka.persistence.Recovery$State$class.process(Recovery.scala:30)<br/> at akka.persistence.ProcessorImpl$$anon$2.process(Processor.scala:103)<br/> at akka.persistence.ProcessorImpl$$anon$2.aroundReceive(Processor.scala:114)<br/> at akka.persistence.Recovery$class.aroundReceive(Recovery.scala:265)<br/> at akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(Eventsourced.scala:428)<br/> at akka.persistence.Eventsourced$$anon$2.doAroundReceive(Eventsourced.scala:82)<br/> at akka.persistence.Eventsourced$$anon$2.aroundReceive(Eventsourced.scala:78)<br/> at akka.persistence.Eventsourced$class.aroundReceive(Eventsourced.scala:369)<br/> at akka.persistence.UntypedPersistentActor.aroundReceive(Eventsourced.scala:428)<br/> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)<br/> at akka.actor.ActorCell.invoke(ActorCell.scala:487)<br/> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)<br/> at akka.dispatch.Mailbox.run(Mailbox.scala:220)<br/> ... 5 more</p> <p>Attachment karaf.log has been added with description: Log messages when we faced the issue that the run() thread was halted</p> <p>We faced the issue again. And, we've observed that IllegalStateException happens at calling the ReadWriteTransaction.submit() in the run method of the TransactionInvokerImpl. The reason for the IllegalStateException was that the submit() method was called when the state of the DOMBrokerTransactionChain was FAILED. The reason that the chain was at FAILED state was that a previous calling to the submit() on the chain failed. As of now, I'm not sure why a previous calling to the submit() was failed.</p> <p>What I know is that just before the IllegalStateException happened, ReadWriteTransaction.read() failed during the phase CAN_COMMIT. I'm not sure even why the read() method needs to enter "CAN_COMMIT", that's a little bit weird to me, since I think reading data itself doesn't require any commit.</p> <p>For more details, please see the attached karaf.log.</p> <p>To know what kind of Exception happens in the run method of the TransactionInvoker, in our local environment, we've changed the condition of the catch block in the line 98 of the TransactionInvokerImpl.java as follows to catch any Exception.</p> <p> <span class="error">&#91;southbound/southbound-impl/src/main/java/org/opendaylight/ovsdb/southbound/transactions/md/TransactionInvokerImpl.java&#93;</span></p> <p> Current: } catch (InterruptedException e) </p> { Changed: } <p> catch (Exception e) {</p> <p>The attached karaf.log was the log file when we observed that the run() thread was halted with the above code change.</p> <p>stable/boron : <a href="https://git.opendaylight.org/gerrit/#/c/44806/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/44806/</a><br/> master : <a href="https://git.opendaylight.org/gerrit/#/c/44090/2" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/44090/2</a></p> <p>I think we can close this bug. I mean I think it's ok to change the status to RESOLVED.</p> <p>The current status is we've already merged the change for this bug into the master and stable/boron branches. And, we've not seen this issue after that.</p> Development External issue ID 5555 External issue URL Issue Type Rank 0|i02213: