https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/BGPCEP-486 bgpcep <p>Race is observed </p> <p>2016-06-24 17:55:24,877 | ERROR | lt-dispatcher-36 | DataTreeChangeListenerActor | 149 - org.opendaylight.controller.sal-distributed-datastore - 1.3.0.Beryllium | Error notifying listener org.opendaylight.controller.md.sal.binding.impl.BindingDOMDataTreeChangeListenerAdapter@2cf0c767<br/> java.lang.IllegalStateException: New transaction PingPongTransaction</p> {delegate=org.opendaylight.controller.cluster.databroker.DOMBrokerReadWriteTransaction@535763c8} <p> raced with transacion PingPongTransaction</p> {delegate=org.opendaylight.controller.cluster.databroker.DOMBrokerReadWriteTransaction@f493600} <p> at org.opendaylight.controller.md.sal.dom.broker.impl.PingPongTransactionChain.slowAllocateTransaction(PingPongTransactionChain.java:136)<span class="error">&#91;122:org.opendaylight.controller.sal-broker-impl:1.3.0.Beryllium&#93;</span><br/> at org.opendaylight.controller.md.sal.dom.broker.impl.PingPongTransactionChain.allocateTransaction(PingPongTransactionChain.java:148)<span class="error">&#91;122:org.opendaylight.controller.sal-broker-impl:1.3.0.Beryllium&#93;</span><br/> at org.opendaylight.controller.md.sal.dom.broker.impl.PingPongTransactionChain.newReadWriteTransaction(PingPongTransactionChain.java:328)<span class="error">&#91;122:org.opendaylight.controller.sal-broker-impl:1.3.0.Beryllium&#93;</span><br/> at org.opendaylight.controller.md.sal.binding.impl.BindingDOMTransactionChainAdapter.newReadWriteTransaction(BindingDOMTransactionChainAdapter.java:62)<span class="error">&#91;124:org.opendaylight.controller.sal-binding-broker-impl:1.3.0.Beryllium&#93;</span><br/> at org.opendaylight.bgpcep.bgp.topology.provider.AbstractTopologyBuilder.onDataTreeChanged(AbstractTopologyBuilder.java:130)<span class="error">&#91;327:org.opendaylight.bgpcep.bgp-topology-provider:0.5.0.Beryllium&#93;</span><br/> at org.opendaylight.controller.md.sal.binding.impl.BindingDOMDataTreeChangeListenerAdapter.onDataTreeChanged(BindingDOMDataTreeChangeListenerAdapter.java:41)<span class="error">&#91;124:org.opendaylight.controller.sal-binding-broker-impl:1.3.0.Beryllium&#93;</span><br/> at org.opendaylight.controller.cluster.datastore.DataTreeChangeListenerActor.dataChanged(DataTreeChangeListenerActor.java:53)<span class="error">&#91;149:org.opendaylight.controller.sal-distributed-datastore:1.3.0.Beryllium&#93;</span><br/> at org.opendaylight.controller.cluster.datastore.DataTreeChangeListenerActor.handleReceive(DataTreeChangeListenerActor.java:37)<span class="error">&#91;149:org.opendaylight.controller.sal-distributed-datastore:1.3.0.Beryllium&#93;</span><br/> at org.opendaylight.controller.cluster.common.actor.AbstractUntypedActor.onReceive(AbstractUntypedActor.java:34)<span class="error">&#91;145:org.opendaylight.controller.sal-clustering-commons:1.3.0.Beryllium&#93;</span><br/> at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:167)<span class="error">&#91;134:com.typesafe.akka.actor:2.3.14&#93;</span><br/> at akka.actor.Actor$class.aroundReceive(Actor.scala:467)<span class="error">&#91;134:com.typesafe.akka.actor:2.3.14&#93;</span><br/> at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:97)<span class="error">&#91;134:com.typesafe.akka.actor:2.3.14&#93;</span><br/> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)<span class="error">&#91;134:com.typesafe.akka.actor:2.3.14&#93;</span><br/> at akka.actor.ActorCell.invoke(ActorCell.scala:487)<span class="error">&#91;134:com.typesafe.akka.actor:2.3.14&#93;</span><br/> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)<span class="error">&#91;134:com.typesafe.akka.actor:2.3.14&#93;</span><br/> at akka.dispatch.Mailbox.run(Mailbox.scala:220)<span class="error">&#91;134:com.typesafe.akka.actor:2.3.14&#93;</span><br/> at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)<span class="error">&#91;134:com.typesafe.akka.actor:2.3.14&#93;</span><br/> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)<span class="error">&#91;131:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c&#93;</span><br/> at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)<span class="error">&#91;131:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c&#93;</span><br/> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)<span class="error">&#91;131:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c&#93;</span><br/> at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)<span class="error">&#91;131:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c&#93;</span></p> <p>Operating System: All<br/> Platform: All</p> BGPCEP-486

PingPongTransaction race when handling data tree change

Bug Resolved Done Kevin Wang Kevin Wang Fri, 24 Jun 2016 23:39:17 +0000 Sun, 3 Mar 2019 11:49:47 +0000 Mon, 15 Aug 2016 09:44:34 +0000 Bugzilla Migration Bugzilla Migration BGP 0 4 <p>Could you please add steps to reproduce or at least some more info when this issue happens? Optionally, could you try it with a latest Beryllium release (SR2)?</p> <p>Thanks,<br/> Milos</p> <p>Cannot reproduce for now. Will reopen if more information about this bug is observed.</p> <p>The node is trying to sync the bgp-linkstate topology with other nodes, however, the transaction fails due to timeout.</p> <p>2016-07-19 17:19:59,376 | ERROR | CommitFutures-10 | AbstractTopologyBuilder | 327 - org.opendaylight.bgpcep.bgp-topology-provider - 0.5.0.Beryllium | Topology builder for KeyedInstanceIdentifier</p> {targetType=interface org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.Topology, path=[org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.NetworkTopology, org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.Topology[key=TopologyKey [_topologyId=Uri [_value=CBB-linkstate-topology-3]]]]} <p> failed in transaction DOM-CHAIN-12-333<br/> TransactionCommitFailedException</p> {message=canCommit encountered an unexpected failure, errorList=[RpcError [message=canCommit encountered an unexpected failure, severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=akka.pattern.AskTimeoutException: Ask timed out on [ActorSelection[Anchor(akka.tcp://opendaylight-cluster-data@mtsnjv9sdbc12-eth1-0.sdcp2.aic.cip.att.com:2550/), Path(/user/shardmanager-operational/member-2-shard-topology-operational/shard-member-3-chn-25-txn-334-1468948738750#-1973141357)]] after [30000 ms]]]} <p> at org.opendaylight.controller.md.sal.dom.broker.impl.TransactionCommitFailedExceptionMapper.newWithCause(TransactionCommitFailedExceptionMapper.java:37)<span class="error">&#91;111:org.opendaylight.controller.sal-broker-impl:1.3.0.Beryllium&#93;</span><br/> at org.opendaylight.controller.md.sal.dom.broker.impl.TransactionCommitFailedExceptionMapper.newWithCause(TransactionCommitFailedExceptionMapper.java:18)<span class="error">&#91;111:org.opendaylight.controller.sal-broker-impl:1.3.0.Beryllium&#93;</span><br/> at org.opendaylight.yangtools.util.concurrent.ExceptionMapper.apply(ExceptionMapper.java:96)<span class="error">&#91;41:org.opendaylight.yangtools.util:0.8.0.Beryllium&#93;</span><br/> at org.opendaylight.controller.cluster.datastore.ConcurrentDOMDataBroker.handleException(ConcurrentDOMDataBroker.java:209)<span class="error">&#91;138:org.opendaylight.controller.sal-distributed-datastore:1.3.0.Beryllium&#93;</span><br/> at org.opendaylight.controller.cluster.datastore.ConcurrentDOMDataBroker.access$000(ConcurrentDOMDataBroker.java:44)<span class="error">&#91;138:org.opendaylight.controller.sal-distributed-datastore:1.3.0.Beryllium&#93;</span><br/> at org.opendaylight.controller.cluster.datastore.ConcurrentDOMDataBroker$1.onFailure(ConcurrentDOMDataBroker.java:117)<span class="error">&#91;138:org.opendaylight.controller.sal-distributed-datastore:1.3.0.Beryllium&#93;</span><br/> at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)<span class="error">&#91;38:com.google.guava:18.0.0&#93;</span><br/> at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)<span class="error">&#91;38:com.google.guava:18.0.0&#93;</span><br/> at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)<span class="error">&#91;38:com.google.guava:18.0.0&#93;</span><br/> at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)<span class="error">&#91;38:com.google.guava:18.0.0&#93;</span><br/> at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)<span class="error">&#91;38:com.google.guava:18.0.0&#93;</span><br/> at com.google.common.util.concurrent.SettableFuture.setException(SettableFuture.java:68)<span class="error">&#91;38:com.google.guava:18.0.0&#93;</span><br/> at org.opendaylight.controller.cluster.datastore.SingleCommitCohortProxy$1.onComplete(SingleCommitCohortProxy.java:60)<span class="error">&#91;138:org.opendaylight.controller.sal-distributed-datastore:1.3.0.Beryllium&#93;</span><br/> at akka.dispatch.OnComplete.internal(Future.scala:247)<span class="error">&#91;123:com.typesafe.akka.actor:2.3.14&#93;</span><br/> at akka.dispatch.OnComplete.internal(Future.scala:245)<span class="error">&#91;123:com.typesafe.akka.actor:2.3.14&#93;</span><br/> at akka.dispatch.japi$CallbackBridge.apply(Future.scala:175)<span class="error">&#91;123:com.typesafe.akka.actor:2.3.14&#93;</span><br/> at akka.dispatch.japi$CallbackBridge.apply(Future.scala:172)<span class="error">&#91;123:com.typesafe.akka.actor:2.3.14&#93;</span><br/> at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)<span class="error">&#91;120:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c&#93;</span><br/> at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)<span class="error">&#91;123:com.typesafe.akka.actor:2.3.14&#93;</span><br/> at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)<span class="error">&#91;123:com.typesafe.akka.actor:2.3.14&#93;</span><br/> at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)<span class="error">&#91;123:com.typesafe.akka.actor:2.3.14&#93;</span><br/> at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)<span class="error">&#91;123:com.typesafe.akka.actor:2.3.14&#93;</span><br/> at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)<span class="error">&#91;120:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c&#93;</span><br/> at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)<span class="error">&#91;123:com.typesafe.akka.actor:2.3.14&#93;</span><br/> at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)<span class="error">&#91;123:com.typesafe.akka.actor:2.3.14&#93;</span><br/> at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)<span class="error">&#91;123:com.typesafe.akka.actor:2.3.14&#93;</span><br/> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)<span class="error">&#91;120:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c&#93;</span><br/> at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)<span class="error">&#91;120:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c&#93;</span><br/> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)<span class="error">&#91;120:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c&#93;</span><br/> at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)<span class="error">&#91;120:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c&#93;</span><br/> Caused by: akka.pattern.AskTimeoutException: Ask timed out on [ActorSelection<span class="error">&#91;Anchor(akka.tcp://opendaylight-cluster-data@mtsnjv9sdbc12-eth1-0.sdcp2.aic.cip.att.com:2550/), Path(/user/shardmanager-operational/member-2-shard-topology-operational/shard-member-3-chn-25-txn-334-1468948738750#-1973141357)&#93;</span>] after <span class="error">&#91;30000 ms&#93;</span><br/> at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:334)<span class="error">&#91;123:com.typesafe.akka.actor:2.3.14&#93;</span><br/> at akka.actor.Scheduler$$anon$7.run(Scheduler.scala:117)<span class="error">&#91;123:com.typesafe.akka.actor:2.3.14&#93;</span><br/> at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:599)<span class="error">&#91;120:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c&#93;</span><br/> at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)<span class="error">&#91;120:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c&#93;</span><br/> at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:597)<span class="error">&#91;120:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c&#93;</span><br/> at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(Scheduler.scala:467)<span class="error">&#91;123:com.typesafe.akka.actor:2.3.14&#93;</span><br/> at akka.actor.LightArrayRevolverScheduler$$anon$8.executeBucket$1(Scheduler.scala:419)<span class="error">&#91;123:com.typesafe.akka.actor:2.3.14&#93;</span><br/> at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:423)<span class="error">&#91;123:com.typesafe.akka.actor:2.3.14&#93;</span><br/> at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)<span class="error">&#91;123:com.typesafe.akka.actor:2.3.14&#93;</span><br/> at java.lang.Thread.run(Thread.java:745)<span class="error">&#91;:1.7.0_76&#93;</span><br/> 2016-07-19 17:19:59,376 | ERROR | CommitFutures-10 | AbstractTopologyBuilder | 327 - org.opendaylight.bgpcep.bgp-topology-provider - 0.5.0.Beryllium | Failed to propagate change by listener org.opendaylight.bgpcep.bgp.topology.provider.LinkstateTopologyBuilder@4907aa42<br/> 2016-07-19 17:19:59,393 | WARN | lt-dispatcher-27 | AbstractTopologyBuilder | 327 - org.opendaylight.bgpcep.bgp-topology-provider - 0.5.0.Beryllium | Data change org.opendaylight.controller.md.sal.binding.impl.LazyDataTreeModification@64d3333d was not completely propagated to listener org.opendaylight.bgpcep.bgp.topology.provider.LinkstateTopologyBuilder@4907aa42, aborting<br/> java.lang.IllegalStateException: Previous transaction member-3-chn-25-txn-335-1468948769013 is not ready yet<br/> at org.opendaylight.controller.cluster.datastore.TransactionChainProxy$Allocated.checkReady(TransactionChainProxy.java:85)<span class="error">&#91;138:org.opendaylight.controller.sal-distributed-datastore:1.3.0.Beryllium&#93;</span><br/> at org.opendaylight.controller.cluster.datastore.TransactionChainProxy.allocateWriteTransaction(TransactionChainProxy.java:192)<span class="error">&#91;138:org.opendaylight.controller.sal-distributed-datastore:1.3.0.Beryllium&#93;</span><br/> at org.opendaylight.controller.cluster.datastore.TransactionChainProxy.newReadWriteTransaction(TransactionChainProxy.java:173)<span class="error">&#91;138:org.opendaylight.controller.sal-distributed-datastore:1.3.0.Beryllium&#93;</span><br/> at org.opendaylight.controller.cluster.databroker.DOMBrokerReadWriteTransaction.createTransaction(DOMBrokerReadWriteTransaction.java:49)<span class="error">&#91;138:org.opendaylight.controller.sal-distributed-datastore:1.3.0.Beryllium&#93;</span><br/> at org.opendaylight.controller.cluster.databroker.DOMBrokerReadWriteTransaction.createTransaction(DOMBrokerReadWriteTransaction.java:22)<span class="error">&#91;138:org.opendaylight.controller.sal-distributed-datastore:1.3.0.Beryllium&#93;</span><br/> at org.opendaylight.controller.cluster.databroker.AbstractDOMBrokerTransaction.getSubtransaction(AbstractDOMBrokerTransaction.java:57)<span class="error">&#91;138:org.opendaylight.controller.sal-distributed-datastore:1.3.0.Beryllium&#93;</span><br/> at org.opendaylight.controller.cluster.databroker.AbstractDOMBrokerWriteTransaction.delete(AbstractDOMBrokerWriteTransaction.java:88)<span class="error">&#91;138:org.opendaylight.controller.sal-distributed-datastore:1.3.0.Beryllium&#93;</span><br/> at org.opendaylight.controller.cluster.databroker.AbstractDOMBrokerWriteTransaction.delete(AbstractDOMBrokerWriteTransaction.java:35)<span class="error">&#91;138:org.opendaylight.controller.sal-distributed-datastore:1.3.0.Beryllium&#93;</span><br/> at org.opendaylight.controller.md.sal.dom.spi.ForwardingDOMDataReadWriteTransaction.delete(ForwardingDOMDataReadWriteTransaction.java:64)<span class="error">&#91;110:org.opendaylight.controller.sal-core-spi:1.3.0.Beryllium&#93;</span><br/> at org.opendaylight.controller.md.sal.dom.spi.ForwardingDOMDataReadWriteTransaction.delete(ForwardingDOMDataReadWriteTransaction.java:28)<span class="error">&#91;110:org.opendaylight.controller.sal-core-spi:1.3.0.Beryllium&#93;</span><br/> at org.opendaylight.controller.md.sal.binding.impl.AbstractWriteTransaction.doDelete(AbstractWriteTransaction.java:130)<span class="error">&#91;113:org.opendaylight.controller.sal-binding-broker-impl:1.3.0.Beryllium&#93;</span><br/> at org.opendaylight.controller.md.sal.binding.impl.BindingDOMWriteTransactionAdapter.delete(BindingDOMWriteTransactionAdapter.java:73)<span class="error">&#91;113:org.opendaylight.controller.sal-binding-broker-impl:1.3.0.Beryllium&#93;</span><br/> at org.opendaylight.bgpcep.bgp.topology.provider.LinkstateTopologyBuilder$NodeHolder.syncState(LinkstateTopologyBuilder.java:187)<span class="error">&#91;327:org.opendaylight.bgpcep.bgp-topology-provider:0.5.0.Beryllium&#93;</span><br/> at org.opendaylight.bgpcep.bgp.topology.provider.LinkstateTopologyBuilder$NodeHolder.access$600(LinkstateTopologyBuilder.java:160)<span class="error">&#91;327:org.opendaylight.bgpcep.bgp-topology-provider:0.5.0.Beryllium&#93;</span><br/> at org.opendaylight.bgpcep.bgp.topology.provider.LinkstateTopologyBuilder.putNode(LinkstateTopologyBuilder.java:478)<span class="error">&#91;327:org.opendaylight.bgpcep.bgp-topology-provider:0.5.0.Beryllium&#93;</span><br/> at org.opendaylight.bgpcep.bgp.topology.provider.LinkstateTopologyBuilder.createPrefix(LinkstateTopologyBuilder.java:825)<span class="error">&#91;327:org.opendaylight.bgpcep.bgp-topology-provider:0.5.0.Beryllium&#93;</span><br/> at org.opendaylight.bgpcep.bgp.topology.provider.LinkstateTopologyBuilder.createObject(LinkstateTopologyBuilder.java:853)<span class="error">&#91;327:org.opendaylight.bgpcep.bgp-topology-provider:0.5.0.Beryllium&#93;</span><br/> at org.opendaylight.bgpcep.bgp.topology.provider.LinkstateTopologyBuilder.createObject(LinkstateTopologyBuilder.java:121)<span class="error">&#91;327:org.opendaylight.bgpcep.bgp-topology-provider:0.5.0.Beryllium&#93;</span><br/> at org.opendaylight.bgpcep.bgp.topology.provider.AbstractTopologyBuilder.routeChanged(AbstractTopologyBuilder.java:165)<span class="error">&#91;327:org.opendaylight.bgpcep.bgp-topology-provider:0.5.0.Beryllium&#93;</span><br/> at org.opendaylight.bgpcep.bgp.topology.provider.AbstractTopologyBuilder.onDataTreeChanged(AbstractTopologyBuilder.java:134)<span class="error">&#91;327:org.opendaylight.bgpcep.bgp-topology-provider:0.5.0.Beryllium&#93;</span><br/> at org.opendaylight.controller.md.sal.binding.impl.BindingDOMDataTreeChangeListenerAdapter.onDataTreeChanged(BindingDOMDataTreeChangeListenerAdapter.java:41)<span class="error">&#91;113:org.opendaylight.controller.sal-binding-broker-impl:1.3.0.Beryllium&#93;</span><br/> at org.opendaylight.controller.cluster.datastore.DataTreeChangeListenerActor.dataChanged(DataTreeChangeListenerActor.java:53)<span class="error">&#91;138:org.opendaylight.controller.sal-distributed-datastore:1.3.0.Beryllium&#93;</span><br/> at org.opendaylight.controller.cluster.datastore.DataTreeChangeListenerActor.handleReceive(DataTreeChangeListenerActor.java:37)<span class="error">&#91;138:org.opendaylight.controller.sal-distributed-datastore:1.3.0.Beryllium&#93;</span><br/> at org.opendaylight.controller.cluster.common.actor.AbstractUntypedActor.onReceive(AbstractUntypedActor.java:34)<span class="error">&#91;134:org.opendaylight.controller.sal-clustering-commons:1.3.0.Beryllium&#93;</span><br/> at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:167)<span class="error">&#91;123:com.typesafe.akka.actor:2.3.14&#93;</span><br/> at akka.actor.Actor$class.aroundReceive(Actor.scala:467)<span class="error">&#91;123:com.typesafe.akka.actor:2.3.14&#93;</span><br/> at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:97)<span class="error">&#91;123:com.typesafe.akka.actor:2.3.14&#93;</span><br/> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)<span class="error">&#91;123:com.typesafe.akka.actor:2.3.14&#93;</span><br/> at akka.actor.ActorCell.invoke(ActorCell.scala:487)<span class="error">&#91;123:com.typesafe.akka.actor:2.3.14&#93;</span><br/> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)<span class="error">&#91;123:com.typesafe.akka.actor:2.3.14&#93;</span><br/> at akka.dispatch.Mailbox.run(Mailbox.scala:220)<span class="error">&#91;123:com.typesafe.akka.actor:2.3.14&#93;</span><br/> at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)<span class="error">&#91;123:com.typesafe.akka.actor:2.3.14&#93;</span><br/> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)<span class="error">&#91;120:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c&#93;</span><br/> at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)<span class="error">&#91;120:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c&#93;</span><br/> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)<span class="error">&#91;120:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c&#93;</span><br/> at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)<span class="error">&#91;120:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c&#93;</span></p> <p>Then in AbstractTopologyBuilder, it tries to cancel the PingPongTransaction, while the cancel fails as PingPongTransactionChain implements it as "Unsupported".</p> <p>2016-07-19 17:19:59,394 | ERROR | lt-dispatcher-27 | DataTreeChangeListenerActor | 138 - org.opendaylight.controller.sal-distributed-datastore - 1.3.0.Beryllium | Error notifying listener org.opendaylight.controller.md.sal.binding.impl.BindingDOMDataTreeChangeListenerAdapter@2b6c9375<br/> java.lang.UnsupportedOperationException: Transaction cancellation is not supported<br/> at org.opendaylight.controller.md.sal.dom.broker.impl.PingPongTransactionChain$4.cancel(PingPongTransactionChain.java:349)<span class="error">&#91;111:org.opendaylight.controller.sal-broker-impl:1.3.0.Beryllium&#93;</span><br/> at org.opendaylight.controller.md.sal.binding.impl.AbstractWriteTransaction.doCancel(AbstractWriteTransaction.java:138)<span class="error">&#91;113:org.opendaylight.controller.sal-binding-broker-impl:1.3.0.Beryllium&#93;</span><br/> at org.opendaylight.controller.md.sal.binding.impl.BindingDOMWriteTransactionAdapter.cancel(BindingDOMWriteTransactionAdapter.java:88)<span class="error">&#91;113:org.opendaylight.controller.sal-binding-broker-impl:1.3.0.Beryllium&#93;</span><br/> at org.opendaylight.bgpcep.bgp.topology.provider.AbstractTopologyBuilder.onDataTreeChanged(AbstractTopologyBuilder.java:137)<span class="error">&#91;327:org.opendaylight.bgpcep.bgp-topology-provider:0.5.0.Beryllium&#93;</span><br/> at org.opendaylight.controller.md.sal.binding.impl.BindingDOMDataTreeChangeListenerAdapter.onDataTreeChanged(BindingDOMDataTreeChangeListenerAdapter.java:41)<span class="error">&#91;113:org.opendaylight.controller.sal-binding-broker-impl:1.3.0.Beryllium&#93;</span><br/> at org.opendaylight.controller.cluster.datastore.DataTreeChangeListenerActor.dataChanged(DataTreeChangeListenerActor.java:53)<span class="error">&#91;138:org.opendaylight.controller.sal-distributed-datastore:1.3.0.Beryllium&#93;</span><br/> at org.opendaylight.controller.cluster.datastore.DataTreeChangeListenerActor.handleReceive(DataTreeChangeListenerActor.java:37)<span class="error">&#91;138:org.opendaylight.controller.sal-distributed-datastore:1.3.0.Beryllium&#93;</span><br/> at org.opendaylight.controller.cluster.common.actor.AbstractUntypedActor.onReceive(AbstractUntypedActor.java:34)<span class="error">&#91;134:org.opendaylight.controller.sal-clustering-commons:1.3.0.Beryllium&#93;</span><br/> at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:167)<span class="error">&#91;123:com.typesafe.akka.actor:2.3.14&#93;</span><br/> at akka.actor.Actor$class.aroundReceive(Actor.scala:467)<span class="error">&#91;123:com.typesafe.akka.actor:2.3.14&#93;</span><br/> at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:97)<span class="error">&#91;123:com.typesafe.akka.actor:2.3.14&#93;</span><br/> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)<span class="error">&#91;123:com.typesafe.akka.actor:2.3.14&#93;</span><br/> at akka.actor.ActorCell.invoke(ActorCell.scala:487)<span class="error">&#91;123:com.typesafe.akka.actor:2.3.14&#93;</span><br/> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)<span class="error">&#91;123:com.typesafe.akka.actor:2.3.14&#93;</span><br/> at akka.dispatch.Mailbox.run(Mailbox.scala:220)<span class="error">&#91;123:com.typesafe.akka.actor:2.3.14&#93;</span><br/> at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)<span class="error">&#91;123:com.typesafe.akka.actor:2.3.14&#93;</span><br/> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)<span class="error">&#91;120:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c&#93;</span><br/> at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)<span class="error">&#91;120:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c&#93;</span><br/> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)<span class="error">&#91;120:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c&#93;</span><br/> at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)<span class="error">&#91;120:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c&#93;</span></p> <p>Eventually the transaction is locked and causing new transaction fail to be created.</p> <p>2016-07-19 17:19:59,395 | ERROR | lt-dispatcher-27 | DataTreeChangeListenerActor | 138 - org.opendaylight.controller.sal-distributed-datastore - 1.3.0.Beryllium | Error notifying listener org.opendaylight.controller.md.sal.binding.impl.BindingDOMDataTreeChangeListenerAdapter@2b6c9375<br/> java.lang.IllegalStateException: New transaction PingPongTransaction</p> {delegate=org.opendaylight.controller.cluster.databroker.DOMBrokerReadWriteTransaction@14239319} <p> raced with transacion PingPongTransaction</p> {delegate=org.opendaylight.controller.cluster.databroker.DOMBrokerReadWriteTransaction@2b8e58a3} <p> at org.opendaylight.controller.md.sal.dom.broker.impl.PingPongTransactionChain.slowAllocateTransaction(PingPongTransactionChain.java:136)<span class="error">&#91;111:org.opendaylight.controller.sal-broker-impl:1.3.0.Beryllium&#93;</span><br/> at org.opendaylight.controller.md.sal.dom.broker.impl.PingPongTransactionChain.allocateTransaction(PingPongTransactionChain.java:148)<span class="error">&#91;111:org.opendaylight.controller.sal-broker-impl:1.3.0.Beryllium&#93;</span><br/> at org.opendaylight.controller.md.sal.dom.broker.impl.PingPongTransactionChain.newReadWriteTransaction(PingPongTransactionChain.java:328)<span class="error">&#91;111:org.opendaylight.controller.sal-broker-impl:1.3.0.Beryllium&#93;</span><br/> at org.opendaylight.controller.md.sal.binding.impl.BindingDOMTransactionChainAdapter.newReadWriteTransaction(BindingDOMTransactionChainAdapter.java:62)<span class="error">&#91;113:org.opendaylight.controller.sal-binding-broker-impl:1.3.0.Beryllium&#93;</span><br/> at org.opendaylight.bgpcep.bgp.topology.provider.AbstractTopologyBuilder.onDataTreeChanged(AbstractTopologyBuilder.java:130)<span class="error">&#91;327:org.opendaylight.bgpcep.bgp-topology-provider:0.5.0.Beryllium&#93;</span><br/> at org.opendaylight.controller.md.sal.binding.impl.BindingDOMDataTreeChangeListenerAdapter.onDataTreeChanged(BindingDOMDataTreeChangeListenerAdapter.java:41)<span class="error">&#91;113:org.opendaylight.controller.sal-binding-broker-impl:1.3.0.Beryllium&#93;</span><br/> at org.opendaylight.controller.cluster.datastore.DataTreeChangeListenerActor.dataChanged(DataTreeChangeListenerActor.java:53)<span class="error">&#91;138:org.opendaylight.controller.sal-distributed-datastore:1.3.0.Beryllium&#93;</span><br/> at org.opendaylight.controller.cluster.datastore.DataTreeChangeListenerActor.handleReceive(DataTreeChangeListenerActor.java:37)<span class="error">&#91;138:org.opendaylight.controller.sal-distributed-datastore:1.3.0.Beryllium&#93;</span><br/> at org.opendaylight.controller.cluster.common.actor.AbstractUntypedActor.onReceive(AbstractUntypedActor.java:34)<span class="error">&#91;134:org.opendaylight.controller.sal-clustering-commons:1.3.0.Beryllium&#93;</span><br/> at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:167)<span class="error">&#91;123:com.typesafe.akka.actor:2.3.14&#93;</span><br/> at akka.actor.Actor$class.aroundReceive(Actor.scala:467)<span class="error">&#91;123:com.typesafe.akka.actor:2.3.14&#93;</span><br/> at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:97)<span class="error">&#91;123:com.typesafe.akka.actor:2.3.14&#93;</span><br/> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)<span class="error">&#91;123:com.typesafe.akka.actor:2.3.14&#93;</span><br/> at akka.actor.ActorCell.invoke(ActorCell.scala:487)<span class="error">&#91;123:com.typesafe.akka.actor:2.3.14&#93;</span><br/> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)<span class="error">&#91;123:com.typesafe.akka.actor:2.3.14&#93;</span><br/> at akka.dispatch.Mailbox.run(Mailbox.scala:220)<span class="error">&#91;123:com.typesafe.akka.actor:2.3.14&#93;</span><br/> at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)<span class="error">&#91;123:com.typesafe.akka.actor:2.3.14&#93;</span><br/> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)<span class="error">&#91;120:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c&#93;</span><br/> at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)<span class="error">&#91;120:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c&#93;</span><br/> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)<span class="error">&#91;120:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c&#93;</span><br/> at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)<span class="error">&#91;120:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c&#93;</span></p> <p>Attachment karaf.log.77.zip has been added with description: karaf log</p> <p>Hi Kevin, regarding your last comment not sure if the attachment correspond to this bug or not..<br/> If it something new, please fill a new bug, notice please, that when a new bug is opened we expect further information like steps to reproduce the bug, what were you testing, version of controller, if you were working on a single node, clustering, 3 node clustering, etc... As more information better are the chance to trace the issue quickly.</p> <p>Regards</p> <p>Hi Claudio,</p> <p>Sorry for the unclear description. The last attachment (karaf log) is for this bug. The problem happens in a 3-node cluster. It happened on Beryllium release. But I suppose it will also happen on SR2 or Boron as the related code hasn't changed.</p> <p>The following is my observation:</p> <p>1. ODL is trying to sync BGP-LS topology with other nodes.</p> <p>2. ODL tried to get member-2 shard transaction:<br/> 2016-07-19 17:19:45,617 | WARN | lt-dispatcher-27 | OperationLimiter | 138 - org.opendaylight.controller.sal-distributed-datastore - 1.3.0.Beryllium | Failed to acquire operation permit for transaction member-3-chn-25-txn-335-1468948769013<br/> 2016-07-19 17:19:50,618 | WARN | lt-dispatcher-27 | OperationLimiter | 138 - org.opendaylight.controller.sal-distributed-datastore - 1.3.0.Beryllium | Failed to acquire operation permit for transaction member-3-chn-25-txn-335-1468948769013<br/> 2016-07-19 17:19:55,648 | WARN | lt-dispatcher-27 | OperationLimiter | 138 - org.opendaylight.controller.sal-distributed-datastore - 1.3.0.Beryllium | Failed to acquire operation permit for transaction member-3-chn-25-txn-335-1468948769013<br/> 2016-07-19 17:19:59,027 | ERROR | ult-dispatcher-4 | TransactionChainProxy | 138 - org.opendaylight.controller.sal-distributed-datastore - 1.3.0.Beryllium | Tx: member-3-chn-25-txn-335-1468948769013 - ready future failed for previous Tx member-3-chn-25-txn-335-1468948769013<br/> 2016-07-19 17:19:59,027 | WARN | ult-dispatcher-2 | ConcurrentDOMDataBroker | 138 - org.opendaylight.controller.sal-distributed-datastore - 1.3.0.Beryllium | Tx: DOM-CHAIN-12-333 Error during phase CAN_COMMIT, starting Abort<br/> akka.pattern.AskTimeoutException: Ask timed out on [ActorSelection<span class="error">&#91;Anchor(akka.tcp://opendaylight-cluster-data@mtsnjv9sdbc12-eth1-0.sdcp2.aic.cip.att.com:2550/), Path(/user/shardmanager-operational/member-2-shard-topology-operational/shard-member-3-chn-25-txn-334-1468948738750#-1973141357)&#93;</span>] after <span class="error">&#91;30000 ms&#93;</span><br/> at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:334)<span class="error">&#91;123:com.typesafe.akka.actor:2.3.14&#93;</span><br/> at akka.actor.Scheduler$$anon$7.run(Scheduler.scala:117)<span class="error">&#91;123:com.typesafe.akka.actor:2.3.14&#93;</span><br/> at scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:599)<span class="error">&#91;120:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c&#93;</span><br/> at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)<span class="error">&#91;120:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c&#93;</span><br/> at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:597)<span class="error">&#91;120:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c&#93;</span><br/> at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(Scheduler.scala:467)<span class="error">&#91;123:com.typesafe.akka.actor:2.3.14&#93;</span><br/> at akka.actor.LightArrayRevolverScheduler$$anon$8.executeBucket$1(Scheduler.scala:419)<span class="error">&#91;123:com.typesafe.akka.actor:2.3.14&#93;</span><br/> at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:423)<span class="error">&#91;123:com.typesafe.akka.actor:2.3.14&#93;</span><br/> at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)<span class="error">&#91;123:com.typesafe.akka.actor:2.3.14&#93;</span><br/> at java.lang.Thread.run(Thread.java:745)<span class="error">&#91;:1.7.0_76&#93;</span></p> <p>3. The shard keeps timing out initially (in the log, it times out on member-2)</p> <p>4. Eventually, the sync transaction fails due to the previous transaction timed out:<br/> 2016-07-19 17:19:59,393 | WARN | lt-dispatcher-27 | AbstractTopologyBuilder | 327 - org.opendaylight.bgpcep.bgp-topology-provider - 0.5.0.Beryllium | Data change org.opendaylight.controller.md.sal.binding.impl.LazyDataTreeModification@64d3333d was not completely propagated to listener org.opendaylight.bgpcep.bgp.topology.provider.LinkstateTopologyBuilder@4907aa42, aborting<br/> java.lang.IllegalStateException: Previous transaction member-3-chn-25-txn-335-1468948769013 is not ready yet</p> <p>5. Then the BGP topology provider tried to cancel the transaction, however, the cancel() is not implemented for PingPongTransactionChain (this could either be a PingPongTransactionChain bug, that the cancel() should be implemented properly, or a AbstractTopologyBuilder bug, that it should not invoke the cancel() method when a transaction fails):<br/> 2016-07-19 17:19:59,394 | ERROR | lt-dispatcher-27 | DataTreeChangeListenerActor | 138 - org.opendaylight.controller.sal-distributed-datastore - 1.3.0.Beryllium | Error notifying listener org.opendaylight.controller.md.sal.binding.impl.BindingDOMDataTreeChangeListenerAdapter@2b6c9375<br/> java.lang.UnsupportedOperationException: Transaction cancellation is not supported</p> <p>6. Consequently, all the following new PingPongTransaction cannot be created. Because the previous PingPongTransaction was not properly canceled, the PingPongTransactionChain was never unlocked. All the new PingPongTransaction was said "raced" with this problematic PingPongTransaction. Engineering believes that the data change on BGP-LS will never happen after this.</p> <p>In summary, the AbstractTopologyBuilder, it is trying to cancel a PingPongTransactionChain, which is not supported. That leads to the problem.</p> <p>Please let me know if you have any other thought.</p> <p>Confirmed with Robert, </p> <p>"PingPongDataBroker does not support cancelling transaction. It really<br/> cannot, as it packs multiple transactions into one.</p> <p>If a submitted transaction fails, the entire TransactionChain becomes<br/> unusable (all subsequent transactions will fail). When that happens, the<br/> TransactionChain needs to be close()d and a new one needs to be<br/> instantiated."</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/42278/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/42278/</a></p> <p>PingPong side of fix: <a href="https://git.opendaylight.org/gerrit/43058" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/43058</a></p> <p>stable/boron: <a href="https://git.opendaylight.org/gerrit/#/c/43928/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/43928/</a></p> Blocks BGPCEP-491 BGPCEP-493 Development External issue ID 6111 External issue URL Issue Type ODL SR Target Milestone Rank 0|i02cl3: