[BGPCEP-237] PingPongTransaction race during BGP stress testing (RIB only)

Sun, 3 Mar 2019 11:50:27 +0000

Afects build distribution-karaf-0.3.0-20150610.152928-2445.tar.gz

Previous build were able to ingest large number of fake routes (with CDS and topology). But this build, when connected to real Internet feed, showed empty topology (while bgp-rib looked OK at first glance).

Karaf log is full of these exceptions (karaf.log has over 200 MB):

2015-06-10 17:22:50,914 | ERROR | lt-dispatcher-14 | DataTreeChangeListenerActor | 179 - org.opendaylight.controller.sal-distributed-datastore - 1.2.0.SNAPSHOT | Error notifying listener org.opendaylight.protocol.bgp.rib.impl.LocRibWriter@2623e71
java.lang.IllegalStateException: New transaction PingPongTransaction

{delegate=org.opendaylight.controller.cluster.databroker.DOMBrokerReadWriteTransaction@b043cf3}

raced with transacion PingPongTransaction

{delegate=org.opendaylight.controller.cluster.databroker.DOMBrokerReadWriteTransaction@11c1f060}

at org.opendaylight.controller.md.sal.dom.broker.impl.PingPongTransactionChain.slowAllocateTransaction(PingPongTransactionChain.java:132)[154:org.opendaylight.controller.sal-broker-impl:1.2.0.SNAPSHOT]
at org.opendaylight.controller.md.sal.dom.broker.impl.PingPongTransactionChain.allocateTransaction(PingPongTransactionChain.java:144)[154:org.opendaylight.controller.sal-broker-impl:1.2.0.SNAPSHOT]
at org.opendaylight.controller.md.sal.dom.broker.impl.PingPongTransactionChain.newReadWriteTransaction(PingPongTransactionChain.java:311)[154:org.opendaylight.controller.sal-broker-impl:1.2.0.SNAPSHOT]
at org.opendaylight.controller.md.sal.dom.broker.impl.PingPongTransactionChain.newWriteOnlyTransaction(PingPongTransactionChain.java:342)[154:org.opendaylight.controller.sal-broker-impl:1.2.0.SNAPSHOT]
at org.opendaylight.protocol.bgp.rib.impl.LocRibWriter.onDataTreeChanged(LocRibWriter.java:115)[235:org.opendaylight.bgpcep.bgp-rib-impl:0.4.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.DataTreeChangeListenerActor.dataChanged(DataTreeChangeListenerActor.java:53)[179:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.DataTreeChangeListenerActor.handleReceive(DataTreeChangeListenerActor.java:37)[179:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.common.actor.AbstractUntypedActor.onReceive(AbstractUntypedActor.java:34)[171:org.opendaylight.controller.sal-clustering-commons:1.2.0.SNAPSHOT]
at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:167)[164:com.typesafe.akka.actor:2.3.10]
at akka.actor.Actor$class.aroundReceive(Actor.scala:467)[164:com.typesafe.akka.actor:2.3.10]
at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:97)[164:com.typesafe.akka.actor:2.3.10]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)[164:com.typesafe.akka.actor:2.3.10]
at akka.actor.ActorCell.invoke(ActorCell.scala:487)[164:com.typesafe.akka.actor:2.3.10]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)[164:com.typesafe.akka.actor:2.3.10]
at akka.dispatch.Mailbox.run(Mailbox.scala:221)[164:com.typesafe.akka.actor:2.3.10]
at akka.dispatch.Mailbox.exec(Mailbox.scala:231)[164:com.typesafe.akka.actor:2.3.10]
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)[161:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b]
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:1253)[161:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b]
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1346)[161:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b]
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)[161:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b]
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)[161:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b]

When run with topology disabled, karaf.log is not hundreds of megabytes long anymore, but there are still few dozens of those exceptions.

OpenDaylight JIRA

[BGPCEP-237] PingPongTransaction race during BGP stress testing (RIB only)