[CONTROLLER-1218] Clustering : No cohort entry found for transaction exception occurs often during Netconf scale test

Thu, 11 Jun 2015 14:08:48 +0000

When running the netconf scale test with 10,000 devices the following exception is seen. This may be one of the root causes for the test failing sometimes,

2015-03-17 13:32:04,097 | WARN | WriteTxCommit-0 | DOMDataCommitCoordinatorImpl | 144 - org.opendaylight.controller.sal-broker-impl - 1.1.1.Helium-SR1-00004_1-SNAPSHOT | Tx: DOM-2450 Error during phase CAN_COMMIT, starting Abort
TransactionCommitFailedException

{message=canCommit execution failed, errorList=[RpcError [message=canCommit execution failed, severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=java.lang.IllegalStateException: member-1-shard-inventory-operational: No cohort entry found for transaction member-1-txn-4914]]}

at org.opendaylight.controller.md.sal.dom.broker.impl.TransactionCommitFailedExceptionMapper.newWithCause(TransactionCommitFailedExceptionMapper.java:37)[144:org.opendaylight.controller.sal-broker-impl:1.1.1.Helium-SR1-00004_1-SNAPSHOT]
at org.opendaylight.controller.md.sal.dom.broker.impl.TransactionCommitFailedExceptionMapper.newWithCause(TransactionCommitFailedExceptionMapper.java:18)[144:org.opendaylight.controller.sal-broker-impl:1.1.1.Helium-SR1-00004_1-SNAPSHOT]
at org.opendaylight.yangtools.util.concurrent.ExceptionMapper.apply(ExceptionMapper.java:80)[65:org.opendaylight.yangtools.util:0.6.3.Helium-SR1-00004_1-SNAPSHOT]
at org.opendaylight.controller.md.sal.dom.broker.impl.DOMDataCommitCoordinatorImpl$CommitCoordinationTask.canCommitBlocking(DOMDataCommitCoordinatorImpl.java:186)[144:org.opendaylight.controller.sal-broker-impl:1.1.1.Helium-SR1-00004_1-SNAPSHOT]
at org.opendaylight.controller.md.sal.dom.broker.impl.DOMDataCommitCoordinatorImpl$CommitCoordinationTask.call(DOMDataCommitCoordinatorImpl.java:150)[144:org.opendaylight.controller.sal-broker-impl:1.1.1.Helium-SR1-00004_1-SNAPSHOT]
at org.opendaylight.controller.md.sal.dom.broker.impl.DOMDataCommitCoordinatorImpl$CommitCoordinationTask.call(DOMDataCommitCoordinatorImpl.java:127)[144:org.opendaylight.controller.sal-broker-impl:1.1.1.Helium-SR1-00004_1-SNAPSHOT]
at org.opendaylight.yangtools.util.concurrent.DeadlockDetectingListeningExecutorService$2.call(DeadlockDetectingListeningExecutorService.java:192)[65:org.opendaylight.yangtools.util:0.6.3.Helium-SR1-00004_1-SNAPSHOT]
at java.util.concurrent.FutureTask.run(Unknown Source)[:1.7.0_67]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)[:1.7.0_67]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)[:1.7.0_67]
at java.lang.Thread.run(Unknown Source)[:1.7.0_67]
Caused by: java.lang.IllegalStateException: member-1-shard-inventory-operational: No cohort entry found for transaction member-1-txn-4914
at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.handleCanCommit(ShardCommitCoordinator.java:96)[329:org.opendaylight.controller.sal-distributed-datastore:1.1.1.Helium-SR1-00004_1-SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.Shard.handleCanCommitTransaction(Shard.java:450)[329:org.opendaylight.controller.sal-distributed-datastore:1.1.1.Helium-SR1-00004_1-SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.Shard.onReceiveCommand(Shard.java:278)[329:org.opendaylight.controller.sal-distributed-datastore:1.1.1.Helium-SR1-00004_1-SNAPSHOT]
at akka.persistence.UntypedPersistentActor.onReceive(Eventsourced.scala:430)[319:com.typesafe.akka.persistence.experimental:2.3.4]
at org.opendaylight.controller.cluster.common.actor.MeteringBehavior.apply(MeteringBehavior.java:96)[321:org.opendaylight.controller.sal-clustering-commons:1.1.1.Helium-SR1-00004_1-SNAPSHOT]
at akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:534)[314:com.typesafe.akka.actor:2.3.4]
at akka.persistence.Recovery$State$class.process(Recovery.scala:30)[319:com.typesafe.akka.persistence.experimental:2.3.4]
at akka.persistence.ProcessorImpl$$anon$2.process(Processor.scala:103)[319:com.typesafe.akka.persistence.experimental:2.3.4]
at akka.persistence.ProcessorImpl$$anon$2.aroundReceive(Processor.scala:114)[319:com.typesafe.akka.persistence.experimental:2.3.4]
at akka.persistence.Recovery$class.aroundReceive(Recovery.scala:256)[319:com.typesafe.akka.persistence.experimental:2.3.4]
at akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(Eventsourced.scala:428)[319:com.typesafe.akka.persistence.experimental:2.3.4]
at akka.persistence.Eventsourced$$anon$2.doAroundReceive(Eventsourced.scala:82)[319:com.typesafe.akka.persistence.experimental:2.3.4]
at akka.persistence.Eventsourced$$anon$2.aroundReceive(Eventsourced.scala:78)[319:com.typesafe.akka.persistence.experimental:2.3.4]
at akka.persistence.Eventsourced$class.aroundReceive(Eventsourced.scala:369)[319:com.typesafe.akka.persistence.experimental:2.3.4]
at akka.persistence.UntypedPersistentActor.aroundReceive(Eventsourced.scala:428)[319:com.typesafe.akka.persistence.experimental:2.3.4]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)[314:com.typesafe.akka.actor:2.3.4]
at akka.actor.ActorCell.invoke(ActorCell.scala:487)[314:com.typesafe.akka.actor:2.3.4]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)[314:com.typesafe.akka.actor:2.3.4]
at akka.dispatch.Mailbox.run(Mailbox.scala:220)[314:com.typesafe.akka.actor:2.3.4]
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)[314:com.typesafe.akka.actor:2.3.4]
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)[311:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b]
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)[311:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b]
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)[311:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b]
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)[311:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b]

OpenDaylight JIRA

[CONTROLLER-1218] Clustering : No cohort entry found for transaction exception occurs often during Netconf scale test