This happened in a longevity job. It was using single transaction writer and moving leader by cluster-admin:make-leader-local. 6 iterations passed correctly, but then (when moving leader from member-2 to member-1 where the writer was), transaction writer returned [0] TransactionCommitFailedException (30 seconds after the make-leader-local call). The cause is new for me, so I am opening a Bug.

Caused by: org.opendaylight.controller.cluster.databroker.actors.dds.BouncingReconnectForwarder$CohortNotFoundException: Cohort for LocalHistoryIdentifier{client=ClientIdentifier

, history=4, cookie=0} not found\n\tat org.opendaylight.controller.cluster.databroker.actors.dds.BouncingReconnectForwarder.findCohort(BouncingReconnectForwarder.java:91)\n\tat org.opendaylight.controller.cluster.databroker.actors.dds.BouncingReconnectForwarder.forwardEntry(BouncingReconnectForwarder.java:61)\n\tat org.opendaylight.controller.cluster.access.client.TransmitQueue.enqueue(TransmitQueue.java:192)\n\tat org.opendaylight.controller.cluster.access.client.AbstractClientConnection.enqueueEntry(AbstractClientConnection.java:172)\n\tat org.opendaylight.controller.cluster.access.client.AbstractClientConnection.sendEntry(AbstractClientConnection.java:215)\n\tat org.opendaylight.controller.cluster.access.client.SimpleReconnectForwarder.forwardEntry(SimpleReconnectForwarder.java:18)\n\tat org.opendaylight.controller.cluster.access.client.TransmitQueue.enqueue(TransmitQueue.java:192)\n\tat org.opendaylight.controller.cluster.access.client.AbstractClientConnection.enqueueEntry(AbstractClientConnection.java:172)\n\tat org.opendaylight.controller.cluster.access.client.AbstractClientConnection.sendEntry(AbstractClientConnection.java:215)\n\tat org.opendaylight.controller.cluster.access.client.AbstractClientConnection.sendRequest(AbstractClientConnection.java:140)\n\tat org.opendaylight.controller.cluster.databroker.actors.dds.ProxyHistory.sendRequest(ProxyHistory.java:449)\n\tat org.opendaylight.controller.cluster.databroker.actors.dds.AbstractProxyTransaction.sendRequest(AbstractProxyTransaction.java:326)\n\tat org.opendaylight.controller.cluster.databroker.actors.dds.AbstractProxyTransaction.directCommit(AbstractProxyTransaction.java:463)\n\tat org.opendaylight.controller.cluster.databroker.actors.dds.DirectTransactionCommitCohort.canCommit(DirectTransactionCommitCohort.java:31)\n\tat org.opendaylight.controller.cluster.databroker.DOMStoreThreePhaseCommitCohortAdaptor.canCommit(DOMStoreThreePhaseCommitCohortAdaptor.java:31)\n\tat org.opendaylight.controller.cluster.databroker.ConcurrentDOMDataBroker.doCanCommit(ConcurrentDOMDataBroker.java:131)\n\t... 12 more

Karaf.log [1] contains an error which I think is common for multiple Robot symptoms, so I will open a separate Bug for that when I confirm it is really there for multiple symptoms. For now, here is the error:

2017-06-25 16:34:35,091 | ERROR | lt-dispatcher-27 | OneForOneStrategy | 174 - com.typesafe.akka.slf4j - 2.4.18 | Transaction member-1-datastore-config-fe-0-chn-4-txn-5084-0 has already been sealed
java.lang.IllegalStateException: Transaction member-1-datastore-config-fe-0-chn-4-txn-5084-0 has already been sealed
at com.google.common.base.Preconditions.checkState(Preconditions.java:197)[57:com.google.guava:18.0.0]
at org.opendaylight.controller.cluster.databroker.actors.dds.AbstractProxyTransaction.checkNotSealed(AbstractProxyTransaction.java:363)[199:org.opendaylight.controller.sal-distributed-datastore:1.5.1.SNAPSHOT]
at org.opendaylight.controller.cluster.databroker.actors.dds.AbstractProxyTransaction.write(AbstractProxyTransaction.java:304)[199:org.opendaylight.controller.sal-distributed-datastore:1.5.1.SNAPSHOT]
at org.opendaylight.controller.cluster.databroker.actors.dds.LocalReadWriteProxyTransaction$1.write(LocalReadWriteProxyTransaction.java:192)[199:org.opendaylight.controller.sal-distributed-datastore:1.5.1.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.InMemoryDataTreeModification.applyNode(InMemoryDataTreeModification.java:237)[78:org.opendaylight.yangtools.yang-data-impl:1.1.1.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.InMemoryDataTreeModification.applyChildren(InMemoryDataTreeModification.java:212)[78:org.opendaylight.yangtools.yang-data-impl:1.1.1.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.InMemoryDataTreeModification.applyNode(InMemoryDataTreeModification.java:234)[78:org.opendaylight.yangtools.yang-data-impl:1.1.1.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.InMemoryDataTreeModification.applyChildren(InMemoryDataTreeModification.java:212)[78:org.opendaylight.yangtools.yang-data-impl:1.1.1.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.InMemoryDataTreeModification.applyNode(InMemoryDataTreeModification.java:234)[78:org.opendaylight.yangtools.yang-data-impl:1.1.1.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.InMemoryDataTreeModification.applyChildren(InMemoryDataTreeModification.java:212)[78:org.opendaylight.yangtools.yang-data-impl:1.1.1.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.InMemoryDataTreeModification.applyNode(InMemoryDataTreeModification.java:234)[78:org.opendaylight.yangtools.yang-data-impl:1.1.1.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.InMemoryDataTreeModification.applyChildren(InMemoryDataTreeModification.java:212)[78:org.opendaylight.yangtools.yang-data-impl:1.1.1.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.InMemoryDataTreeModification.applyNode(InMemoryDataTreeModification.java:234)[78:org.opendaylight.yangtools.yang-data-impl:1.1.1.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.InMemoryDataTreeModification.applyToCursor(InMemoryDataTreeModification.java:248)[78:org.opendaylight.yangtools.yang-data-impl:1.1.1.SNAPSHOT]
at org.opendaylight.controller.cluster.databroker.actors.dds.LocalReadWriteProxyTransaction.flushState(LocalReadWriteProxyTransaction.java:189)[199:org.opendaylight.controller.sal-distributed-datastore:1.5.1.SNAPSHOT]
at org.opendaylight.controller.cluster.databroker.actors.dds.AbstractProxyTransaction.replayMessages(AbstractProxyTransaction.java:691)[199:org.opendaylight.controller.sal-distributed-datastore:1.5.1.SNAPSHOT]
at org.opendaylight.controller.cluster.databroker.actors.dds.ProxyHistory$ReconnectCohort.replayRequests(ProxyHistory.java:240)[199:org.opendaylight.controller.sal-distributed-datastore:1.5.1.SNAPSHOT]
at org.opendaylight.controller.cluster.databroker.actors.dds.AbstractClientHistory$1.replayRequests(AbstractClientHistory.java:311)[199:org.opendaylight.controller.sal-distributed-datastore:1.5.1.SNAPSHOT]
at org.opendaylight.controller.cluster.databroker.actors.dds.AbstractDataStoreClientBehavior.lambda$connectionUp$0(AbstractDataStoreClientBehavior.java:147)[199:org.opendaylight.controller.sal-distributed-datastore:1.5.1.SNAPSHOT]
at org.opendaylight.controller.cluster.access.client.ClientActorBehavior.backendConnectFinished(ClientActorBehavior.java:296)[197:org.opendaylight.controller.cds-access-client:1.1.1.SNAPSHOT]
at org.opendaylight.controller.cluster.access.client.ClientActorBehavior.lambda$null$1(ClientActorBehavior.java:359)[197:org.opendaylight.controller.cds-access-client:1.1.1.SNAPSHOT]
at org.opendaylight.controller.cluster.access.client.ClientActorBehavior.onReceiveCommand(ClientActorBehavior.java:121)[197:org.opendaylight.controller.cds-access-client:1.1.1.SNAPSHOT]
at org.opendaylight.controller.cluster.access.client.ClientActorBehavior.onReceiveCommand(ClientActorBehavior.java:45)[197:org.opendaylight.controller.cds-access-client:1.1.1.SNAPSHOT]
at org.opendaylight.controller.cluster.access.client.AbstractClientActor.onReceiveCommand(AbstractClientActor.java:59)[197:org.opendaylight.controller.cds-access-client:1.1.1.SNAPSHOT]
at akka.persistence.UntypedPersistentActor.onReceive(PersistentActor.scala:170)[180:com.typesafe.akka.persistence:2.4.18]
at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:165)[173:com.typesafe.akka.actor:2.4.18]
at akka.actor.Actor$class.aroundReceive(Actor.scala:502)[173:com.typesafe.akka.actor:2.4.18]
at akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(PersistentActor.scala:168)[180:com.typesafe.akka.persistence:2.4.18]
at akka.persistence.Eventsourced$$anon$1.stateReceive(Eventsourced.scala:727)[180:com.typesafe.akka.persistence:2.4.18]
at akka.persistence.Eventsourced$class.aroundReceive(Eventsourced.scala:183)[180:com.typesafe.akka.persistence:2.4.18]
at akka.persistence.UntypedPersistentActor.aroundReceive(PersistentActor.scala:168)[180:com.typesafe.akka.persistence:2.4.18]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)[173:com.typesafe.akka.actor:2.4.18]
at akka.actor.ActorCell.invoke(ActorCell.scala:495)[173:com.typesafe.akka.actor:2.4.18]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)[173:com.typesafe.akka.actor:2.4.18]
at akka.dispatch.Mailbox.run(Mailbox.scala:224)[173:com.typesafe.akka.actor:2.4.18]
at akka.dispatch.Mailbox.exec(Mailbox.scala:234)[173:com.typesafe.akka.actor:2.4.18]
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)[169:org.scala-lang.scala-library:2.11.11.v20170413-090219-8a413ba7cc]
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)[169:org.scala-lang.scala-library:2.11.11.v20170413-090219-8a413ba7cc]
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)[169:org.scala-lang.scala-library:2.11.11.v20170413-090219-8a413ba7cc]
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)[169:org.scala-lang.scala-library:2.11.11.v20170413-090219-8a413ba7cc]

[0] https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-ddb-expl-lead-movement-longevity-only-carbon/12/log.html.gz#s1-s2-t1-k2-k1-k1-k1-k1-k1-k1-k2-k1-k1-k2-k10
[1] https://logs.opendaylight.org/releng/jenkins092/controller-csit-3node-ddb-expl-lead-movement-longevity-only-carbon/12/odl1_karaf.log.gz