Clustering: Store tree and candidate base differ

This exception is seen during L2Switch testing. As per Robert Varga this can happen when " the datatree has been changed between the time the transaction was prepared (pre-commit) and actually committed"

I do not see how it is possible for this to happen because both in the Distributed Data store and the In Memory Datastore we prevent modification when the transaction is in the ready state. Preliminarily this appears to be a bug in InMemoryDataTree.

The one known way to reproduce this issue is by running the integration distribution with the L2 switch and doing a ping all with a 6 switch 2 host configuration.

2014-09-23 00:22:43,820 | WARN | ult-dispatcher-3 | OneForOneStrategy | 168 - com.typesafe.akka.slf4j - 2.3.4 | Store tree org.opendaylight.yangtools.yang.data.api.schema.tree.spi.MaterializedContainerNode@5b70715b and candidate base org.opendaylight.yangtools.yang.data.api.schema.tree.spi.MaterializedContainerNode@33223c43 differ.

2014-09-23 00:22:43,820 | WARN | ult-dispatcher-3 | ShardManager | 168 - com.typesafe.akka.slf4j - 2.3.4 | Supervisor Strategy of resume applied
at com.google.common.base.Preconditions.checkState(Preconditions.java:176)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.InMemoryDataTree.commit(InMemoryDataTree.java:119)
at org.opendaylight.controller.md.sal.dom.store.impl.InMemoryDOMDataStore$ThreePhaseCommitImpl.commit(InMemoryDOMDataStore.java:434)
at org.opendaylight.controller.md.sal.dom.store.impl.InMemoryDOMDataStore$ChainedTransactionCommitImpl.commit(InMemoryDOMDataStore.java:355)
at org.opendaylight.controller.cluster.datastore.Shard.commit(Shard.java:349)
at org.opendaylight.controller.cluster.datastore.Shard.applyState(Shard.java:534)
at org.opendaylight.controller.cluster.raft.RaftActor.onReceiveCommand(RaftActor.java:267)
at org.opendaylight.controller.cluster.datastore.Shard.onReceiveCommand(Shard.java:232)
at akka.persistence.UntypedPersistentActor.onReceive(Eventsourced.scala:430)
at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:167)
at akka.persistence.Recovery$State$class.process(Recovery.scala:30)
at akka.persistence.ProcessorImpl$$anon$2.process(Processor.scala:103)
at akka.persistence.ProcessorImpl$$anon$2.aroundReceive(Processor.scala:114)
at akka.persistence.Recovery$class.aroundReceive(Recovery.scala:256)
at akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(Eventsourced.scala:428)
at akka.persistence.Eventsourced$$anon$2.doAroundReceive(Eventsourced.scala:82)
at akka.persistence.Eventsourced$$anon$2.aroundReceive(Eventsourced.scala:78)
at akka.persistence.Eventsourced$class.aroundReceive(Eventsourced.scala:369)
at akka.persistence.UntypedPersistentActor.aroundReceive(Eventsourced.scala:428)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
at akka.dispatch.Mailbox.run(Mailbox.scala:220)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)