ERROR Log Observations - CSIT (sal datastore related)

The 3node Netvirt Openstack Tests are failing, In this report, we would like to capture the clustering related errors seen in karaf log for further analysis.

Some datastore related errors observed in karaf.log from all the 3 nodes in Netvirt. I guess, not all of the messages are actual failures, But it will be better to have some explanation for the failure being reported.

references:
https://jenkins.opendaylight.org/releng/view/netvirt/job/netvirt-csit-3node-openstack-liberty-openstack-beryllium/57/artifact/odl1_karaf.log.tar.xz

https://jenkins.opendaylight.org/releng/view/netvirt/job/netvirt-csit-3node-openstack-liberty-openstack-beryllium/57/artifact/odl2_karaf.log.tar.xz

https://jenkins.opendaylight.org/releng/view/netvirt/job/netvirt-csit-3node-openstack-liberty-openstack-beryllium/57/artifact/odl3_karaf.log.tar.xz

Errors in log
-----------
1. 2016-06-17 20:00:42,100 | ERROR | config-pusher | nerSelectionStrategyConfigReader | 141 - org.opendaylight.controller.sal-distributed-datastore - 1.3.3.SNAPSHOT | Cou
ld not read strategy configuration file, will use default configuration

2.

----------start-------------
-06-17 20:01:27,698 | ERROR | lt-dispatcher-27 | SimpleShardDataTreeCohort | 141 - org.opendaylight.controller.sal-distributed-datastore - 1.3.3.SNAPSHOT | Tra
nsaction failed to commit
java.lang.IllegalStateException: Store tree org.opendaylight.yangtools.yang.data.api.schema.tree.spi.MaterializedContainerNode@59bf525c and candidate base org.opendayligh
t.yangtools.yang.data.api.schema.tree.spi.MaterializedContainerNode@505478af differ.
at org.opendaylight.yangtools.yang.data.impl.schema.tree.InMemoryDataTree.commit(InMemoryDataTree.java:119)[55:org.opendaylight.yangtools.yang-data-impl:0.8.3.SNA
PSHOT]
at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.commit(SimpleShardDataTreeCohort.java:106)[141:org.opendaylight.controller.sal-distribu
ted-datastore:1.3.3.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.ChainedCommitCohort.commit(ChainedCommitCohort.java:32)[141:org.opendaylight.controller.sal-distributed-datastore
:1.3.3.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator$CohortEntry.commit(ShardCommitCoordinator.java:670)[141:org.opendaylight.controller.sal-di
stributed-datastore:1.3.3.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.Shard.finishCommit(Shard.java:352)[141:org.opendaylight.controller.sal-distributed-datastore:1.3.3.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.Shard.finishCommit(Shard.java:420)[141:org.opendaylight.controller.sal-distributed-datastore:1.3.3.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.Shard.applyState(Shard.java:668)[141:org.opendaylight.controller.sal-distributed-datastore:1.3.3.SNAPSHOT]
at org.opendaylight.controller.cluster.raft.RaftActor.handleCommand(RaftActor.java:225)[138:org.opendaylight.controller.sal-akka-raft:1.3.3.SNAPSHOT]
at org.opendaylight.controller.cluster.common.actor.AbstractUntypedPersistentActor.onReceiveCommand(AbstractUntypedPersistentActor.java:36)[137:org.opendaylight.c
ontroller.sal-clustering-commons:1.3.3.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.Shard.onReceiveCommand(Shard.java:276)[141:org.opendaylight.controller.sal-distributed-datastore:1.3.3.SNAPSHOT]
at akka.persistence.UntypedPersistentActor.onReceive(Eventsourced.scala:430)[131:com.typesafe.akka.persistence.experimental:2.3.14]
at org.opendaylight.controller.cluster.common.actor.MeteringBehavior.apply(MeteringBehavior.java:97)[137:org.opendaylight.controller.sal-clustering-commons:1.3.3.
SNAPSHOT]
at akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:534)[126:com.typesafe.akka.actor:2.3.14]
at akka.persistence.Recovery$State$class.process(Recovery.scala:30)[131:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.persistence.ProcessorImpl$$anon$2.process(Processor.scala:103)[131:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.persistence.ProcessorImpl$$anon$2.aroundReceive(Processor.scala:114)[131:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.persistence.Recovery$class.aroundReceive(Recovery.scala:265)[131:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(Eventsourced.scala:428)[131:com.typesafe.akka.persistence.experiment
al:2.3.14]
at akka.persistence.Eventsourced$$anon$2.doAroundReceive(Eventsourced.scala:82)[131:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.persistence.Eventsourced$$anon$2.aroundReceive(Eventsourced.scala:78)[131:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.persistence.Eventsourced$class.aroundReceive(Eventsourced.scala:369)[131:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.persistence.UntypedPersistentActor.aroundReceive(Eventsourced.scala:428)[131:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)[126:com.typesafe.akka.actor:2.3.14]
at akka.actor.ActorCell.invoke(ActorCell.scala:487)[126:com.typesafe.akka.actor:2.3.14]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)[126:com.typesafe.akka.actor:2.3.14]
at akka.dispatch.Mailbox.run(Mailbox.scala:220)[126:com.typesafe.akka.actor:2.3.14]
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)[126:com.typesafe.akka.actor:2.3.14]
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)[123:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)[123:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)[123:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)[123:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]
@

at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)[126:com.typesafe.akka.actor:2.3.14]
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)[123:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)[123:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)[123:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)[123:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]
2016-06-17 20:01:27,706 | WARN | lt-dispatcher-27 | Shard | 138 - org.opendaylight.controller.sal-akka-raft - 1.3.3.SNAPSHOT | member-2-shard-
topology-operational: commit failed for transaction member-1-chn-1-txn-1-1466193683878 - retrying as foreign candidate
java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Store tree org.opendaylight.yangtools.yang.data.api.schema.tree.spi.MaterializedContainerNode@59
bf525c and candidate base org.opendaylight.yangtools.yang.data.api.schema.tree.spi.MaterializedContainerNode@505478af differ.
at com.google.common.util.concurrent.Futures$ImmediateFailedFuture.get(Futures.java:190)[39:com.google.guava:18.0.0]
at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator$CohortEntry.commit(ShardCommitCoordinator.java:670)[141:org.opendaylight.controller.sal-di
stributed-datastore:1.3.3.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.Shard.finishCommit(Shard.java:352)[141:org.opendaylight.controller.sal-distributed-datastore:1.3.3.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.Shard.finishCommit(Shard.java:420)[141:org.opendaylight.controller.sal-distributed-datastore:1.3.3.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.Shard.applyState(Shard.java:668)[141:org.opendaylight.controller.sal-distributed-datastore:1.3.3.SNAPSHOT]
at org.opendaylight.controller.cluster.raft.RaftActor.handleCommand(RaftActor.java:225)[138:org.opendaylight.controller.sal-akka-raft:1.3.3.SNAPSHOT]
at org.opendaylight.controller.cluster.common.actor.AbstractUntypedPersistentActor.onReceiveCommand(AbstractUntypedPersistentActor.java:36)[137:org.opendaylight.c
ontroller.sal-clustering-commons:1.3.3.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.Shard.onReceiveCommand(Shard.java:276)[141:org.opendaylight.controller.sal-distributed-datastore:1.3.3.SNAPSHOT]
at akka.persistence.UntypedPersistentActor.onReceive(Eventsourced.scala:430)[131:com.typesafe.akka.persistence.experimental:2.3.14]
at org.opendaylight.controller.cluster.common.actor.MeteringBehavior.apply(MeteringBehavior.java:97)[137:org.opendaylight.controller.sal-clustering-commons:1.3.3.
SNAPSHOT]
at akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:534)[126:com.typesafe.akka.actor:2.3.14]
at akka.persistence.Recovery$State$class.process(Recovery.scala:30)[131:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.persistence.ProcessorImpl$$anon$2.process(Processor.scala:103)[131:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.persistence.ProcessorImpl$$anon$2.aroundReceive(Processor.scala:114)[131:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.persistence.Recovery$class.aroundReceive(Recovery.scala:265)[131:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(Eventsourced.scala:428)[131:com.typesafe.akka.persistence.experiment
al:2.3.14]
at akka.persistence.Eventsourced$$anon$2.doAroundReceive(Eventsourced.scala:82)[131:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.persistence.Eventsourced$$anon$2.aroundReceive(Eventsourced.scala:78)[131:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.persistence.Eventsourced$class.aroundReceive(Eventsourced.scala:369)[131:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.persistence.UntypedPersistentActor.aroundReceive(Eventsourced.scala:428)[131:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)[126:com.typesafe.akka.actor:2.3.14]
at akka.actor.ActorCell.invoke(ActorCell.scala:487)[126:com.typesafe.akka.actor:2.3.14]

SNAPSHOT]
at akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:534)[126:com.typesafe.akka.actor:2.3.14]
at akka.persistence.Recovery$State$class.process(Recovery.scala:30)[131:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.persistence.ProcessorImpl$$anon$2.process(Processor.scala:103)[131:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.persistence.ProcessorImpl$$anon$2.aroundReceive(Processor.scala:114)[131:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.persistence.Recovery$class.aroundReceive(Recovery.scala:265)[131:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(Eventsourced.scala:428)[131:com.typesafe.akka.persistence.experiment
al:2.3.14]
at akka.persistence.Eventsourced$$anon$2.doAroundReceive(Eventsourced.scala:82)[131:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.persistence.Eventsourced$$anon$2.aroundReceive(Eventsourced.scala:78)[131:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.persistence.Eventsourced$class.aroundReceive(Eventsourced.scala:369)[131:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.persistence.UntypedPersistentActor.aroundReceive(Eventsourced.scala:428)[131:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)[126:com.typesafe.akka.actor:2.3.14]
at akka.actor.ActorCell.invoke(ActorCell.scala:487)[126:com.typesafe.akka.actor:2.3.14]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)[126:com.typesafe.akka.actor:2.3.14]
at akka.dispatch.Mailbox.run(Mailbox.scala:220)[126:com.typesafe.akka.actor:2.3.14]
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)[126:com.typesafe.akka.actor:2.3.14]
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)[123:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)[123:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)[123:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)[123:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]
Caused by: java.lang.IllegalStateException: Store tree org.opendaylight.yangtools.yang.data.api.schema.tree.spi.MaterializedContainerNode@59bf525c and candidate base org.
opendaylight.yangtools.yang.data.api.schema.tree.spi.MaterializedContainerNode@505478af differ.
at org.opendaylight.yangtools.yang.data.impl.schema.tree.InMemoryDataTree.commit(InMemoryDataTree.java:119)[55:org.opendaylight.yangtools.yang-data-impl:0.8.3.SNA
PSHOT]
at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.commit(SimpleShardDataTreeCohort.java:106)[141:org.opendaylight.controller.sal-distribu
ted-datastore:1.3.3.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.ChainedCommitCohort.commit(ChainedCommitCohort.java:32)[141:org.opendaylight.controller.sal-distributed-datastore
:1.3.3.SNAPSHOT]
... 28 more

-----------------end----------------