[NETVIRT-33] ERROR Log Observations - CSIT (sal datastore related) Created: 17/Jun/16  Updated: 15/Jul/16  Resolved: 15/Jul/16

Status: Resolved
Project: netvirt
Component/s: General
Affects Version/s: Beryllium
Fix Version/s: None

Type: Bug
Reporter: Venkatrangan Govindarajan Assignee: Arthi Bhattacharjee
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: All
Platform: All

External issue ID: 6080

 Description   

The 3node Netvirt Openstack Tests are failing, In this report, we would like to capture the clustering related errors seen in karaf log for further analysis.

Some datastore related errors observed in karaf.log from all the 3 nodes in Netvirt. I guess, not all of the messages are actual failures, But it will be better to have some explanation for the failure being reported.

references:
https://jenkins.opendaylight.org/releng/view/netvirt/job/netvirt-csit-3node-openstack-liberty-openstack-beryllium/57/artifact/odl1_karaf.log.tar.xz

https://jenkins.opendaylight.org/releng/view/netvirt/job/netvirt-csit-3node-openstack-liberty-openstack-beryllium/57/artifact/odl2_karaf.log.tar.xz

https://jenkins.opendaylight.org/releng/view/netvirt/job/netvirt-csit-3node-openstack-liberty-openstack-beryllium/57/artifact/odl3_karaf.log.tar.xz

Errors in log
-----------
1. 2016-06-17 20:00:42,100 | ERROR | config-pusher | nerSelectionStrategyConfigReader | 141 - org.opendaylight.controller.sal-distributed-datastore - 1.3.3.SNAPSHOT | Cou
ld not read strategy configuration file, will use default configuration

2.

----------start-------------
-06-17 20:01:27,698 | ERROR | lt-dispatcher-27 | SimpleShardDataTreeCohort | 141 - org.opendaylight.controller.sal-distributed-datastore - 1.3.3.SNAPSHOT | Tra
nsaction failed to commit
java.lang.IllegalStateException: Store tree org.opendaylight.yangtools.yang.data.api.schema.tree.spi.MaterializedContainerNode@59bf525c and candidate base org.opendayligh
t.yangtools.yang.data.api.schema.tree.spi.MaterializedContainerNode@505478af differ.
at org.opendaylight.yangtools.yang.data.impl.schema.tree.InMemoryDataTree.commit(InMemoryDataTree.java:119)[55:org.opendaylight.yangtools.yang-data-impl:0.8.3.SNA
PSHOT]
at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.commit(SimpleShardDataTreeCohort.java:106)[141:org.opendaylight.controller.sal-distribu
ted-datastore:1.3.3.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.ChainedCommitCohort.commit(ChainedCommitCohort.java:32)[141:org.opendaylight.controller.sal-distributed-datastore
:1.3.3.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator$CohortEntry.commit(ShardCommitCoordinator.java:670)[141:org.opendaylight.controller.sal-di
stributed-datastore:1.3.3.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.Shard.finishCommit(Shard.java:352)[141:org.opendaylight.controller.sal-distributed-datastore:1.3.3.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.Shard.finishCommit(Shard.java:420)[141:org.opendaylight.controller.sal-distributed-datastore:1.3.3.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.Shard.applyState(Shard.java:668)[141:org.opendaylight.controller.sal-distributed-datastore:1.3.3.SNAPSHOT]
at org.opendaylight.controller.cluster.raft.RaftActor.handleCommand(RaftActor.java:225)[138:org.opendaylight.controller.sal-akka-raft:1.3.3.SNAPSHOT]
at org.opendaylight.controller.cluster.common.actor.AbstractUntypedPersistentActor.onReceiveCommand(AbstractUntypedPersistentActor.java:36)[137:org.opendaylight.c
ontroller.sal-clustering-commons:1.3.3.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.Shard.onReceiveCommand(Shard.java:276)[141:org.opendaylight.controller.sal-distributed-datastore:1.3.3.SNAPSHOT]
at akka.persistence.UntypedPersistentActor.onReceive(Eventsourced.scala:430)[131:com.typesafe.akka.persistence.experimental:2.3.14]
at org.opendaylight.controller.cluster.common.actor.MeteringBehavior.apply(MeteringBehavior.java:97)[137:org.opendaylight.controller.sal-clustering-commons:1.3.3.
SNAPSHOT]
at akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:534)[126:com.typesafe.akka.actor:2.3.14]
at akka.persistence.Recovery$State$class.process(Recovery.scala:30)[131:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.persistence.ProcessorImpl$$anon$2.process(Processor.scala:103)[131:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.persistence.ProcessorImpl$$anon$2.aroundReceive(Processor.scala:114)[131:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.persistence.Recovery$class.aroundReceive(Recovery.scala:265)[131:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(Eventsourced.scala:428)[131:com.typesafe.akka.persistence.experiment
al:2.3.14]
at akka.persistence.Eventsourced$$anon$2.doAroundReceive(Eventsourced.scala:82)[131:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.persistence.Eventsourced$$anon$2.aroundReceive(Eventsourced.scala:78)[131:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.persistence.Eventsourced$class.aroundReceive(Eventsourced.scala:369)[131:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.persistence.UntypedPersistentActor.aroundReceive(Eventsourced.scala:428)[131:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)[126:com.typesafe.akka.actor:2.3.14]
at akka.actor.ActorCell.invoke(ActorCell.scala:487)[126:com.typesafe.akka.actor:2.3.14]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)[126:com.typesafe.akka.actor:2.3.14]
at akka.dispatch.Mailbox.run(Mailbox.scala:220)[126:com.typesafe.akka.actor:2.3.14]
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)[126:com.typesafe.akka.actor:2.3.14]
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)[123:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)[123:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)[123:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)[123:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]
@

at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)[126:com.typesafe.akka.actor:2.3.14]
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)[123:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)[123:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)[123:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)[123:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]
2016-06-17 20:01:27,706 | WARN | lt-dispatcher-27 | Shard | 138 - org.opendaylight.controller.sal-akka-raft - 1.3.3.SNAPSHOT | member-2-shard-
topology-operational: commit failed for transaction member-1-chn-1-txn-1-1466193683878 - retrying as foreign candidate
java.util.concurrent.ExecutionException: java.lang.IllegalStateException: Store tree org.opendaylight.yangtools.yang.data.api.schema.tree.spi.MaterializedContainerNode@59
bf525c and candidate base org.opendaylight.yangtools.yang.data.api.schema.tree.spi.MaterializedContainerNode@505478af differ.
at com.google.common.util.concurrent.Futures$ImmediateFailedFuture.get(Futures.java:190)[39:com.google.guava:18.0.0]
at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator$CohortEntry.commit(ShardCommitCoordinator.java:670)[141:org.opendaylight.controller.sal-di
stributed-datastore:1.3.3.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.Shard.finishCommit(Shard.java:352)[141:org.opendaylight.controller.sal-distributed-datastore:1.3.3.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.Shard.finishCommit(Shard.java:420)[141:org.opendaylight.controller.sal-distributed-datastore:1.3.3.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.Shard.applyState(Shard.java:668)[141:org.opendaylight.controller.sal-distributed-datastore:1.3.3.SNAPSHOT]
at org.opendaylight.controller.cluster.raft.RaftActor.handleCommand(RaftActor.java:225)[138:org.opendaylight.controller.sal-akka-raft:1.3.3.SNAPSHOT]
at org.opendaylight.controller.cluster.common.actor.AbstractUntypedPersistentActor.onReceiveCommand(AbstractUntypedPersistentActor.java:36)[137:org.opendaylight.c
ontroller.sal-clustering-commons:1.3.3.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.Shard.onReceiveCommand(Shard.java:276)[141:org.opendaylight.controller.sal-distributed-datastore:1.3.3.SNAPSHOT]
at akka.persistence.UntypedPersistentActor.onReceive(Eventsourced.scala:430)[131:com.typesafe.akka.persistence.experimental:2.3.14]
at org.opendaylight.controller.cluster.common.actor.MeteringBehavior.apply(MeteringBehavior.java:97)[137:org.opendaylight.controller.sal-clustering-commons:1.3.3.
SNAPSHOT]
at akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:534)[126:com.typesafe.akka.actor:2.3.14]
at akka.persistence.Recovery$State$class.process(Recovery.scala:30)[131:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.persistence.ProcessorImpl$$anon$2.process(Processor.scala:103)[131:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.persistence.ProcessorImpl$$anon$2.aroundReceive(Processor.scala:114)[131:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.persistence.Recovery$class.aroundReceive(Recovery.scala:265)[131:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(Eventsourced.scala:428)[131:com.typesafe.akka.persistence.experiment
al:2.3.14]
at akka.persistence.Eventsourced$$anon$2.doAroundReceive(Eventsourced.scala:82)[131:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.persistence.Eventsourced$$anon$2.aroundReceive(Eventsourced.scala:78)[131:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.persistence.Eventsourced$class.aroundReceive(Eventsourced.scala:369)[131:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.persistence.UntypedPersistentActor.aroundReceive(Eventsourced.scala:428)[131:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)[126:com.typesafe.akka.actor:2.3.14]
at akka.actor.ActorCell.invoke(ActorCell.scala:487)[126:com.typesafe.akka.actor:2.3.14]

SNAPSHOT]
at akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:534)[126:com.typesafe.akka.actor:2.3.14]
at akka.persistence.Recovery$State$class.process(Recovery.scala:30)[131:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.persistence.ProcessorImpl$$anon$2.process(Processor.scala:103)[131:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.persistence.ProcessorImpl$$anon$2.aroundReceive(Processor.scala:114)[131:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.persistence.Recovery$class.aroundReceive(Recovery.scala:265)[131:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(Eventsourced.scala:428)[131:com.typesafe.akka.persistence.experiment
al:2.3.14]
at akka.persistence.Eventsourced$$anon$2.doAroundReceive(Eventsourced.scala:82)[131:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.persistence.Eventsourced$$anon$2.aroundReceive(Eventsourced.scala:78)[131:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.persistence.Eventsourced$class.aroundReceive(Eventsourced.scala:369)[131:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.persistence.UntypedPersistentActor.aroundReceive(Eventsourced.scala:428)[131:com.typesafe.akka.persistence.experimental:2.3.14]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)[126:com.typesafe.akka.actor:2.3.14]
at akka.actor.ActorCell.invoke(ActorCell.scala:487)[126:com.typesafe.akka.actor:2.3.14]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)[126:com.typesafe.akka.actor:2.3.14]
at akka.dispatch.Mailbox.run(Mailbox.scala:220)[126:com.typesafe.akka.actor:2.3.14]
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)[126:com.typesafe.akka.actor:2.3.14]
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)[123:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)[123:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)[123:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)[123:org.scala-lang.scala-library:2.11.7.v20150622-112736-1fbce4612c]
Caused by: java.lang.IllegalStateException: Store tree org.opendaylight.yangtools.yang.data.api.schema.tree.spi.MaterializedContainerNode@59bf525c and candidate base org.
opendaylight.yangtools.yang.data.api.schema.tree.spi.MaterializedContainerNode@505478af differ.
at org.opendaylight.yangtools.yang.data.impl.schema.tree.InMemoryDataTree.commit(InMemoryDataTree.java:119)[55:org.opendaylight.yangtools.yang-data-impl:0.8.3.SNA
PSHOT]
at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.commit(SimpleShardDataTreeCohort.java:106)[141:org.opendaylight.controller.sal-distribu
ted-datastore:1.3.3.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.ChainedCommitCohort.commit(ChainedCommitCohort.java:32)[141:org.opendaylight.controller.sal-distributed-datastore
:1.3.3.SNAPSHOT]
... 28 more

-----------------end----------------

 Comments   
Comment by Arthi Bhattacharjee [ 15/Jul/16 ]

Hi Venkat,

• While pushing Entity Owner configuration snapshot, these errors are been occurred. Since there is no property set at the beginning, the property is null and thus it throws an error.

• The configuration ID “org.opendaylight.controller.cluster.entity.owner.selection.strategies" is used to fetch the configuration object. Since there is no configuration object available, it returns a new configuration object with property null.
REF:: https://osgi.org/javadoc/r4v42/org/osgi/service/cm/ConfigurationAdmin.html#getConfiguration(java.lang.String)

• We must have a configuration file (.cfg) in /etc folder. Since that file is missing, we manually added that file and tried it. The error is not produced.
REF:: https://lists.opendaylight.org/pipermail/controller-dev/2016-January/011501.html

Generated at Wed Feb 07 20:20:31 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.