-
Bug
-
Resolution: Done
-
None
-
None
-
None
-
Operating System: All
Platform: All
-
3087
this is a blocker for CI
the same flow that is used and passing in the Helium tests fails with the Lithium redesign feature set
(odl-openflowplugin-flow-services-ui-li)
log messages seem to indicate something broken in the shard inventory ? clustering related?
The flow XML is this:
<?xml version="1.0" encoding="UTF-8" standalone="no"?> <flow xmlns="urn:opendaylight:flow:inventory"> <strict>false</strict> <instructions> <instruction> <order>0</order> <apply-actions> <action> <order>0</order> <dec-nw-ttl/> </action> </apply-actions> </instruction> </instructions> <table_id>2</table_id> <id>124</id> <cookie_mask>255</cookie_mask> <installHw>false</installHw> <match> <ethernet-match> <ethernet-type> <type>2048</type> </ethernet-type> </ethernet-match> <ipv4-destination>10.0.1.0/24</ipv4-destination> </match> <cookie>1</cookie> <flow-name>FooXf1</flow-name> <priority>1</priority> <barrier>false</barrier> </flow>
The rest call seems to hang (from robot and/or POSTMAN) and eventually the karaf log shows these errors:
2015-04-29 16:30:30,782 | ERROR | lt-dispatcher-70 | DataTreeChangeListenerProxy | 179 - org.opendaylight.controller.sal-distributed-datastore - 1.2.0.SNAPSHOT | Failed to find local shard inventory - DataTreeChangeListener org.opendaylight.controller.md.sal.binding.impl.BindingDOMDataTreeChangeListenerAdapter@13bc5396 at path /(urn:opendaylight:inventory?revision=2013-08-19)nodes/node/node/AugmentationIdentifier
{childNames=[(urn:opendaylight:flow:inventory?revision=2013-08-19)table, (urn:opendaylight:flow:inventory?revision=2013-08-19)meter, (urn:opendaylight:flow:inventory?revision=2013-08-19)supported-instructions, (urn:opendaylight:flow:inventory?revision=2013-08-19)switch-features, (urn:opendaylight:flow:inventory?revision=2013-08-19)group, (urn:opendaylight:flow:inventory?revision=2013-08-19)manufacturer, (urn:opendaylight:flow:inventory?revision=2013-08-19)hardware, (urn:opendaylight:flow:inventory?revision=2013-08-19)ip-address, (urn:opendaylight:flow:inventory?revision=2013-08-19)software, (urn:opendaylight:flow:inventory?revision=2013-08-19)supported-actions, (urn:opendaylight:flow:inventory?revision=2013-08-19)supported-match-types, (urn:opendaylight:flow:inventory?revision=2013-08-19)description, (urn:opendaylight:flow:inventory?revision=2013-08-19)serial-number]}/(urn:opendaylight:flow:inventory?revision=2013-08-19)table/table/flow/flow cannot be registered: org.opendaylight.controller.cluster.datastore.exceptions.NotInitializedException: Found primary shard member-1-shard-inventory-config but it's not initialized yet. Please try again later
2015-04-29 16:30:30,791 | ERROR | lt-dispatcher-18 | DataTreeChangeListenerProxy | 179 - org.opendaylight.controller.sal-distributed-datastore - 1.2.0.SNAPSHOT | Failed to find local shard inventory - DataTreeChangeListener org.opendaylight.controller.md.sal.binding.impl.BindingDOMDataTreeChangeListenerAdapter@3113a6f9 at path /(urn:opendaylight:inventory?revision=2013-08-19)nodes/node/node/AugmentationIdentifier
/(urn:opendaylight:flow:inventory?revision=2013-08-19)group/group cannot be registered: org.opendaylight.controller.cluster.datastore.exceptions.NotInitializedException: Found primary shard member-1-shard-inventory-config but it's not initialized yet. Please try again later
2015-04-29 16:30:30,801 | ERROR | lt-dispatcher-70 | DataTreeChangeListenerProxy | 179 - org.opendaylight.controller.sal-distributed-datastore - 1.2.0.SNAPSHOT | Failed to find local shard inventory - DataTreeChangeListener org.opendaylight.controller.md.sal.binding.impl.BindingDOMDataTreeChangeListenerAdapter@1373e9b at path /(urn:opendaylight:inventory?revision=2013-08-19)nodes/node/node/AugmentationIdentifier
/(urn:opendaylight:flow:inventory?revision=2013-08-19)meter/meter cannot be registered: org.opendaylight.controller.cluster.datastore.exceptions.NotInitializedException: Found primary shard member-1-shard-inventory-config but it's not initialized yet. Please try again later
2015-04-29 16:30:30,802 | ERROR | ult-dispatcher-4 | DataTreeChangeListenerProxy | 179 - org.opendaylight.controller.sal-distributed-datastore - 1.2.0.SNAPSHOT | Failed to find local shard inventory - DataTreeChangeListener org.opendaylight.controller.md.sal.binding.impl.BindingDOMDataTreeChangeListenerAdapter@2eebfeaa at path /(urn:opendaylight:inventory?revision=2013-08-19)nodes/node/node/AugmentationIdentifier
/(urn:opendaylight:flow:inventory?revision=2013-08-19)table/table cannot be registered: org.opendaylight.controller.cluster.datastore.exceptions.NotInitializedException: Found primary shard member-1-shard-inventory-config but it's not initialized yet. Please try again later
2015-04-29 16:30:34,074 | ERROR | lt-dispatcher-15 | FlowNodeReconciliationImpl | 188 - org.opendaylight.openflowplugin.applications.forwardingrules-manager - 0.1.0.SNAPSHOT | Fail with read Config/DS for Node InstanceIdentifier
!
java.util.concurrent.ExecutionException: ReadFailedException{message=Error reading data for path /(urn:opendaylight:inventory?revision=2013-08-19)nodes/node/node[
]/AugmentationIdentifier
{childNames=[(urn:opendaylight:flow:inventory?revision=2013-08-19)table, (urn:opendaylight:flow:inventory?revision=2013-08-19)meter, (urn:opendaylight:flow:inventory?revision=2013-08-19)supported-instructions, (urn:opendaylight:flow:inventory?revision=2013-08-19)switch-features, (urn:opendaylight:flow:inventory?revision=2013-08-19)group, (urn:opendaylight:flow:inventory?revision=2013-08-19)manufacturer, (urn:opendaylight:flow:inventory?revision=2013-08-19)hardware, (urn:opendaylight:flow:inventory?revision=2013-08-19)ip-address, (urn:opendaylight:flow:inventory?revision=2013-08-19)software, (urn:opendaylight:flow:inventory?revision=2013-08-19)supported-actions, (urn:opendaylight:flow:inventory?revision=2013-08-19)supported-match-types, (urn:opendaylight:flow:inventory?revision=2013-08-19)description, (urn:opendaylight:flow:inventory?revision=2013-08-19)serial-number]}, errorList=[RpcError [message=Error reading data for path /(urn:opendaylight:inventory?revision=2013-08-19)nodes/node/node[
{(urn:opendaylight:inventory?revision=2013-08-19)id=openflow:262584037476162}]/AugmentationIdentifier
{childNames=[(urn:opendaylight:flow:inventory?revision=2013-08-19)table, (urn:opendaylight:flow:inventory?revision=2013-08-19)meter, (urn:opendaylight:flow:inventory?revision=2013-08-19)supported-instructions, (urn:opendaylight:flow:inventory?revision=2013-08-19)switch-features, (urn:opendaylight:flow:inventory?revision=2013-08-19)group, (urn:opendaylight:flow:inventory?revision=2013-08-19)manufacturer, (urn:opendaylight:flow:inventory?revision=2013-08-19)hardware, (urn:opendaylight:flow:inventory?revision=2013-08-19)ip-address, (urn:opendaylight:flow:inventory?revision=2013-08-19)software, (urn:opendaylight:flow:inventory?revision=2013-08-19)supported-actions, (urn:opendaylight:flow:inventory?revision=2013-08-19)supported-match-types, (urn:opendaylight:flow:inventory?revision=2013-08-19)description, (urn:opendaylight:flow:inventory?revision=2013-08-19)serial-number]}, severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=org.opendaylight.controller.cluster.datastore.exceptions.NotInitializedException: Found primary shard member-1-shard-inventory-config but it's not initialized yet. Please try again later]]}
at org.opendaylight.yangtools.util.concurrent.MappingCheckedFuture.wrapInExecutionException(MappingCheckedFuture.java:64)[64:org.opendaylight.yangtools.util:0.7.0.SNAPSHOT]
at org.opendaylight.yangtools.util.concurrent.MappingCheckedFuture.get(MappingCheckedFuture.java:77)[64:org.opendaylight.yangtools.util:0.7.0.SNAPSHOT]
at org.opendaylight.openflowplugin.applications.frm.impl.FlowNodeReconciliationImpl.reconciliation(FlowNodeReconciliationImpl.java:144)[188:org.opendaylight.openflowplugin.applications.forwardingrules-manager:0.1.0.SNAPSHOT]
at org.opendaylight.openflowplugin.applications.frm.impl.FlowNodeReconciliationImpl.flowNodeConnected(FlowNodeReconciliationImpl.java:134)[188:org.opendaylight.openflowplugin.applications.forwardingrules-manager:0.1.0.SNAPSHOT]
at org.opendaylight.openflowplugin.applications.frm.impl.FlowNodeReconciliationImpl.onDataChanged(FlowNodeReconciliationImpl.java:120)[188:org.opendaylight.openflowplugin.applications.forwardingrules-manager:0.1.0.SNAPSHOT]
at org.opendaylight.controller.md.sal.binding.impl.AbstractForwardedDataBroker$TranslatingDataChangeInvoker.onDataChanged(AbstractForwardedDataBroker.java:134)[157:org.opendaylight.controller.sal-binding-broker-impl:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.DataChangeListener.dataChanged(DataChangeListener.java:66)[179:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.DataChangeListener.handleReceive(DataChangeListener.java:39)[179:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.common.actor.AbstractUntypedActor.onReceive(AbstractUntypedActor.java:34)[171:org.opendaylight.controller.sal-clustering-commons:1.2.0.SNAPSHOT]
at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:167)[164:com.typesafe.akka.actor:2.3.9]
at akka.actor.Actor$class.aroundReceive(Actor.scala:465)[164:com.typesafe.akka.actor:2.3.9]
at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:97)[164:com.typesafe.akka.actor:2.3.9]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)[164:com.typesafe.akka.actor:2.3.9]
at akka.actor.ActorCell.invoke(ActorCell.scala:487)[164:com.typesafe.akka.actor:2.3.9]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)[164:com.typesafe.akka.actor:2.3.9]
at akka.dispatch.Mailbox.run(Mailbox.scala:221)[164:com.typesafe.akka.actor:2.3.9]
at akka.dispatch.Mailbox.exec(Mailbox.scala:231)[164:com.typesafe.akka.actor:2.3.9]
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)[161:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b]
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)[161:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b]
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)[161:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b]
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)[161:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b]
Caused by: ReadFailedException{message=Error reading data for path /(urn:opendaylight:inventory?revision=2013-08-19)nodes/node/node[
]/AugmentationIdentifier
{childNames=[(urn:opendaylight:flow:inventory?revision=2013-08-19)table, (urn:opendaylight:flow:inventory?revision=2013-08-19)meter, (urn:opendaylight:flow:inventory?revision=2013-08-19)supported-instructions, (urn:opendaylight:flow:inventory?revision=2013-08-19)switch-features, (urn:opendaylight:flow:inventory?revision=2013-08-19)group, (urn:opendaylight:flow:inventory?revision=2013-08-19)manufacturer, (urn:opendaylight:flow:inventory?revision=2013-08-19)hardware, (urn:opendaylight:flow:inventory?revision=2013-08-19)ip-address, (urn:opendaylight:flow:inventory?revision=2013-08-19)software, (urn:opendaylight:flow:inventory?revision=2013-08-19)supported-actions, (urn:opendaylight:flow:inventory?revision=2013-08-19)supported-match-types, (urn:opendaylight:flow:inventory?revision=2013-08-19)description, (urn:opendaylight:flow:inventory?revision=2013-08-19)serial-number]}, errorList=[RpcError [message=Error reading data for path /(urn:opendaylight:inventory?revision=2013-08-19)nodes/node/node[
{(urn:opendaylight:inventory?revision=2013-08-19)id=openflow:262584037476162}]/AugmentationIdentifier
{childNames=[(urn:opendaylight:flow:inventory?revision=2013-08-19)table, (urn:opendaylight:flow:inventory?revision=2013-08-19)meter, (urn:opendaylight:flow:inventory?revision=2013-08-19)supported-instructions, (urn:opendaylight:flow:inventory?revision=2013-08-19)switch-features, (urn:opendaylight:flow:inventory?revision=2013-08-19)group, (urn:opendaylight:flow:inventory?revision=2013-08-19)manufacturer, (urn:opendaylight:flow:inventory?revision=2013-08-19)hardware, (urn:opendaylight:flow:inventory?revision=2013-08-19)ip-address, (urn:opendaylight:flow:inventory?revision=2013-08-19)software, (urn:opendaylight:flow:inventory?revision=2013-08-19)supported-actions, (urn:opendaylight:flow:inventory?revision=2013-08-19)supported-match-types, (urn:opendaylight:flow:inventory?revision=2013-08-19)description, (urn:opendaylight:flow:inventory?revision=2013-08-19)serial-number]}, severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=org.opendaylight.controller.cluster.datastore.exceptions.NotInitializedException: Found primary shard member-1-shard-inventory-config but it's not initialized yet. Please try again later]]}
at org.opendaylight.controller.cluster.datastore.NoOpTransactionContext.readData(NoOpTransactionContext.java:80)[179:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.TransactionProxy$2.invoke(TransactionProxy.java:186)[179:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.TransactionFutureCallback.executeTxOperatonsOnComplete(TransactionFutureCallback.java:252)[179:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.TransactionFutureCallback.createTransactionContext(TransactionFutureCallback.java:221)[179:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.TransactionProxy$12.onComplete(TransactionProxy.java:512)[179:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.TransactionProxy$12.onComplete(TransactionProxy.java:508)[179:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at akka.dispatch.OnComplete.internal(Future.scala:246)[164:com.typesafe.akka.actor:2.3.9]
at akka.dispatch.OnComplete.internal(Future.scala:244)[164:com.typesafe.akka.actor:2.3.9]
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:174)[164:com.typesafe.akka.actor:2.3.9]
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:171)[164:com.typesafe.akka.actor:2.3.9]
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)[161:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b]
at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67)[164:com.typesafe.akka.actor:2.3.9]
at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82)[164:com.typesafe.akka.actor:2.3.9]
at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)[164:com.typesafe.akka.actor:2.3.9]
at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)[164:com.typesafe.akka.actor:2.3.9]
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)[161:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b]
at akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58)[164:com.typesafe.akka.actor:2.3.9]
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41)[164:com.typesafe.akka.actor:2.3.9]
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:401)[164:com.typesafe.akka.actor:2.3.9]
... 4 more
Caused by: org.opendaylight.controller.cluster.datastore.exceptions.NotInitializedException: Found primary shard member-1-shard-inventory-config but it's not initialized yet. Please try again later
at org.opendaylight.controller.cluster.datastore.ShardManager.createNotInitializedException(ShardManager.java:417)[179:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.ShardManager.onShardNotInitializedTimeout(ShardManager.java:232)[179:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.ShardManager.handleCommand(ShardManager.java:191)[179:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.common.actor.AbstractUntypedPersistentActor.onReceiveCommand(AbstractUntypedPersistentActor.java:36)[171:org.opendaylight.controller.sal-clustering-commons:1.2.0.SNAPSHOT]
at akka.persistence.UntypedPersistentActor.onReceive(Eventsourced.scala:430)[169:com.typesafe.akka.persistence.experimental:2.3.9]
at org.opendaylight.controller.cluster.common.actor.MeteringBehavior.apply(MeteringBehavior.java:97)[171:org.opendaylight.controller.sal-clustering-commons:1.2.0.SNAPSHOT]
at akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:534)[164:com.typesafe.akka.actor:2.3.9]
at akka.persistence.Recovery$State$class.process(Recovery.scala:30)[169:com.typesafe.akka.persistence.experimental:2.3.9]
at akka.persistence.ProcessorImpl$$anon$2.process(Processor.scala:103)[169:com.typesafe.akka.persistence.experimental:2.3.9]
at akka.persistence.ProcessorImpl$$anon$2.aroundReceive(Processor.scala:114)[169:com.typesafe.akka.persistence.experimental:2.3.9]
at akka.persistence.Recovery$class.aroundReceive(Recovery.scala:265)[169:com.typesafe.akka.persistence.experimental:2.3.9]
at akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(Eventsourced.scala:428)[169:com.typesafe.akka.persistence.experimental:2.3.9]
at akka.persistence.Eventsourced$$anon$2.doAroundReceive(Eventsourced.scala:82)[169:com.typesafe.akka.persistence.experimental:2.3.9]
at akka.persistence.Eventsourced$$anon$2.aroundReceive(Eventsourced.scala:78)[169:com.typesafe.akka.persistence.experimental:2.3.9]
at akka.persistence.Eventsourced$class.aroundReceive(Eventsourced.scala:369)[169:com.typesafe.akka.persistence.experimental:2.3.9]
at akka.persistence.UntypedPersistentActor.aroundReceive(Eventsourced.scala:428)[169:com.typesafe.akka.persistence.experimental:2.3.9]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)[164:com.typesafe.akka.actor:2.3.9]
at akka.actor.ActorCell.invoke(ActorCell.scala:487)[164:com.typesafe.akka.actor:2.3.9]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)[164:com.typesafe.akka.actor:2.3.9]
at akka.dispatch.Mailbox.run(Mailbox.scala:221)[164:com.typesafe.akka.actor:2.3.9]
at akka.dispatch.Mailbox.exec(Mailbox.scala:231)[164:com.typesafe.akka.actor:2.3.9]
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)[161:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b]
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:1253)[161:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b]
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1346)[161:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b]
... 2 more
