[#OPNFLWPLUG-424] [Lithium redesign] cannot push a flow to config DS

[OPNFLWPLUG-424] [Lithium redesign] cannot push a flow to config DS Created: 29/Apr/15 Updated: 27/Sep/21 Resolved: 05/May/15
Status:	Verified
Project:	OpenFlowPlugin
Component/s:	General
Affects Version/s:	None
Fix Version/s:	None

Type:

Bug

Reporter:

Jamo Luhrsen

Assignee:

Michal Rehak

Resolution:

Done

Votes:

Labels:

None

Remaining Estimate:

Not Specified

Time Spent:

Not Specified

Original Estimate:

Not Specified

Environment:

Operating System: All
Platform: All

External issue ID:

3087

Description

this is a blocker for CI

the same flow that is used and passing in the Helium tests fails with the Lithium redesign feature set
(odl-openflowplugin-flow-services-ui-li)

log messages seem to indicate something broken in the shard inventory ? clustering related?

The flow XML is this:

<?xml version="1.0" encoding="UTF-8" standalone="no"?> <flow xmlns="urn:opendaylight:flow:inventory"> <strict>false</strict> <instructions> <instruction> <order>0</order> <apply-actions> <action> <order>0</order> <dec-nw-ttl/> </action> </apply-actions> </instruction> </instructions> <table_id>2</table_id> <id>124</id> <cookie_mask>255</cookie_mask> <installHw>false</installHw> <match> <ethernet-match> <ethernet-type> <type>2048</type> </ethernet-type> </ethernet-match> <ipv4-destination>10.0.1.0/24</ipv4-destination> </match> <cookie>1</cookie> <flow-name>FooXf1</flow-name> <priority>1</priority> <barrier>false</barrier> </flow>

The rest call seems to hang (from robot and/or POSTMAN) and eventually the karaf log shows these errors:

2015-04-29 16:30:30,782 | ERROR | lt-dispatcher-70 | DataTreeChangeListenerProxy | 179 - org.opendaylight.controller.sal-distributed-datastore - 1.2.0.SNAPSHOT | Failed to find local shard inventory - DataTreeChangeListener org.opendaylight.controller.md.sal.binding.impl.BindingDOMDataTreeChangeListenerAdapter@13bc5396 at path /(urn:opendaylight:inventory?revision=2013-08-19)nodes/node/node/AugmentationIdentifier

{childNames=[(urn:opendaylight:flow:inventory?revision=2013-08-19)table, (urn:opendaylight:flow:inventory?revision=2013-08-19)meter, (urn:opendaylight:flow:inventory?revision=2013-08-19)supported-instructions, (urn:opendaylight:flow:inventory?revision=2013-08-19)switch-features, (urn:opendaylight:flow:inventory?revision=2013-08-19)group, (urn:opendaylight:flow:inventory?revision=2013-08-19)manufacturer, (urn:opendaylight:flow:inventory?revision=2013-08-19)hardware, (urn:opendaylight:flow:inventory?revision=2013-08-19)ip-address, (urn:opendaylight:flow:inventory?revision=2013-08-19)software, (urn:opendaylight:flow:inventory?revision=2013-08-19)supported-actions, (urn:opendaylight:flow:inventory?revision=2013-08-19)supported-match-types, (urn:opendaylight:flow:inventory?revision=2013-08-19)description, (urn:opendaylight:flow:inventory?revision=2013-08-19)serial-number]}

/(urn:opendaylight:flow:inventory?revision=2013-08-19)table/table/flow/flow cannot be registered: org.opendaylight.controller.cluster.datastore.exceptions.NotInitializedException: Found primary shard member-1-shard-inventory-config but it's not initialized yet. Please try again later
2015-04-29 16:30:30,791 | ERROR | lt-dispatcher-18 | DataTreeChangeListenerProxy | 179 - org.opendaylight.controller.sal-distributed-datastore - 1.2.0.SNAPSHOT | Failed to find local shard inventory - DataTreeChangeListener org.opendaylight.controller.md.sal.binding.impl.BindingDOMDataTreeChangeListenerAdapter@3113a6f9 at path /(urn:opendaylight:inventory?revision=2013-08-19)nodes/node/node/AugmentationIdentifier

/(urn:opendaylight:flow:inventory?revision=2013-08-19)group/group cannot be registered: org.opendaylight.controller.cluster.datastore.exceptions.NotInitializedException: Found primary shard member-1-shard-inventory-config but it's not initialized yet. Please try again later
2015-04-29 16:30:30,801 | ERROR | lt-dispatcher-70 | DataTreeChangeListenerProxy | 179 - org.opendaylight.controller.sal-distributed-datastore - 1.2.0.SNAPSHOT | Failed to find local shard inventory - DataTreeChangeListener org.opendaylight.controller.md.sal.binding.impl.BindingDOMDataTreeChangeListenerAdapter@1373e9b at path /(urn:opendaylight:inventory?revision=2013-08-19)nodes/node/node/AugmentationIdentifier

/(urn:opendaylight:flow:inventory?revision=2013-08-19)meter/meter cannot be registered: org.opendaylight.controller.cluster.datastore.exceptions.NotInitializedException: Found primary shard member-1-shard-inventory-config but it's not initialized yet. Please try again later
2015-04-29 16:30:30,802 | ERROR | ult-dispatcher-4 | DataTreeChangeListenerProxy | 179 - org.opendaylight.controller.sal-distributed-datastore - 1.2.0.SNAPSHOT | Failed to find local shard inventory - DataTreeChangeListener org.opendaylight.controller.md.sal.binding.impl.BindingDOMDataTreeChangeListenerAdapter@2eebfeaa at path /(urn:opendaylight:inventory?revision=2013-08-19)nodes/node/node/AugmentationIdentifier

/(urn:opendaylight:flow:inventory?revision=2013-08-19)table/table cannot be registered: org.opendaylight.controller.cluster.datastore.exceptions.NotInitializedException: Found primary shard member-1-shard-inventory-config but it's not initialized yet. Please try again later
2015-04-29 16:30:34,074 | ERROR | lt-dispatcher-15 | FlowNodeReconciliationImpl | 188 - org.opendaylight.openflowplugin.applications.forwardingrules-manager - 0.1.0.SNAPSHOT | Fail with read Config/DS for Node InstanceIdentifier

{targetType=interface org.opendaylight.yang.gen.v1.urn.opendaylight.flow.inventory.rev130819.FlowCapableNode, path=[org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.Nodes, org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node[key=NodeKey [_id=Uri [_value=openflow:262584037476162]]], org.opendaylight.yang.gen.v1.urn.opendaylight.flow.inventory.rev130819.FlowCapableNode]}

!
java.util.concurrent.ExecutionException: ReadFailedException{message=Error reading data for path /(urn:opendaylight:inventory?revision=2013-08-19)nodes/node/node[

{(urn:opendaylight:inventory?revision=2013-08-19)id=openflow:262584037476162}

]/AugmentationIdentifier

, errorList=[RpcError [message=Error reading data for path /(urn:opendaylight:inventory?revision=2013-08-19)nodes/node/node[

{(urn:opendaylight:inventory?revision=2013-08-19)id=openflow:262584037476162}

]/AugmentationIdentifier

{(urn:opendaylight:inventory?revision=2013-08-19)id=openflow:262584037476162}

]/AugmentationIdentifier

, errorList=[RpcError [message=Error reading data for path /(urn:opendaylight:inventory?revision=2013-08-19)nodes/node/node[

{(urn:opendaylight:inventory?revision=2013-08-19)id=openflow:262584037476162}

]/AugmentationIdentifier

, severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=org.opendaylight.controller.cluster.datastore.exceptions.NotInitializedException: Found primary shard member-1-shard-inventory-config but it's not initialized yet. Please try again later]]}
at org.opendaylight.controller.cluster.datastore.NoOpTransactionContext.readData(NoOpTransactionContext.java:80)[179:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.TransactionProxy$2.invoke(TransactionProxy.java:186)[179:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.TransactionFutureCallback.executeTxOperatonsOnComplete(TransactionFutureCallback.java:252)[179:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.TransactionFutureCallback.createTransactionContext(TransactionFutureCallback.java:221)[179:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.TransactionProxy$12.onComplete(TransactionProxy.java:512)[179:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.TransactionProxy$12.onComplete(TransactionProxy.java:508)[179:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at akka.dispatch.OnComplete.internal(Future.scala:246)[164:com.typesafe.akka.actor:2.3.9]
at akka.dispatch.OnComplete.internal(Future.scala:244)[164:com.typesafe.akka.actor:2.3.9]
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:174)[164:com.typesafe.akka.actor:2.3.9]
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:171)[164:com.typesafe.akka.actor:2.3.9]
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)[161:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b]
at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67)[164:com.typesafe.akka.actor:2.3.9]
at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82)[164:com.typesafe.akka.actor:2.3.9]
at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)[164:com.typesafe.akka.actor:2.3.9]
at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)[164:com.typesafe.akka.actor:2.3.9]
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)[161:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b]
at akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58)[164:com.typesafe.akka.actor:2.3.9]
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41)[164:com.typesafe.akka.actor:2.3.9]
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:401)[164:com.typesafe.akka.actor:2.3.9]
... 4 more
Caused by: org.opendaylight.controller.cluster.datastore.exceptions.NotInitializedException: Found primary shard member-1-shard-inventory-config but it's not initialized yet. Please try again later
at org.opendaylight.controller.cluster.datastore.ShardManager.createNotInitializedException(ShardManager.java:417)[179:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.ShardManager.onShardNotInitializedTimeout(ShardManager.java:232)[179:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.ShardManager.handleCommand(ShardManager.java:191)[179:org.opendaylight.controller.sal-distributed-datastore:1.2.0.SNAPSHOT]
at org.opendaylight.controller.cluster.common.actor.AbstractUntypedPersistentActor.onReceiveCommand(AbstractUntypedPersistentActor.java:36)[171:org.opendaylight.controller.sal-clustering-commons:1.2.0.SNAPSHOT]
at akka.persistence.UntypedPersistentActor.onReceive(Eventsourced.scala:430)[169:com.typesafe.akka.persistence.experimental:2.3.9]
at org.opendaylight.controller.cluster.common.actor.MeteringBehavior.apply(MeteringBehavior.java:97)[171:org.opendaylight.controller.sal-clustering-commons:1.2.0.SNAPSHOT]
at akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:534)[164:com.typesafe.akka.actor:2.3.9]
at akka.persistence.Recovery$State$class.process(Recovery.scala:30)[169:com.typesafe.akka.persistence.experimental:2.3.9]
at akka.persistence.ProcessorImpl$$anon$2.process(Processor.scala:103)[169:com.typesafe.akka.persistence.experimental:2.3.9]
at akka.persistence.ProcessorImpl$$anon$2.aroundReceive(Processor.scala:114)[169:com.typesafe.akka.persistence.experimental:2.3.9]
at akka.persistence.Recovery$class.aroundReceive(Recovery.scala:265)[169:com.typesafe.akka.persistence.experimental:2.3.9]
at akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(Eventsourced.scala:428)[169:com.typesafe.akka.persistence.experimental:2.3.9]
at akka.persistence.Eventsourced$$anon$2.doAroundReceive(Eventsourced.scala:82)[169:com.typesafe.akka.persistence.experimental:2.3.9]
at akka.persistence.Eventsourced$$anon$2.aroundReceive(Eventsourced.scala:78)[169:com.typesafe.akka.persistence.experimental:2.3.9]
at akka.persistence.Eventsourced$class.aroundReceive(Eventsourced.scala:369)[169:com.typesafe.akka.persistence.experimental:2.3.9]
at akka.persistence.UntypedPersistentActor.aroundReceive(Eventsourced.scala:428)[169:com.typesafe.akka.persistence.experimental:2.3.9]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)[164:com.typesafe.akka.actor:2.3.9]
at akka.actor.ActorCell.invoke(ActorCell.scala:487)[164:com.typesafe.akka.actor:2.3.9]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)[164:com.typesafe.akka.actor:2.3.9]
at akka.dispatch.Mailbox.run(Mailbox.scala:221)[164:com.typesafe.akka.actor:2.3.9]
at akka.dispatch.Mailbox.exec(Mailbox.scala:231)[164:com.typesafe.akka.actor:2.3.9]
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)[161:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b]
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:1253)[161:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b]
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1346)[161:org.scala-lang.scala-library:2.10.4.v20140209-180020-VFINAL-b66a39653b]
... 2 more

Comments

Comment by Jamo Luhrsen [ 04/May/15 ]

This appears to have been resolved, although I could not determine the exact patch that
fixed it. In a local environment, the issue is seen 100% of the time using this build:

distribution-karaf-0.3.0-20150501.190502-1412.zip

Using the next build on nexus, I cannot reproduce it:

distribution-karaf-0.3.0-20150501.193252-1413.zip

However, it's odd that I can see build 1412 passing in CI. I did not expect that to
be the case:

https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-cds-flow-services-lithium-redesign-only-master/111/

Another observation is that with build 1412, it takes much longer for the openflow port
(6633) to come up. The karaf log has some messages from DeadlockMonitor, where as
1413 does not:

EX:
2015-05-04 15:59:20,847 | WARN | saction-50-52'}} | DeadlockMonitor | 113 - org.opendaylight.controller.config-manager - 0.3.0.SNAPSHOT | ModuleIdentifier

{factoryName='distributed-config-datastore-provider', instanceName='distributed-config-store-module'}

did not finish after 88984 ms

Comment by Jamo Luhrsen [ 05/May/15 ]

verified again with nexus build:
distribution-karaf-0.3.0-20150505.173521-1449.zip

Generated at Wed Feb 07 20:32:26 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.

[OPNFLWPLUG-424] [Lithium redesign] cannot push a flow to config DS Created: 29/Apr/15 Updated: 27/Sep/21 Resolved: 05/May/15