[NETVIRT-206] Flows do not exists after OVS restarted on Controller Node and Compute Node. Created: 17/Oct/16  Updated: 09/Mar/18  Resolved: 05/Dec/16

Status: Resolved
Project: netvirt
Component/s: None
Affects Version/s: Boron
Fix Version/s: None

Type: Bug
Reporter: Karthikeyan Assignee: Bertrand Low
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: All
Platform: All


Attachments: Zip Archive ODL1.zip     Zip Archive ODL2.zip     Zip Archive ODL3.zip    
External issue ID: 6944

 Description   

Steps to reproduce in a 3 node cluster:

Used stable/bo.

Used ODL 3 node (feature: odl-ovsdb-openstack)

2 Openstack nodes, 1 control and 1 compute node

1.create a network1.

Below are steps of restart OVS on Control Node.

2.Stop OVS with the following command.
 systemctl stop openvswitch

3.Check ODL log to confirm that processing of OVS stop are finished.

4.Start OVS with the following command.
 systemctl start openvswitch

Now checked the flows, but flows are not existed.
Out of 3 attempts we can able to reproduce one time.

Used following distribution when we faced this issue

https://nexus.opendaylight.org/content/groups/public/org/opendaylight/integration/distribution-karaf/0.5.0-Boron/distribution-karaf-0.5.0-Boron.tar.gz



 Comments   
Comment by Karthikeyan [ 17/Oct/16 ]

Attachment ODL1.zip has been added with description: karaf logs for odl1.

Comment by Karthikeyan [ 17/Oct/16 ]

Attachment ODL2.zip has been added with description: karaf logs for odl2.

Comment by Karthikeyan [ 17/Oct/16 ]

Attachment ODL3.zip has been added with description: karaf logs for odl3

Comment by Karthikeyan [ 17/Oct/16 ]

ModifiedNodeDoesNotExistException occurred,

2016-10-14 15:20:55,598 | WARN | ult-dispatcher-4 | ShardDataTree | 193 - org.opendaylight.controller.sal-distributed-datastore - 1.4.0.Boron | member-1-shard-inventory-config: Store Tx member-1-datastore-config-fe-0-txn-476: Data validation failed for path /(urn:opendaylight:inventory?revision=2013-08-19)nodes/node/node[

{(urn:opendaylight:inventory?revision=2013-08-19)id=openflow:97626771813463}

].
org.opendaylight.yangtools.yang.data.api.schema.tree.ModifiedNodeDoesNotExistException: Node /(urn:opendaylight:inventory?revision=2013-08-19)nodes/node/node[

{(urn:opendaylight:inventory?revision=2013-08-19)id=openflow:97626771813463}

] does not exist. Cannot apply modification to its children.
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkTouchApplicable(AbstractNodeContainerModificationStrategy.java:281)[81:org.opendaylight.yangtools.yang-data-impl:1.0.0.Boron]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:125)[81:org.opendaylight.yangtools.yang-data-impl:1.0.0.Boron]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:305)[81:org.opendaylight.yangtools.yang-data-impl:1.0.0.Boron]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkTouchApplicable(AbstractNodeContainerModificationStrategy.java:288)[81:org.opendaylight.yangtools.yang-data-impl:1.0.0.Boron]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.MinMaxElementsValidation.checkTouchApplicable(MinMaxElementsValidation.java:106)[81:org.opendaylight.yangtools.yang-data-impl:1.0.0.Boron]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:125)[81:org.opendaylight.yangtools.yang-data-impl:1.0.0.Boron]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:305)[81:org.opendaylight.yangtools.yang-data-impl:1.0.0.Boron]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkTouchApplicable(AbstractNodeContainerModificationStrategy.java:288)[81:org.opendaylight.yangtools.yang-data-impl:1.0.0.Boron]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:125)[81:org.opendaylight.yangtools.yang-data-impl:1.0.0.Boron]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.StructuralContainerModificationStrategy.checkApplicable(StructuralContainerModificationStrategy.java:99)[81:org.opendaylight.yangtools.yang-data-impl:1.0.0.Boron]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:305)[81:org.opendaylight.yangtools.yang-data-impl:1.0.0.Boron]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkTouchApplicable(AbstractNodeContainerModificationStrategy.java:288)[81:org.opendaylight.yangtools.yang-data-impl:1.0.0.Boron]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:125)[81:org.opendaylight.yangtools.yang-data-impl:1.0.0.Boron]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.RootModificationApplyOperation.checkApplicable(RootModificationApplyOperation.java:72)[81:org.opendaylight.yangtools.yang-data-impl:1.0.0.Boron]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractDataTreeTip.validate(AbstractDataTreeTip.java:37)[81:org.opendaylight.yangtools.yang-data-impl:1.0.0.Boron]
at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextTransaction(ShardDataTree.java:524)[193:org.opendaylight.controller.sal-distributed-datastore:1.4.0.Boron]
at org.opendaylight.controller.cluster.datastore.ShardDataTree.startCanCommit(ShardDataTree.java:560)[193:org.opendaylight.controller.sal-distributed-datastore:1.4.0.Boron]
at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.canCommit(SimpleShardDataTreeCohort.java:81)[193:org.opendaylight.controller.sal-distributed-datastore:1.4.0.Boron]
at org.opendaylight.controller.cluster.datastore.CohortEntry.canCommit(CohortEntry.java:98)[193:org.opendaylight.controller.sal-distributed-datastore:1.4.0.Boron]
at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.handleCanCommit(ShardCommitCoordinator.java:237)[193:org.opendaylight.controller.sal-distributed-datastore:1.4.0.Boron]
at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.handleReadyLocalTransaction(ShardCommitCoordinator.java:201)[193:org.opendaylight.controller.sal-distributed-datastore:1.4.0.Boron]
at org.opendaylight.controller.cluster.datastore.Shard.handleReadyLocalTransaction(Shard.java:433)[193:org.opendaylight.controller.sal-distributed-datastore:1.4.0.Boron]
at org.opendaylight.controller.cluster.datastore.Shard.handleNonRaftCommand(Shard.java:239)[193:org.opendaylight.controller.sal-distributed-datastore:1.4.0.Boron]
at org.opendaylight.controller.cluster.raft.RaftActor.handleCommand(RaftActor.java:291)[189:org.opendaylight.controller.sal-akka-raft:1.4.0.Boron]
at org.opendaylight.controller.cluster.common.actor.AbstractUntypedPersistentActor.onReceiveCommand(AbstractUntypedPersistentActor.java:29)[188:org.opendaylight.controller.sal-clustering-commons:1.4.0.Boron]
at akka.persistence.UntypedPersistentActor.onReceive(PersistentActor.scala:170)[182:com.typesafe.akka.persistence:2.4.7]
at org.opendaylight.controller.cluster.common.actor.MeteringBehavior.apply(MeteringBehavior.java:97)[188:org.opendaylight.controller.sal-clustering-commons:1.4.0.Boron]
at akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:544)[176:com.typesafe.akka.actor:2.4.7]
at akka.actor.Actor$class.aroundReceive(Actor.scala:484)[176:com.typesafe.akka.actor:2.4.7]
at akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(PersistentActor.scala:168)[182:com.typesafe.akka.persistence:2.4.7]
at akka.persistence.Eventsourced$$anon$1.stateReceive(Eventsourced.scala:633)[182:com.typesafe.akka.persistence:2.4.7]
at akka.persistence.Eventsourced$class.aroundReceive(Eventsourced.scala:179)[182:com.typesafe.akka.persistence:2.4.7]
at akka.persistence.UntypedPersistentActor.aroundReceive(PersistentActor.scala:168)[182:com.typesafe.akka.persistence:2.4.7]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)[176:com.typesafe.akka.actor:2.4.7]
at akka.actor.ActorCell.invoke(ActorCell.scala:495)[176:com.typesafe.akka.actor:2.4.7]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)[176:com.typesafe.akka.actor:2.4.7]
at akka.dispatch.Mailbox.run(Mailbox.scala:224)[176:com.typesafe.akka.actor:2.4.7]
at akka.dispatch.Mailbox.exec(Mailbox.scala:234)[176:com.typesafe.akka.actor:2.4.7]
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)[172:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)[172:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)[172:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)[172:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]

Comment by Venkatrangan Govindarajan [ 17/Oct/16 ]

karthikeyan,

Few clarifications are needed here. When the service is started, what is the output of sudo ovs-vsctl show.

a. Did Netvirt create the br-int after the restart and set-manager again?

b. Also, Did you create instances and attempt the ping?, am wondering the flows related to tunnel are programmed when instances are created. Can you please provide more details so that we can check this immediately?

Comment by Karthikeyan [ 18/Oct/16 ]

(In reply to Venkatrangan Govindarajan from comment #4)
> karthikeyan,
>
> Few clarifications are needed here. When the service is started, what is
> the output of sudo ovs-vsctl show.
>
> a. Did Netvirt create the br-int after the restart and set-manager again?
>
> b. Also, Did you create instances and attempt the ping?, am wondering the
> flows related to tunnel are programmed when instances are created. Can you
> please provide more details so that we can check this immediately?

When the service is started, there is no flows installed. and Yes, br-int is created after the restart. after set-manager, again there is no flows installed.

We created instance's, and while checking IP is not assigned. but flows are installed only related to VM (table:20, 40, 90 and 110).

Comment by Bertrand Low [ 05/Nov/16 ]

This patch for NETVIRT-240 should also address this issue:
https://git.opendaylight.org/gerrit/#/c/48024

Comment by wangqianyu [ 14/Nov/16 ]

(In reply to Bertrand Low from comment #6)
> This patch for NETVIRT-240 should also address this issue:
> https://git.opendaylight.org/gerrit/#/c/48024

if ovs connected to odl in active mode, the stop of ovs will cause configured and optional data be delete in odl.

Comment by Bertrand Low [ 16/Nov/16 ]

Hi QianYu,

the patch for NETVIRT-240 appears to resolve this bug now. Do you agree that we can close this bug as well now?

thanks,
Bertrand

Comment by Bertrand Low [ 29/Nov/16 ]

Bug has been fixed by patch https://git.opendaylight.org/gerrit/#/c/48024

Fix has been verified so bug can be closed.

Comment by wangqianyu [ 30/Nov/16 ]

(In reply to Bertrand Low from comment #8)
> Hi QianYu,
>
> the patch for NETVIRT-240 appears to resolve this bug now. Do you agree that we
> can close this bug as well now?
>
> thanks,
> Bertrand

yes, i agreed. please close this bug

thanks

Comment by Karthikeyan [ 05/Dec/16 ]

Bug has been fixed by the below patch,
https://git.opendaylight.org/gerrit/#/c/48024

Hence closing this bug.

Generated at Wed Feb 07 20:20:58 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.