[OVSDB-266] Instances are not reachable in Openstack when ODL used as cluster (using OFPlugin He design) Created: 21/Jan/16  Updated: 10/Feb/16  Resolved: 10/Feb/16

Status: Resolved
Project: ovsdb
Component/s: openstack.net-virt
Affects Version/s: unspecified
Fix Version/s: None

Type: Bug
Reporter: Venkatrangan Govindarajan Assignee: Anil Vishnoi
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: All
Platform: All

Attachments: File odl1_log.tgzz     File odl2_log.tgz     File odl3_log.tgz    
External issue ID: 5038
Priority: High

 Description   

Created an ODL cluster with all instances running odl-ovsdb-openstack feature using the OpenFlow Plugin (He) design.

stacked up Openstack Control and Compute nodes, Then

i) created Network + instances
result: Instances do not have the IP, the discover fails

ii) Created router and added one more network and created instances
result: Same behavior

From early observations, the flow entries sem to be missing a lot of items in the chain.

These are the flow entries in control node.
ubuntu@odl-os-control-node:~/devstack$ sudo ovs-ofctl dump-flows br-int -OOpenFlow13
OFPST_FLOW reply (OF1.3) (xid=0x2):
cookie=0x0, duration=2396.606s, table=0, n_packets=2, n_bytes=140, in_port=1,dl_src=fa:16:3e:80:24:a2 actions=set_field:0x401->tun_id,load:0x1->NXM_NX_REG0[],goto_table:20
cookie=0x0, duration=3113.360s, table=0, n_packets=960, n_bytes=107520, dl_type=0x88cc actions=CONTROLLER:65535
cookie=0x0, duration=760.168s, table=20, n_packets=0, n_bytes=0, priority=1024,arp,tun_id=0x3f8,arp_tpa=2.3.4.4,arp_op=1 actions=move:NXM_OF_ETH_SRC[]>NXM_OF_ETH_DST[],set_field:fa:16:3e:04:dc:01>eth_src,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]>NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]>NXM_OF_ARP_TPA[],load:0xfa163e04dc01->NXM_NX_ARP_SHA[],load:0x2030404->NXM_OF_ARP_SPA[],IN_PORT
cookie=0x0, duration=982.965s, table=20, n_packets=0, n_bytes=0, priority=1024,arp,tun_id=0x401,arp_tpa=1.2.3.1,arp_op=1 actions=move:NXM_OF_ETH_SRC[]>NXM_OF_ETH_DST[],set_field:fa:16:3e:aa:3e:78>eth_src,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]>NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]>NXM_OF_ARP_TPA[],load:0xfa163eaa3e78->NXM_NX_ARP_SHA[],load:0x1020301->NXM_OF_ARP_SPA[],IN_PORT
cookie=0x0, duration=3113.572s, table=60, n_packets=0, n_bytes=0, priority=0 actions=goto_table:70
cookie=0x0, duration=2229.565s, table=90, n_packets=0, n_bytes=0, priority=61010,arp,arp_tha=fa:16:3e:e9:a2:e4 actions=goto_table:100

Flow entries in compute node

ubuntu@odl-os-compute-node:~/devstack$ sudo ovs-ofctl dump-flows br-int -OOpenFlow13
OFPST_FLOW reply (OF1.3) (xid=0x2):
cookie=0x0, duration=953.315s, table=20, n_packets=0, n_bytes=0, priority=1024,arp,tun_id=0x3f8,arp_tpa=2.3.4.4,arp_op=1 actions=move:NXM_OF_ETH_SRC[]>NXM_OF_ETH_DST[],set_field:fa:16:3e:04:dc:01>eth_src,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]>NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]>NXM_OF_ARP_TPA[],load:0xfa163e04dc01->NXM_NX_ARP_SHA[],load:0x2030404->NXM_OF_ARP_SPA[],IN_PORT

Will attach the logs.

 Comments   
Comment by Venkatrangan Govindarajan [ 21/Jan/16 ]

Attachment odl1_log.tgzz has been added with description: ODL1 Logs

Comment by Venkatrangan Govindarajan [ 21/Jan/16 ]

Attachment odl2_log.tgz has been added with description: ODL2 logs

Comment by Venkatrangan Govindarajan [ 21/Jan/16 ]

Attachment odl3_log.tgz has been added with description: ODL3 logs

Comment by Venkatrangan Govindarajan [ 21/Jan/16 ]

Exceptions in Logs indicate the following

a.

2016-01-20 18:48:29,866 | ERROR | ntDispatcherImpl | AbstractServiceInstance | 282 - org.opendaylight.ovsdb.openstack.net-virt-providers - 1.2.1.SNAPSHOT | TransactionCommitFailedException{message=Data did not pass validation., errorList=[RpcError [message=Data did not pass validation., severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=org.opendaylight.yangtools.yang.data.api.schema.tree.ModifiedNodeDoesNot

Comment by Venkatrangan Govindarajan [ 21/Jan/16 ]

used: stable/beryllium branch

Comment by Sam Hague [ 22/Jan/16 ]

Anil,

have you seen this exception with clustering? This is coming out in the ODL1 log very early so wondering if that impacts the rest of the testing.

2016-01-20 18:12:45,228 | WARN | ult-dispatcher-3 | ShardManager | 161 - org.opendaylight.controller.sal-distributed-datastore - 1.3.0.SNAPSHOT | Supervisor Strategy caught unexpected exception - resuming
java.lang.NullPointerException
at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:210)[61:com.google.guava:18.0.0]
at org.opendaylight.yangtools.yang.data.api.YangInstanceIdentifier$AbstractPathArgument.<init>(YangInstanceIdentifier.java:413)[76:org.opendaylight.yangtools.yang-data-api:0.8.0.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.api.YangInstanceIdentifier$NodeWithValue.<init>(YangInstanceIdentifier.java:584)[76:org.opendaylight.yangtools.yang-data-api:0.8.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.node.utils.stream.NormalizedNodeInputStreamReader.readNormalizedNodeInternal(NormalizedNodeInputStreamReader.java:138)[157:org.opendaylight.controller.sal-clustering-commons:1.3.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.node.utils.stream.NormalizedNodeInputStreamReader.readNormalizedNode(NormalizedNodeInputStreamReader.java:97)[157:org.opendaylight.controller.sal-clustering-commons:1.3.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.DataTreeCandidatePayload.readNode(DataTreeCandidatePayload.java:184)[161:org.opendaylight.controller.sal-distributed-datastore:1.3.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.DataTreeCandidatePayload.readChildren(DataTreeCandidatePayload.java:146)[161:org.opendaylight.controller.sal-distributed-datastore:1.3.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.DataTreeCandidatePayload.readModifiedNode(DataTreeCandidatePayload.java:161)[161:org.opendaylight.controller.sal-distributed-datastore:1.3.0.SNAPSHOT]

Comment by Sam Hague [ 22/Jan/16 ]

Anil, log2 is showing the below. I am also seeing this in my testing recently. is it possible some openflowplugin-he changes just committed for clustering are having an impact?

2016-01-20 18:12:44,131 | ERROR | ult-dispatcher-4 | LocalThreePhaseCommitCohort | 161 - org.opendaylight.controller.sal-distributed-datastore - 1.3.0.SNAPSHOT | Failed to prepare transaction member-2-chn-5-txn-4 on backend
OptimisticLockFailedException

{message=Optimistic lock failed., errorList=[RpcError [message=Optimistic lock failed., severity=ERROR, errorType=APPLICATION, tag=resource-denied, applicationTag=null, info=null, cause=org.opendaylight.yangtools.yang.data.api.schema.tree.ConflictingModificationAppliedException: Node was created by other transaction.]]}

at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.canCommit(SimpleShardDataTreeCohort.java:54)[161:org.opendaylight.controller.sal-distributed-datastore:1.3.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator$CohortEntry.canCommit(ShardCommitCoordinator.java:662)[161:org.opendaylight.controller.sal-distributed-datastore:1.3.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.doCanCommit(ShardCommitCoordinator.java:330)[161:org.opendaylight.controller.sal-distributed-datastore:1.3.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.handleCanCommit(ShardCommitCoordinator.java:291)[161:org.opendaylight.controller.sal-distributed-datastore:1.3.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.handleReadyLocalTransaction(ShardCommitCoordinator.java:263)[161:org.opendaylight.controller.sal-distributed-datastore:1.3.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.Shard.handleReadyLocalTransaction(Shard.java:470)[161:org.opendaylight.controller.sal-distributed-datastore:1.3.0.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.Shard.onReceiveCommand(Shard.java:236)[161:org.opendaylight.controller.sal-distributed-datastore:1.3.0.

Comment by Anil Vishnoi [ 22/Jan/16 ]

(In reply to Sam Hague from comment #7)
> Anil, log2 is showing the below. I am also seeing this in my testing
> recently. is it possible some openflowplugin-he changes just committed for
> clustering are having an impact?
>
> 2016-01-20 18:12:44,131 | ERROR | ult-dispatcher-4 |
> LocalThreePhaseCommitCohort | 161 -
> org.opendaylight.controller.sal-distributed-datastore - 1.3.0.SNAPSHOT |
> Failed to prepare transaction member-2-chn-5-txn-4 on backend
> OptimisticLockFailedException

{message=Optimistic lock failed., > errorList=[RpcError [message=Optimistic lock failed., severity=ERROR, > errorType=APPLICATION, tag=resource-denied, applicationTag=null, info=null, > cause=org.opendaylight.yangtools.yang.data.api.schema.tree. > ConflictingModificationAppliedException: Node was created by other > transaction.]]}

> at
> org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.
> canCommit(SimpleShardDataTreeCohort.java:54)[161:org.opendaylight.controller.
> sal-distributed-datastore:1.3.0.SNAPSHOT]
> at
> org.opendaylight.controller.cluster.datastore.
> ShardCommitCoordinator$CohortEntry.canCommit(ShardCommitCoordinator.java:
> 662)[161:org.opendaylight.controller.sal-distributed-datastore:1.3.0.
> SNAPSHOT]
> at
> org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.
> doCanCommit(ShardCommitCoordinator.java:330)[161:org.opendaylight.controller.
> sal-distributed-datastore:1.3.0.SNAPSHOT]
> at
> org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.
> handleCanCommit(ShardCommitCoordinator.java:291)[161:org.opendaylight.
> controller.sal-distributed-datastore:1.3.0.SNAPSHOT]
> at
> org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.
> handleReadyLocalTransaction(ShardCommitCoordinator.java:263)[161:org.
> opendaylight.controller.sal-distributed-datastore:1.3.0.SNAPSHOT]
> at
> org.opendaylight.controller.cluster.datastore.Shard.
> handleReadyLocalTransaction(Shard.java:470)[161:org.opendaylight.controller.
> sal-distributed-datastore:1.3.0.SNAPSHOT]
> at
> org.opendaylight.controller.cluster.datastore.Shard.onReceiveCommand(Shard.
> java:236)[161:org.opendaylight.controller.sal-distributed-datastore:1.3.0.

(In reply to Sam Hague from comment #6)
> Anil,
>
> have you seen this exception with clustering? This is coming out in the ODL1
> log very early so wondering if that impacts the rest of the testing.
>
> 2016-01-20 18:12:45,228 | WARN | ult-dispatcher-3 | ShardManager
> | 161 - org.opendaylight.controller.sal-distributed-datastore -
> 1.3.0.SNAPSHOT | Supervisor Strategy caught unexpected exception - resuming
> java.lang.NullPointerException
> at
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:210)[61:
> com.google.guava:18.0.0]
> at
> org.opendaylight.yangtools.yang.data.api.
> YangInstanceIdentifier$AbstractPathArgument.<init>(YangInstanceIdentifier.
> java:413)[76:org.opendaylight.yangtools.yang-data-api:0.8.0.SNAPSHOT]
> at
> org.opendaylight.yangtools.yang.data.api.
> YangInstanceIdentifier$NodeWithValue.<init>(YangInstanceIdentifier.java:
> 584)[76:org.opendaylight.yangtools.yang-data-api:0.8.0.SNAPSHOT]
> at
> org.opendaylight.controller.cluster.datastore.node.utils.stream.
> NormalizedNodeInputStreamReader.
> readNormalizedNodeInternal(NormalizedNodeInputStreamReader.java:138)[157:org.
> opendaylight.controller.sal-clustering-commons:1.3.0.SNAPSHOT]
> at
> org.opendaylight.controller.cluster.datastore.node.utils.stream.
> NormalizedNodeInputStreamReader.
> readNormalizedNode(NormalizedNodeInputStreamReader.java:97)[157:org.
> opendaylight.controller.sal-clustering-commons:1.3.0.SNAPSHOT]
> at
> org.opendaylight.controller.cluster.datastore.DataTreeCandidatePayload.
> readNode(DataTreeCandidatePayload.java:184)[161:org.opendaylight.controller.
> sal-distributed-datastore:1.3.0.SNAPSHOT]
> at
> org.opendaylight.controller.cluster.datastore.DataTreeCandidatePayload.
> readChildren(DataTreeCandidatePayload.java:146)[161:org.opendaylight.
> controller.sal-distributed-datastore:1.3.0.SNAPSHOT]
> at
> org.opendaylight.controller.cluster.datastore.DataTreeCandidatePayload.
> readModifiedNode(DataTreeCandidatePayload.java:161)[161:org.opendaylight.
> controller.sal-distributed-datastore:1.3.0.SNAPSHOT]

I opened bug for this issue So this is the bug i opendhttps://bugs.opendaylight.org/show_bug.cgi?id=5019 and Tom raised the priority to Critical. I think it can impact the functioning of clustering and impact the testing as well.

Comment by Anil Vishnoi [ 22/Jan/16 ]

(In reply to Sam Hague from comment #7)
> Anil, log2 is showing the below. I am also seeing this in my testing
> recently. is it possible some openflowplugin-he changes just committed for
> clustering are having an impact?
>
> 2016-01-20 18:12:44,131 | ERROR | ult-dispatcher-4 |
> LocalThreePhaseCommitCohort | 161 -
> org.opendaylight.controller.sal-distributed-datastore - 1.3.0.SNAPSHOT |
> Failed to prepare transaction member-2-chn-5-txn-4 on backend
> OptimisticLockFailedException

{message=Optimistic lock failed., > errorList=[RpcError [message=Optimistic lock failed., severity=ERROR, > errorType=APPLICATION, tag=resource-denied, applicationTag=null, info=null, > cause=org.opendaylight.yangtools.yang.data.api.schema.tree. > ConflictingModificationAppliedException: Node was created by other > transaction.]]}

> at
> org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.
> canCommit(SimpleShardDataTreeCohort.java:54)[161:org.opendaylight.controller.
> sal-distributed-datastore:1.3.0.SNAPSHOT]
> at
> org.opendaylight.controller.cluster.datastore.
> ShardCommitCoordinator$CohortEntry.canCommit(ShardCommitCoordinator.java:
> 662)[161:org.opendaylight.controller.sal-distributed-datastore:1.3.0.
> SNAPSHOT]
> at
> org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.
> doCanCommit(ShardCommitCoordinator.java:330)[161:org.opendaylight.controller.
> sal-distributed-datastore:1.3.0.SNAPSHOT]
> at
> org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.
> handleCanCommit(ShardCommitCoordinator.java:291)[161:org.opendaylight.
> controller.sal-distributed-datastore:1.3.0.SNAPSHOT]
> at
> org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.
> handleReadyLocalTransaction(ShardCommitCoordinator.java:263)[161:org.
> opendaylight.controller.sal-distributed-datastore:1.3.0.SNAPSHOT]
> at
> org.opendaylight.controller.cluster.datastore.Shard.
> handleReadyLocalTransaction(Shard.java:470)[161:org.opendaylight.controller.
> sal-distributed-datastore:1.3.0.SNAPSHOT]
> at
> org.opendaylight.controller.cluster.datastore.Shard.onReceiveCommand(Shard.
> java:236)[161:org.opendaylight.controller.sal-distributed-datastore:1.3.0.

Yes, it can be both either net-virt or openflowplugin, where two controller instances are trying to modify the same node. I am looking into it.

Comment by Anil Vishnoi [ 02/Feb/16 ]

I believe ConflictingModificationAppliedException is resolved by the recent fixes in openflowplugin and ovsdb code. Also other exception "Supervisor Strategy caught unexpected exception" is also fixed if i look at the status of all the patches mentioned on this bug https://bugs.opendaylight.org/show_bug.cgi?id=5019.

I will test it and will provide more update on it.

Comment by Venkatrangan Govindarajan [ 02/Feb/16 ]

Tested this one today,

ODL1 -> handle Neutron requests

ODL2,ODL3 -> act as OVS Managers and Openflow controllers

-> Flow entries are getting installed for br-int in both control and compute nodes

-> Instances getting created in compute node not able to get IP address assigned

Flow entries in compute node for reference, The instances do not get connectivity. whereas the instacnes in control node are working fine.

---Flow entries Compute Node---
ubuntu@odl-os-compute-node:~/devstack$ sudo ovs-ofctl dump-flows br-int -OOpenFlow13
OFPST_FLOW reply (OF1.3) (xid=0x2):
cookie=0x0, duration=441.799s, table=0, n_packets=9, n_bytes=1467, in_port=5,dl_src=fa:16:3e:3d:13:0e actions=set_field:0x411->tun_id,load:0x1->NXM_NX_REG0[],goto_table:20
cookie=0x0, duration=438.117s, table=0, n_packets=9, n_bytes=1464, in_port=6,dl_src=fa:16:3e:df:07:b7 actions=set_field:0x402->tun_id,load:0x1->NXM_NX_REG0[],goto_table:20
cookie=0x0, duration=1878.343s, table=0, n_packets=9, n_bytes=698, priority=0 actions=goto_table:20
cookie=0x0, duration=441.783s, table=0, n_packets=0, n_bytes=0, priority=8192,in_port=5 actions=drop
cookie=0x0, duration=438.105s, table=0, n_packets=0, n_bytes=0, priority=8192,in_port=6 actions=drop
cookie=0x0, duration=441.671s, table=0, n_packets=0, n_bytes=0, tun_id=0x411,in_port=1 actions=load:0x2->NXM_NX_REG0[],goto_table:20
cookie=0x0, duration=437.983s, table=0, n_packets=0, n_bytes=0, tun_id=0x402,in_port=1 actions=load:0x2->NXM_NX_REG0[],goto_table:20
cookie=0x0, duration=1878.343s, table=0, n_packets=436, n_bytes=48882, dl_type=0x88cc actions=CONTROLLER:65535
cookie=0x0, duration=1878.343s, table=20, n_packets=79, n_bytes=9606, priority=0 actions=goto_table:30
cookie=0x0, duration=444.485s, table=20, n_packets=0, n_bytes=0, priority=1024,arp,tun_id=0x402,arp_tpa=1.2.3.3,arp_op=1 actions=set_field:fa:16:3e:df:07:b7->eth_src,IN_PORT
cookie=0x0, duration=941.990s, table=20, n_packets=0, n_bytes=0, priority=1024,arp,tun_id=0x411,arp_tpa=2.3.4.2,arp_op=1 actions=set_field:fa:16:3e:9d:dd:c6->eth_src,IN_PORT
cookie=0x0, duration=964.590s, table=20, n_packets=0, n_bytes=0, priority=1024,arp,tun_id=0x402,arp_tpa=1.2.3.2,arp_op=1 actions=set_field:fa:16:3e:10:5d:33->eth_src,IN_PORT
cookie=0x0, duration=964.762s, table=20, n_packets=0, n_bytes=0, priority=1024,arp,tun_id=0x402,arp_tpa=1.2.3.4,arp_op=1 actions=set_field:fa:16:3e:27:ef:95->eth_src,IN_PORT
cookie=0x0, duration=942.333s, table=20, n_packets=0, n_bytes=0, priority=1024,arp,tun_id=0x411,arp_tpa=2.3.4.4,arp_op=1 actions=set_field:fa:16:3e:38:b8:7d->eth_src,IN_PORT
cookie=0x0, duration=444.589s, table=20, n_packets=0, n_bytes=0, priority=1024,arp,tun_id=0x411,arp_tpa=2.3.4.3,arp_op=1 actions=set_field:fa:16:3e:3d:13:0e->eth_src,IN_PORT
cookie=0x0, duration=1878.343s, table=30, n_packets=79, n_bytes=9606, priority=0 actions=goto_table:40
cookie=0x0, duration=442.451s, table=40, n_packets=0, n_bytes=0, priority=36001,ip,in_port=5,dl_src=fa:16:3e:3d:13:0e,nw_src=2.3.4.3 actions=goto_table:50
cookie=0x0, duration=439.054s, table=40, n_packets=0, n_bytes=0, priority=36001,ip,in_port=6,dl_src=fa:16:3e:df:07:b7,nw_src=1.2.3.3 actions=goto_table:50
cookie=0x0, duration=442.581s, table=40, n_packets=6, n_bytes=1995, priority=61012,udp,tp_src=68,tp_dst=67 actions=goto_table:50
cookie=0x0, duration=442.565s, table=40, n_packets=0, n_bytes=0, priority=61012,udp6,tp_src=546,tp_dst=547 actions=goto_table:50
cookie=0x0, duration=1878.332s, table=40, n_packets=63, n_bytes=4262, priority=0 actions=goto_table:50
cookie=0x0, duration=436.821s, table=40, n_packets=0, n_bytes=0, priority=61007,ip,dl_src=fa:16:3e:df:07:b7 actions=goto_table:50
cookie=0x0, duration=440.580s, table=40, n_packets=0, n_bytes=0, priority=61007,ip,dl_src=fa:16:3e:3d:13:0e actions=goto_table:50
cookie=0x0, duration=442.482s, table=40, n_packets=0, n_bytes=0, priority=61011,udp6,in_port=5,tp_src=547,tp_dst=546 actions=drop
cookie=0x0, duration=439.075s, table=40, n_packets=0, n_bytes=0, priority=61011,udp6,in_port=6,tp_src=547,tp_dst=546 actions=drop
cookie=0x0, duration=442.521s, table=40, n_packets=0, n_bytes=0, priority=61011,udp,in_port=5,tp_src=67,tp_dst=68 actions=drop
cookie=0x0, duration=439.112s, table=40, n_packets=0, n_bytes=0, priority=61011,udp,in_port=6,tp_src=67,tp_dst=68 actions=drop
cookie=0x0, duration=442.538s, table=40, n_packets=0, n_bytes=0, priority=61010,arp,arp_sha=fa:16:3e:3d:13:0e actions=goto_table:50
cookie=0x0, duration=439.129s, table=40, n_packets=0, n_bytes=0, priority=61010,arp,arp_sha=fa:16:3e:df:07:b7 actions=goto_table:50
cookie=0x0, duration=1878.304s, table=50, n_packets=79, n_bytes=9606, priority=0 actions=goto_table:60
cookie=0x0, duration=1878.289s, table=60, n_packets=79, n_bytes=9606, priority=0 actions=goto_table:70
cookie=0x0, duration=1878.269s, table=70, n_packets=79, n_bytes=9606, priority=0 actions=goto_table:80
cookie=0x0, duration=1878.246s, table=80, n_packets=79, n_bytes=9606, priority=0 actions=goto_table:90
cookie=0x0, duration=1878.233s, table=90, n_packets=79, n_bytes=9606, priority=0 actions=goto_table:100
cookie=0x0, duration=440.979s, table=90, n_packets=0, n_bytes=0, priority=61007,ip,dl_dst=fa:16:3e:3d:13:0e actions=goto_table:100
cookie=0x0, duration=437.205s, table=90, n_packets=0, n_bytes=0, priority=61007,ip,dl_dst=fa:16:3e:df:07:b7 actions=goto_table:100
cookie=0x0, duration=439.198s, table=90, n_packets=0, n_bytes=0, priority=61010,arp,arp_tha=fa:16:3e:df:07:b7 actions=goto_table:100
cookie=0x0, duration=442.601s, table=90, n_packets=0, n_bytes=0, priority=61010,arp,arp_tha=fa:16:3e:3d:13:0e actions=goto_table:100
cookie=0x0, duration=442.684s, table=90, n_packets=0, n_bytes=0, priority=61006,udp,dl_src=fa:16:3e:9d:dd:c6,tp_src=67,tp_dst=68 actions=goto_table:100
cookie=0x0, duration=439.212s, table=90, n_packets=0, n_bytes=0, priority=61006,udp6,dl_src=fa:16:3e:10:5d:33,tp_src=547,tp_dst=546 actions=goto_table:100
cookie=0x0, duration=442.629s, table=90, n_packets=0, n_bytes=0, priority=61006,udp6,dl_src=fa:16:3e:9d:dd:c6,tp_src=547,tp_dst=546 actions=goto_table:100
cookie=0x0, duration=439.238s, table=90, n_packets=0, n_bytes=0, priority=61006,udp,dl_src=fa:16:3e:10:5d:33,tp_src=67,tp_dst=68 actions=goto_table:100
cookie=0x0, duration=1878.213s, table=100, n_packets=79, n_bytes=9606, priority=0 actions=goto_table:110
cookie=0x0, duration=438.014s, table=110, n_packets=0, n_bytes=0, priority=8192,tun_id=0x402 actions=drop
cookie=0x0, duration=441.689s, table=110, n_packets=0, n_bytes=0, priority=8192,tun_id=0x411 actions=drop
cookie=0x0, duration=1878.200s, table=110, n_packets=9, n_bytes=698, priority=0 actions=drop
cookie=0x0, duration=441.725s, table=110, n_packets=9, n_bytes=1467, priority=16383,reg0=0x1,tun_id=0x411,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=output:5,output:1
cookie=0x0, duration=441.747s, table=110, n_packets=0, n_bytes=0, priority=16384,reg0=0x2,tun_id=0x411,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=output:5
cookie=0x0, duration=438.053s, table=110, n_packets=9, n_bytes=1464, priority=16383,reg0=0x1,tun_id=0x402,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=output:6,output:1
cookie=0x0, duration=438.073s, table=110, n_packets=0, n_bytes=0, priority=16384,reg0=0x2,tun_id=0x402,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=output:6
cookie=0x0, duration=1217.125s, table=110, n_packets=0, n_bytes=0, tun_id=0x402,dl_dst=fa:16:3e:10:5d:33 actions=output:1
cookie=0x0, duration=934.812s, table=110, n_packets=0, n_bytes=0, tun_id=0x411,dl_dst=fa:16:3e:38:b8:7d actions=output:1
cookie=0x0, duration=438.089s, table=110, n_packets=0, n_bytes=0, tun_id=0x402,dl_dst=fa:16:3e:df:07:b7 actions=output:6
cookie=0x0, duration=441.767s, table=110, n_packets=0, n_bytes=0, tun_id=0x411,dl_dst=fa:16:3e:3d:13:0e actions=output:5
cookie=0x0, duration=960.787s, table=110, n_packets=0, n_bytes=0, tun_id=0x402,dl_dst=fa:16:3e:27:ef:95 actions=output:1
cookie=0x0, duration=1197.287s, table=110, n_packets=0, n_bytes=0, tun_id=0x411,dl_dst=fa:16:3e:9d:dd:c6 actions=output:1

-----------------

----Flow entries control Node---

ubuntu@odl-os-control-node:~/devstack$ sudo ovs-ofctl dump-flows br-int -OOpenflow13
OFPST_FLOW reply (OF1.3) (xid=0x2):
cookie=0x0, duration=1326.588s, table=0, n_packets=13, n_bytes=1488, in_port=4,dl_src=fa:16:3e:9d:dd:c6 actions=set_field:0x411->tun_id,load:0x1->NXM_NX_REG0[],goto_table:20
cookie=0x0, duration=1089.925s, table=0, n_packets=35, n_bytes=2278, in_port=5,dl_src=fa:16:3e:27:ef:95 actions=set_field:0x402->tun_id,load:0x1->NXM_NX_REG0[],goto_table:20
cookie=0x0, duration=1347.140s, table=0, n_packets=10, n_bytes=1230, in_port=1,dl_src=fa:16:3e:10:5d:33 actions=set_field:0x402->tun_id,load:0x1->NXM_NX_REG0[],goto_table:20
cookie=0x0, duration=1063.960s, table=0, n_packets=35, n_bytes=2280, in_port=6,dl_src=fa:16:3e:38:b8:7d actions=set_field:0x411->tun_id,load:0x1->NXM_NX_REG0[],goto_table:20
cookie=0x0, duration=2378.348s, table=0, n_packets=3, n_bytes=258, priority=0 actions=goto_table:20
cookie=0x0, duration=1063.952s, table=0, n_packets=0, n_bytes=0, priority=8192,in_port=6 actions=drop
cookie=0x0, duration=1089.907s, table=0, n_packets=0, n_bytes=0, priority=8192,in_port=5 actions=drop
cookie=0x0, duration=1347.088s, table=0, n_packets=0, n_bytes=0, priority=8192,in_port=1 actions=drop
cookie=0x0, duration=1326.562s, table=0, n_packets=0, n_bytes=0, priority=8192,in_port=4 actions=drop
cookie=0x0, duration=1346.139s, table=0, n_packets=18, n_bytes=2928, tun_id=0x402,in_port=3 actions=goto_table:20
cookie=0x0, duration=1326.313s, table=0, n_packets=18, n_bytes=2934, tun_id=0x411,in_port=3 actions=goto_table:20
cookie=0x0, duration=2378.227s, table=0, n_packets=540, n_bytes=60480, dl_type=0x88cc actions=CONTROLLER:65535
cookie=0x0, duration=2378.348s, table=20, n_packets=90, n_bytes=11632, priority=0 actions=goto_table:30
cookie=0x0, duration=573.343s, table=20, n_packets=0, n_bytes=0, priority=1024,arp,tun_id=0x402,arp_tpa=1.2.3.3,arp_op=1 actions=set_field:fa:16:3e:df:07:b7->eth_src,IN_PORT
cookie=0x0, duration=1098.051s, table=20, n_packets=3, n_bytes=126, priority=1024,arp,tun_id=0x402,arp_tpa=1.2.3.4,arp_op=1 actions=set_field:fa:16:3e:27:ef:95->eth_src,IN_PORT
cookie=0x0, duration=1347.179s, table=20, n_packets=18, n_bytes=756, priority=1024,arp,tun_id=0x402,arp_tpa=1.2.3.2,arp_op=1 actions=set_field:fa:16:3e:10:5d:33->eth_src,IN_PORT
cookie=0x0, duration=1326.795s, table=20, n_packets=18, n_bytes=756, priority=1024,arp,tun_id=0x411,arp_tpa=2.3.4.2,arp_op=1 actions=set_field:fa:16:3e:9d:dd:c6->eth_src,IN_PORT
cookie=0x0, duration=1074.555s, table=20, n_packets=3, n_bytes=126, priority=1024,arp,tun_id=0x411,arp_tpa=2.3.4.4,arp_op=1 actions=set_field:fa:16:3e:38:b8:7d->eth_src,IN_PORT
cookie=0x0, duration=573.458s, table=20, n_packets=0, n_bytes=0, priority=1024,arp,tun_id=0x411,arp_tpa=2.3.4.3,arp_op=1 actions=set_field:fa:16:3e:3d:13:0e->eth_src,IN_PORT
cookie=0x0, duration=2378.335s, table=30, n_packets=90, n_bytes=11632, priority=0 actions=goto_table:40
cookie=0x0, duration=1090.405s, table=40, n_packets=0, n_bytes=0, priority=36001,ip,in_port=5,dl_src=fa:16:3e:27:ef:95,nw_src=1.2.3.4 actions=goto_table:50
cookie=0x0, duration=1065.289s, table=40, n_packets=0, n_bytes=0, priority=36001,ip,in_port=6,dl_src=fa:16:3e:38:b8:7d,nw_src=2.3.4.4 actions=goto_table:50
cookie=0x0, duration=2378.286s, table=40, n_packets=56, n_bytes=5532, priority=0 actions=goto_table:50
cookie=0x0, duration=1347.279s, table=40, n_packets=16, n_bytes=5344, priority=61012,udp,tp_src=68,tp_dst=67 actions=goto_table:50
cookie=0x0, duration=1347.279s, table=40, n_packets=0, n_bytes=0, priority=61012,udp6,tp_src=546,tp_dst=547 actions=goto_table:50
cookie=0x0, duration=1062.540s, table=40, n_packets=0, n_bytes=0, priority=61007,ip,dl_src=fa:16:3e:38:b8:7d actions=goto_table:50
cookie=0x0, duration=1089.090s, table=40, n_packets=0, n_bytes=0, priority=61007,ip,dl_src=fa:16:3e:27:ef:95 actions=goto_table:50
cookie=0x0, duration=1090.432s, table=40, n_packets=0, n_bytes=0, priority=61011,udp6,in_port=5,tp_src=547,tp_dst=546 actions=drop
cookie=0x0, duration=1065.322s, table=40, n_packets=0, n_bytes=0, priority=61011,udp6,in_port=6,tp_src=547,tp_dst=546 actions=drop
cookie=0x0, duration=1090.464s, table=40, n_packets=0, n_bytes=0, priority=61011,udp,in_port=5,tp_src=67,tp_dst=68 actions=drop
cookie=0x0, duration=1065.326s, table=40, n_packets=0, n_bytes=0, priority=61011,udp,in_port=6,tp_src=67,tp_dst=68 actions=drop
cookie=0x0, duration=1090.502s, table=40, n_packets=9, n_bytes=378, priority=61010,arp,arp_sha=fa:16:3e:27:ef:95 actions=goto_table:50
cookie=0x0, duration=1065.326s, table=40, n_packets=9, n_bytes=378, priority=61010,arp,arp_sha=fa:16:3e:38:b8:7d actions=goto_table:50
cookie=0x0, duration=2378.348s, table=50, n_packets=90, n_bytes=11632, priority=0 actions=goto_table:60
cookie=0x0, duration=2378.335s, table=60, n_packets=90, n_bytes=11632, priority=0 actions=goto_table:70
cookie=0x0, duration=2378.348s, table=70, n_packets=90, n_bytes=11632, priority=0 actions=goto_table:80
cookie=0x0, duration=2378.348s, table=80, n_packets=90, n_bytes=11632, priority=0 actions=goto_table:90
cookie=0x0, duration=2378.336s, table=90, n_packets=86, n_bytes=10204, priority=0 actions=goto_table:100
cookie=0x0, duration=572.414s, table=90, n_packets=0, n_bytes=0, priority=61007,ip,dl_dst=fa:16:3e:38:b8:7d actions=goto_table:100
cookie=0x0, duration=572.358s, table=90, n_packets=0, n_bytes=0, priority=61007,ip,dl_dst=fa:16:3e:27:ef:95 actions=goto_table:100
cookie=0x0, duration=1090.573s, table=90, n_packets=0, n_bytes=0, priority=61010,arp,arp_tha=fa:16:3e:27:ef:95 actions=goto_table:100
cookie=0x0, duration=1065.407s, table=90, n_packets=0, n_bytes=0, priority=61010,arp,arp_tha=fa:16:3e:38:b8:7d actions=goto_table:100
cookie=0x0, duration=1065.468s, table=90, n_packets=0, n_bytes=0, priority=61006,udp,dl_src=fa:16:3e:9d:dd:c6,tp_src=67,tp_dst=68 actions=goto_table:100
cookie=0x0, duration=1065.435s, table=90, n_packets=0, n_bytes=0, priority=61006,udp6,dl_src=fa:16:3e:9d:dd:c6,tp_src=547,tp_dst=546 actions=goto_table:100
cookie=0x0, duration=1090.592s, table=90, n_packets=0, n_bytes=0, priority=61006,udp6,dl_src=fa:16:3e:10:5d:33,tp_src=547,tp_dst=546 actions=goto_table:100
cookie=0x0, duration=1090.607s, table=90, n_packets=0, n_bytes=0, priority=61006,udp,dl_src=fa:16:3e:10:5d:33,tp_src=67,tp_dst=68 actions=goto_table:100
cookie=0x0, duration=2378.335s, table=100, n_packets=90, n_bytes=11632, priority=0 actions=goto_table:110
cookie=0x0, duration=1346.162s, table=110, n_packets=18, n_bytes=2928, priority=8192,tun_id=0x402 actions=drop
cookie=0x0, duration=1326.314s, table=110, n_packets=18, n_bytes=2934, priority=8192,tun_id=0x411 actions=drop
cookie=0x0, duration=2378.233s, table=110, n_packets=5, n_bytes=418, priority=0 actions=drop
cookie=0x0, duration=1326.428s, table=110, n_packets=25, n_bytes=2172, priority=16383,reg0=0x1,tun_id=0x411,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=output:4,output:3,output:6
cookie=0x0, duration=1326.493s, table=110, n_packets=0, n_bytes=0, priority=16384,reg0=0x2,tun_id=0x411,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=output:4,output:6
cookie=0x0, duration=1346.127s, table=110, n_packets=20, n_bytes=1752, priority=16383,reg0=0x1,tun_id=0x402,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=output:1,output:3,output:5
cookie=0x0, duration=1346.124s, table=110, n_packets=0, n_bytes=0, priority=16384,reg0=0x2,tun_id=0x402,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=output:1,output:5
cookie=0x0, duration=1346.995s, table=110, n_packets=0, n_bytes=0, tun_id=0x402,dl_dst=fa:16:3e:10:5d:33 actions=output:1
cookie=0x0, duration=1063.933s, table=110, n_packets=2, n_bytes=714, tun_id=0x411,dl_dst=fa:16:3e:38:b8:7d actions=output:6
cookie=0x0, duration=566.842s, table=110, n_packets=0, n_bytes=0, tun_id=0x402,dl_dst=fa:16:3e:df:07:b7 actions=output:3
cookie=0x0, duration=570.549s, table=110, n_packets=0, n_bytes=0, tun_id=0x411,dl_dst=fa:16:3e:3d:13:0e actions=output:3
cookie=0x0, duration=1089.885s, table=110, n_packets=2, n_bytes=714, tun_id=0x402,dl_dst=fa:16:3e:27:ef:95 actions=output:5
cookie=0x0, duration=1326.537s, table=110, n_packets=0, n_bytes=0, tun_id=0x411,dl_dst=fa:16:3e:9d:dd:c6 actions=output:4

Comment by Venkatrangan Govindarajan [ 03/Feb/16 ]

Programming tunnel flows are failing

93 | INFO | ntDispatcherImpl | OF13Provider | 282 - org.opendaylight.ovsdb.openstack.net-virt-providers - 1.2.1.SNAPSHOT | Tunnel vxlan-12.14.16.3 is present in br-int of ovsdb://uuid/7379aa64-0416-45e2-9531-351397cc742a/bridge/br-int
2016-02-02 20:16:05,193 | INFO | ntDispatcherImpl | OF13Provider | 282 - org.opendaylight.ovsdb.openstack.net-virt-providers - 1.2.1.SNAPSHOT | programTunnelRules: could not find ofPort for Port tap496a2a2b-bb on NodeUri [_value=ovsdb://uuid/bf460c25-5849-4874-9758-4fcbafbf9f5e/bridge/br-int]
2016-02-02 20:16:05,193 | INFO | ntDispatcherImpl | OF13Provider | 282 - org.opendaylight.ovsdb.openstack.net-virt-providers - 1.2.1.SNAPSHOT | programTunnelRules: could not find ofPort for Port tap496a2a2b-bb on NodeUri [_value=ovsdb://uuid/7379aa64-0416-45e2-9531-351397cc742a/bridge/br-int]
@ @

Comment by Anil Vishnoi [ 09/Feb/16 ]

Hi Venkat,

I did testing with the latest stable/beryllium branch and i don't see this issue any more. I can see spawned VM gets the IP address and it can ping other spawned VM as well.

Can you please test it and close this bug.

Comment by Venkatrangan Govindarajan [ 10/Feb/16 ]

Yeah Now we observe all instacnes are able to reach dhcp and acquire IP address

Generated at Wed Feb 07 20:35:56 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.