[NETVIRT-762] Issue in external network communication when multiple networks are configured Created: 05/Jul/17  Updated: 08/Apr/19  Resolved: 10/May/18

Status: Resolved
Project: netvirt
Component/s: General
Affects Version/s: Carbon
Fix Version/s: Fluorine

Type: Bug Priority: Medium
Reporter: YOGA LAKSHMI SWETHA PAYYAVULA Assignee: Vinh Nguyen
Resolution: Won't Do Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: All
Platform: All

Attachments: PNG File external_UC1.PNG     PNG File external_UC2.PNG     Text File flows_8802.txt    
External issue ID: 8802

 Description   

Issue Occurrence: When two internal networks are created and with one of the internal network communication is established with the external network.

Set-up details: ODL Cluster running carbon distro, 1 control and 2 compute nodes with ocata(Packstack)

Reproducing the Issue:

1. Create Internal Network and Subnet (net1)
2. Create a VM for this network.(vm1)
3. Create external network and subnet (externl_net1)
4. Create a router and set gateway to the external network and add a interface to the internal network (net1)
5. Create a floating ip and associate it to vm1.
6. Check for the communication from vm1 to external host - SUCCESS
7. Now create another internal network and subnet(net2)
8. Create a VM for this network (vm2)
9. Create a router and add both the network's interface.
10. In the vm1 add the route gateway for the net2 and check for communication from vm1 to external host - FAIL

This issue can be reproduced even when we create both the internal networks one after another and then the external network. Then create one router for both the internal networks and another router for the internal and external network.
Here when we try to establish the communication between the internal and external host it fails .

 Comments   
Comment by YOGA LAKSHMI SWETHA PAYYAVULA [ 27/Jul/17 ]

Adding route gateway for net2 meaning, adding the route entry for the second network in the VM1 of the network1. PFB for the command:

sudo ip route add <network2> via <gateway_ip associated>

Comment by balakrishnan k [ 21/Aug/17 ]

Attachment flows_8802.txt has been added with description: flows

Comment by balakrishnan k [ 21/Aug/17 ]

Tested the case below are the observation
After step 9 when we attach net1 interface to second router the flow installed for first router was replaced by new router interface created for the net1.
there are multiple issue in pipeline when we attach the network to second router.
issue1: table 21 flow getting replaced by new router interface .
initial flow :
net1 router interface attached to router 1(50.0.0.1)
cookie=0x8000003, duration=9077.371s, table=21, n_packets=0, n_bytes=0, priority=42,icmp,metadata=0x324b8/0xfffffe,nw_dst=50.0.0.1,icmp_type=8,icmp_code=0 actions=move:NXM_OF_ETH_SRC[]>NXM_OF_ETH_DST[],set_field:fa:16:3e:06:66:cd>eth_src,move:NXM_OF_IP_SRC[]>NXM_OF_IP_DST[],set_field:50.0.0.1>ip_src,set_field:0->icmp_type,load:0->NXM_OF_IN_PORT[],resubmit(,21)

after attaching the net1 to router2 (50.0.0.3)

cookie=0x8000003, duration=4574.199s, table=21, n_packets=0, n_bytes=0, priority=42,icmp,metadata=0x324c6/0xfffffe,nw_dst=50.0.0.3,icmp_type=8,icmp_code=0 actions=move:NXM_OF_ETH_SRC[]>NXM_OF_ETH_DST[],set_field:fa:16:3e:8c:d0:4f>eth_src,move:NXM_OF_IP_SRC[]>NXM_OF_IP_DST[],set_field:50.0.0.3>ip_src,set_field:0->icmp_type,load:0->NXM_OF_IN_PORT[],resubmit(,21)

50.0.0.1 flow replaced by 50.0.0.3

issue2: table 81 ARP flow not added for new interface (50.0.0.3)

issue3: table 81 ARP flow MAC changed for old interface. (50.0.0.1)
before attaching sceond interface.
cookie=0x3a220004, duration=731.277s, table=81, n_packets=1, n_bytes=42, priority=100,arp,metadata=0x90000200000324b8/0xffffff0000fffffe,arp_tpa=50.0.0.1,arp_op=1 actions=move:NXM_OF_ETH_SRC[]>NXM_OF_ETH_DST[],set_field:fa:16:3e:06:66:cd>eth_src,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]>NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]>NXM_OF_ARP_TPA[],load:0xfa163e0666cd->NXM_NX_ARP_SHA[],load:0x32000001->NXM_OF_ARP_SPA[],load:0->NXM_OF_IN_PORT[],load:0x200->NXM_NX_REG6[],resubmit(,220)

after attaching second router interface (50.0.0.3)

cookie=0x3a220004, duration=4574.196s, table=81, n_packets=0, n_bytes=0, priority=100,arp,metadata=0x90000200000324c6/0xffffff0000fffffe,arp_tpa=50.0.0.1,arp_op=1 actions=move:NXM_OF_ETH_SRC[]>NXM_OF_ETH_DST[],set_field:fe:16:3e:d1:ca:0d>eth_src,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]>NXM_NX_ARP_THA[],move:NXM_OF_ARP_SPA[]>NXM_OF_ARP_TPA[],load:0xfe163ed1ca0d->NXM_NX_ARP_SHA[],load:0x32000001->NXM_OF_ARP_SPA[],load:0->NXM_OF_IN_PORT[],load:0x200->NXM_NX_REG6[],resubmit(,220)

issue 4: group table flow modified.
attached the commplete flow for before and after adding second router interface for net1.

refer : "attachment 1965"

Comment by balakrishnan k [ 22/Aug/17 ]

Current implementation doesn't support multiple router for same subnet.
Marking this as enhancement Bug.

Comment by Vivekanandan Narasimhan [ 22/Aug/17 ]

Please mark the same bug as an enhancement, as what you ask really is 

–
Thanks,

Vivek

From: Balakrishnan Karuppasamy balakrishnan.ka@hcl.com
Sent: Tuesday, August 22, 2017 3:48 PM
To: N Vivekanandan <n.vivekanandan@ericsson.com>; Chetan Arakere Gowdru <chetan.arakere@altencalsoftlabs.com>; 'Aswin Suryanarayanan' <asuryana@redhat.com>
Cc: netvirt-dev@lists.opendaylight.org
Subject: RE: [netvirt-dev] L3 PIpeline flow clarification

Hi Vivek,
Thanks for the update.
This use case worked when we use ODL for L2 service and
neutron-l3-agent for L3 service.
Now we trying to use ODL for L2/L3 service found multiple router use case not working.
Can we mark the bug[1] as enhancement bug or do we need to create separate bug for this issue.
[1] https://bugs.opendaylight.org/show_bug.cgi?id=8802

Regards,
Bala

From: N Vivekanandan n.vivekanandan@ericsson.com
Sent: Tuesday, August 22, 2017 3:19 PM
To: Balakrishnan Karuppasamy; Chetan Arakere Gowdru; 'Aswin Suryanarayanan'
Cc: netvirt-dev@lists.opendaylight.org
Subject: RE: [netvirt-dev] L3 PIpeline flow clarification

Hi Bala,

I don’t think we claim support in ODL for the use-case where the same subnet is made part of multiple routers.
And we see both UC1 and UC2 rely on the above expectation.

Was this use-case working earlier ?

We would request you to file an enhancement bug if this use-case needs to be
supported in the upcoming release of ODL.

–
Thanks,

Vivek

From: Balakrishnan Karuppasamy balakrishnan.ka@hcl.com
Sent: Tuesday, August 22, 2017 12:27 PM
To: N Vivekanandan <n.vivekanandan@ericsson.com>; Chetan Arakere Gowdru <chetan.arakere@altencalsoftlabs.com>; 'Aswin Suryanarayanan' <asuryana@redhat.com>
Cc: netvirt-dev@lists.opendaylight.org
Subject: RE: [netvirt-dev] L3 PIpeline flow clarification

HI Vivek,
Please find the topology diagram for UC1 in openstack

Please find the response inline.

Regards,
Bala

From: N Vivekanandan n.vivekanandan@ericsson.com
Sent: Monday, August 21, 2017 8:49 PM
To: Balakrishnan Karuppasamy; Chetan Arakere Gowdru; 'Aswin Suryanarayanan'
Cc: netvirt-dev@lists.opendaylight.org
Subject: RE: [netvirt-dev] L3 PIpeline flow clarification

Hi Bala,

Can you please explain what you indicate by int-router and ext-router in the diagrams?
Refer the openstack topology diagram

There has to be only one router (whatever you name it – may be int-router here) and
that routers router-gateway will be set to external-network. Is that what you want to indicate
in UC1?

UC1 has two routers.
Int-router connecting two internal network, int-router doesn’t have router gateway set.
Two interface attached to the router (50.0.0.3,60.0.0.1)
ext-router connecting internal-Nw1 and external network.
This router has router gateway set to external network.
Expectation :
Traffic from VM1 to external should be route to interface 50.0.0.1
Traffic from VM1 to internal Nw2 should be route to interface 50.0.0.3

For UC2, how is the VM part of multiple routers?

We can connect internal Nw1 to multiple router first router will take default gateway IP
Which is mentioned during network creation.
While attaching the same network to another router we need to specify the fixed IP
To the router interface.

–
Thanks,

Vivek

From: netvirt-dev-bounces@lists.opendaylight.org netvirt-dev-bounces@lists.opendaylight.org On Behalf Of Balakrishnan Karuppasamy
Sent: Monday, August 21, 2017 7:02 PM
To: Chetan Arakere Gowdru <chetan.arakere@altencalsoftlabs.com>; 'Aswin Suryanarayanan' <asuryana@redhat.com>
Cc: netvirt-dev@lists.opendaylight.org
Subject: Re: [netvirt-dev] L3 PIpeline flow clarification

Hi Chetan,
We have tested UC1 multiple time found it is not working.
We have already created bug for the issue.
https://bugs.opendaylight.org/show_bug.cgi?id=8802

Update our initial observation in the bug.
Still analyzing the other issues for UC1.
Please let us know in case of any queries.

Regrads,
Bala

From: Chetan A G chetan.arakere@altencalsoftlabs.com
Sent: Friday, August 18, 2017 11:19 AM
To: Balakrishnan Karuppasamy; 'Aswin Suryanarayanan'
Cc: netvirt-dev@lists.opendaylight.org
Subject: RE: [netvirt-dev] L3 PIpeline flow clarification

Hi Bala,

UC1 is currently supported(Please raise a Bug and share logs to debug this scenario if it not working)
UC2 is currently not supported.

Thanks,
Chetan

From: Balakrishnan Karuppasamy balakrishnan.ka@hcl.com
Sent: 17 August 2017 13:13
To: Chetan A G <chetan.arakere@altencalsoftlabs.com>; 'Aswin Suryanarayanan' <asuryana@redhat.com>
Cc: netvirt-dev@lists.opendaylight.org
Subject: RE: [netvirt-dev] L3 PIpeline flow clarification

Hi Chetan,

Thanks for the clear explanation.
could you please confirm the below use case is supported in new nwtvirt?

Use case1:
Connecting internal NW1 to external NW as well as internal NW2

In this use case VM1 to external host communication working until we attach internal-NW2 interface to router “int-router”

Use case 2:

Connecting internal NW to two external network.

Both cases we have added route entry in created VM instance to forward the packets to respective router interface.
For two external network use case we using physnet1 , physnet2 as provider mapping.

Regards,
Bala

From: Chetan A G chetan.arakere@altencalsoftlabs.com
Sent: Thursday, August 17, 2017 9:00 AM
To: Balakrishnan Karuppasamy; 'Aswin Suryanarayanan'
Cc: netvirt-dev@lists.opendaylight.org
Subject: RE: [netvirt-dev] L3 PIpeline flow clarification

Hi Bala,

Please find the below SNAT and DNAT pipeline flows

SNAT to external-network (Controller-based SNAT)

Ref : https://jenkins.opendaylight.org/releng/view/CSIT-1node/job/netvirt-csit-1node-openstack-newton-nodl-v2-upstream-stateful-carbon/941/robot/report/log.html#s1-s1-s3-t17-k3-k1-k2-k1-k15-k4

1. cookie=0x8000004, duration=96.292s, table=21, n_packets=6, n_bytes=588, priority=10,ip,metadata=0x30d50/0xfffffe actions=goto_table:26
cookie=0x8000006, duration=96.195s, table=26, n_packets=0, n_bytes=0, priority=5,ip,metadata=0x30d50/0xfffffe actions=goto_table:46
(if traffic initiated from VM in NAPT switch)
or
cookie=0x8000005, duration=96.195s, table=36, n_packets=8, n_bytes=557, priority=10,ip,tun_id=0x11170 actions=write_metadata:0x30d50/0xfffffe,goto_table:46
(if traffic initiated from non-NAPT switch)

2. cookie=0x81286b1, duration=96.195s, table=46, n_packets=1, n_bytes=74, priority=5,ip,metadata=0x30d50/0xfffffe actions=CONTROLLER:65535,write_metadata:0x30d50/0xfffffe
(packet punted to controller and dynamic programming happens(46->47 & 44->47))

3) cookie=0x81296a8, duration=6.139s, table=46, n_packets=7, n_bytes=483, idle_timeout=300, send_flow_rem priority=10,tcp,metadata=0x30d50/0xfffffe,nw_src=90.0.0.7,tp_src=34470 actions=set_field:10.10.10.4->ip_src,set_field:49152->tcp_src,set_field:fa:16:3e:f6:03:3a->eth_src,write_metadata:0x30d44/0xffffff,goto_table:47

4) cookie=0x8000006, duration=96.191s, table=47, n_packets=7, n_bytes=483, priority=5,ip,metadata=0x30d44/0xfffffe actions=load:0->NXM_OF_IN_PORT[],resubmit(,21)

5) cookie=0x8000003, duration=91.252s, table=21, n_packets=7, n_bytes=483, priority=42,ip,metadata=0x30d44/0xfffffe,nw_dst=10.10.10.250 actions=set_field:f6:00:00:ff:01:01->eth_dst,load:0xa00->NXM_NX_REG6[],resubmit(,220)

external-network to SNAT(Controller-based SNAT)

1) cookie=0x8000003, duration=96.145s, table=21, n_packets=6, n_bytes=421, priority=42,ip,metadata=0x30d44/0xfffffe,nw_dst=10.10.10.4 actions=write_metadata:0x30d44/0xfffffe,goto_table:44

2) cookie=0x81296a8, duration=6.639s, table=44, n_packets=6, n_bytes=421, send_flow_rem priority=10,tcp,nw_dst=10.10.10.4,tp_dst=49152 actions=set_field:90.0.0.7->ip_dst,set_field:34470->tcp_dst,write_metadata:0x30d50/0xfffffe,goto_table:47

3) cookie=0x8000006, duration=96.191s, table=47, n_packets=6, n_bytes=421, priority=5,ip,metadata=0x30d50/0xfffffe actions=load:0->NXM_OF_IN_PORT[],resubmit(,21)

DNAT to external-network

Ref : https://jenkins.opendaylight.org/releng/view/CSIT-1node/job/netvirt-csit-1node-openstack-newton-nodl-v2-upstream-stateful-carbon/941/robot/report/log.html#s1-s1-s3-t13-k3-k1-k3-k1-k15-k4

1) cookie=0x8000004, duration=50.931s, table=21, n_packets=3, n_bytes=294, priority=10,ip,metadata=0x30d50/0xfffffe actions=goto_table:26

2) cookie=0x8000004, duration=33.806s, table=26, n_packets=3, n_bytes=294, priority=10,ip,metadata=0x30d50/0xfffffe,nw_src=90.0.0.12 actions=set_field:10.10.10.3->ip_src,write_metadata:0x30d44/0xfffffe,goto_table:28

3) cookie=0x8000004, duration=33.806s, table=28, n_packets=3, n_bytes=294, priority=10,ip,metadata=0x30d44/0xfffffe,nw_src=10.10.10.3 actions=set_field:fa:16:3e:42:09:fd->eth_src,resubmit(,21)

4) cookie=0x8000004, duration=74.467s, table=21, n_packets=3, n_bytes=294, priority=10,ip,metadata=0x30d44/0xfffffe actions=group:225002

external-network to DNAT

1) cookie=0x8000003, duration=33.784s, table=21, n_packets=3, n_bytes=294, priority=42,ip,metadata=0x30d44/0xfffffe,nw_dst=10.10.10.3 actions=set_field:fa:16:3e:42:09:fd->eth_dst,goto_table:25

2) cookie=0x8000004, duration=33.835s, table=25, n_packets=3, n_bytes=294, priority=10,ip,dl_dst=fa:16:3e:42:09:fd,nw_dst=10.10.10.3 actions=set_field:90.0.0.12->ip_dst,write_metadata:0x30d50/0xfffffe,goto_table:27

3) cookie=0x8000004, duration=33.831s, table=27, n_packets=3, n_bytes=294, priority=10,ip,metadata=0x30d50/0xfffffe,nw_dst=90.0.0.12 actions=resubmit(,21)

Thanks,
Chetan

From: netvirt-dev-bounces@lists.opendaylight.org netvirt-dev-bounces@lists.opendaylight.org On Behalf Of Balakrishnan Karuppasamy
Sent: 16 August 2017 17:30
To: Aswin Suryanarayanan <asuryana@redhat.com>
Cc: netvirt-dev@lists.opendaylight.org
Subject: Re: [netvirt-dev] L3 PIpeline flow clarification

HI Aswin,

Please find the response inline.

Regards,
Bala

From: Aswin Suryanarayanan asuryana@redhat.com
Sent: Wednesday, August 16, 2017 5:05 PM
To: Balakrishnan Karuppasamy
Cc: netvirt-dev@lists.opendaylight.org
Subject: Re: L3 PIpeline flow clarification

Hi Bala,
What exactly you are looking for? External n/w connectivity?

Yes , I am looking for external network connectivity pipeline flow for SNAT and DNAT use case.

Packet to external n/w from vm without floating ip is send to controller from table 46 the flows will be programmed in table 44(Inbound NAT) and 46(Outbound NAT).

You can have look at [1] for the flows programmed when packet is send to the controller from table 22.

[1]http://docs.opendaylight.org/en/stable-carbon/submodules/netvirt/docs/specs/discovery_of_directly_connected_pnfs.html
Thanks I will go through the link .
Thanks
Aswin

On Wed, Aug 16, 2017 at 3:43 PM, Balakrishnan Karuppasamy <balakrishnan.ka@hcl.com> wrote:
Hi Aswin,
I am analyzing L3pipeline flow in new netvirt.
I need some clarification ,once the packet sent to controller port is any dynamic flow added in pipeline?
While checking the packet flow I could not trace the packets when it was sent to controller.

1. cookie=0x8000004, duration=4483.783s, table=22, n_packets=0, n_bytes=0, priority=0 actions=CONTROLLER:65535
2. cookie=0x8129e19, duration=115.456s, table=46, n_packets=0, n_bytes=0, priority=5,ip,metadata=0x33c20/0xfffffe actions=CONTROLLER:65535,write_metadata:0x33c20/0xfffffe
3. cookie=0x1080000, duration=4483.776s, table=19, n_packets=0, n_bytes=0, priority=100,arp,arp_op=2 actions=CONTROLLER:65535,resubmit(,17)
The above three flows writes any dynamic flows in pipeline?
Could you please tell me in which table the dynamic flows will be installed?
Is there any documentation available which explains L3 pipeline flow of ODL

Regards,
Bala

Comment by YOGA LAKSHMI SWETHA PAYYAVULA [ 15/Sep/17 ]

Attachment external_UC1.PNG has been added with description: UC1 topology

Comment by YOGA LAKSHMI SWETHA PAYYAVULA [ 15/Sep/17 ]

UC1 scenario works fine but the UC2 scenario seems to have issues. With this regard we had a clarification with the netvirt-dev community. PFB the link for the mail thread, where they have confirmed the issue and have requested to mark the bug as an enhancement.

https://lists.opendaylight.org/pipermail/netvirt-dev/2017-August/005328.html

Comment by YOGA LAKSHMI SWETHA PAYYAVULA [ 15/Sep/17 ]

Attachment external_UC2.PNG has been added with description: UC2 topology

Comment by YOGA LAKSHMI SWETHA PAYYAVULA [ 18/Sep/17 ]

UC1 and UC2 scenario both seems to have issues. With this regard we had a clarification with the netvirt-dev community. PFB the link for the mail thread, where they have confirmed the issue and have requested to mark the bug as an enhancement.

https://lists.opendaylight.org/pipermail/netvirt-dev/2017-August/005328.html

Comment by Sam Hague [ 06/Apr/18 ]

VinhNguyen is this still an issue?

Comment by Vinh Nguyen [ 06/Apr/18 ]

Thanks Sam, we still aim to fix in flourine.

Comment by Vinh Nguyen [ 09/May/18 ]

Won't fix since the use cases are not needed

Comment by Vinh Nguyen [ 10/May/18 ]

The use cases in this issue are not needed. Moreover, we should exploit other existing openstack/opendaylight functionalities (static route, etc..) for solution.

Generated at Wed Feb 07 20:22:24 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.