[NETVIRT-695] Issue in SNAT,DNAT communication - OCATA. Created: 29/May/17  Updated: 17/Jun/17  Resolved: 17/Jun/17

Status: Resolved
Project: netvirt
Component/s: General
Affects Version/s: Carbon
Fix Version/s: None

Type: Bug
Reporter: YOGA LAKSHMI SWETHA PAYYAVULA Assignee: Karthikeyan Krishnan
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: All
Platform: All


Attachments: Zip Archive 8557_Bug logs.zip     JPEG File Example_ARP_Request_Response.jpg     Zip Archive NAT_logs.zip     Zip Archive dump for snat and dnat.zip    
External issue ID: 8557

 Description   

Set-up used :

Allinone node - Ocata devstack

One ODL with carbon version distro:
https://nexus.opendaylight.org/content/repositories/opendaylight.snapshot/org/opendaylight/integration/distribution-karaf/0.6.0-SNAPSHOT/distribution-karaf-0.6.0-20170522.002211-5360.zip

The below is the settings in Local.conf :

SERVICE_HOST=$HOST_IP
MYSQL_HOST=$SERVICE_HOST
RABBIT_HOST=$SERVICE_HOST
GLANCE_HOSTPORT=$SERVICE_HOST:9292
ADMIN_PASSWORD=secret
DATABASE_PASSWORD=secret
RABBIT_PASSWORD=secret
SERVICE_PASSWORD=secret

    1. Neutron options
      Q_USE_SECGROUP=True
      FLOATING_RANGE="11.12.13.0/24"
      IPV4_ADDRS_SAFE_TO_USE="10.0.0.0/22"
      Q_FLOATING_ALLOCATION_POOL=start=11.12.13.250,end=11.12.13.254
      PUBLIC_NETWORK_GATEWAY="11.12.13.1"
      PUBLIC_INTERFACE=eth0
  1. Open vSwitch provider networking configuration
    Q_USE_PROVIDERNET_FOR_PUBLIC=True
    OVS_PHYSICAL_BRIDGE=br-ex
    PUBLIC_BRIDGE=br-ex
    ODL_PROVIDER_MAPPINGS=public:br-ex

LOGFILE=stack.sh.log
SCREEN_LOGDIR=/opt/stack/data/log
LOG_COLOR=True
RECLONE=yes
OFFLINE=False
disable_service swift
disable_service cinder
disable_service n-net
disable_service q-vpn
enable_service q-svc
enable_service q-dhcp
enable_service q-meta
enable_service n-cauth
enable_service tempest
enable_service n-cpu
enable_service placement-api

enable_plugin networking-odl https://github.com/openstack/networking-odl stable/ocata
NEUTRON_CREATE_INITIAL_NETWORKS=False
Q_PLUGIN=ml2
Q_ML2_TENANT_NETWORK_TYPE=vxlan
Q_OVS_USE_VETH=True

ENABLE_TENANT_TUNNELS=True

ODL_PORT=8080
ODL_MODE=externalodl
ODL_PORT_BINDING_CONTROLLER=network-topology

LIBVIRT_TYPE=qemu
ODL_MGR_IP=10.106.138.151

MYSQL_PASSWORD=mysql
RABBIT_PASSWORD=rabbit
SERVICE_TOKEN=service
SERVICE_PASSWORD=admin
ADMIN_PASSWORD=admin

LIBVIRT_TYPE=qemu

ODL_V2DRIVER=True
ODL_OVS_MANAGERS=10.106.138.151
PUBLIC_PHYSICAL_NETWORK=datacenter

PUBLIC_INTERFACE=ens35

[[post-config|/etc/neutron/plugins/ml2/ml2_conf.ini]]
[agent]
minimize_polling=True

[[post-config|/etc/neutron/dhcp_agent.ini]]
[DEFAULT]
force_metadata = True
enable_isolated_metadata = True

disable_service q-l3
[[post-config|$NEUTRON_CONF]]
[DEFAULT]
service_plugins = networking_odl.l3.l3_odl.OpenDaylightL3RouterPlugin
[[post-config|/etc/nova/nova.conf]]

[DEFAULT]
force_config_drive = False

Steps to reproduce the bug :

After stacking the devstack successfully,
Create an external network with network provider type flat and assign the subnet range

external network

neutron -v net-create public --router:external --provider:network_type=flat --provider:physical_network=datacenter

external network subnet:

neutron -v subnet-create public 11.12.13.0/24 --name external-subnet --gateway 11.12.13.250 --allocation-pool start=11.12.13.2,end=11.12.13.249

Then create the internal network and router in GUI.
Router set the gateway for the external network then add an interface for the internal network.

Create the floating ip

Now create a vm for the internal network and associate the floating ip to the vm.

Then from the VM instance ping or ssh the external network instance – SNAT scenario failed

Then from the external instance ping or ssh to the internal network vm – DNAT scenario failed



 Comments   
Comment by YOGA LAKSHMI SWETHA PAYYAVULA [ 29/May/17 ]

I also added the br-ex bridge and added a port to it, while creating the networks the patch port was created as shown below :

[stack@localhost devstack]$ sudo ovs-vsctl show
e07e8ce9-4bf6-489b-8859-424d52981406
Manager "tcp:10.106.138.151:6640"
is_connected: true
Manager "ptcp:6641:127.0.0.1"
is_connected: true
Bridge br-int
Controller "tcp:10.106.138.151:6653"
is_connected: true
Controller "tcp:172.16.3.25:6653"
is_connected: true
fail_mode: secure
Port "tap4168d878-d1"
Interface "tap4168d878-d1"
Port br-int
Interface br-int
type: internal
Port "tap2e801cfe-b0"
Interface "tap2e801cfe-b0"
Port "tap22b9df7b-81"
Interface "tap22b9df7b-81"
Port br-ex-patch
Interface br-ex-patch
type: patch
options:

{peer=br-ex-int-patch}

Bridge br-ex
Port br-ex-int-patch
Interface br-ex-int-patch
type: patch
options:

{peer=br-ex-patch}

Port "ens35"
Interface "ens35"
Port br-ex
Interface br-ex
type: internal
ovs_version: "2.6.1"

Comment by YOGA LAKSHMI SWETHA PAYYAVULA [ 01/Jun/17 ]

Please find the attachment for the dump flows and the odl log details

Comment by YOGA LAKSHMI SWETHA PAYYAVULA [ 01/Jun/17 ]

Attachment 8557_Bug logs.zip has been added with description: log info

Comment by YOGA LAKSHMI SWETHA PAYYAVULA [ 01/Jun/17 ]

attached the dump flows

Comment by YOGA LAKSHMI SWETHA PAYYAVULA [ 01/Jun/17 ]

Attachment dump for snat and dnat.zip has been added with description: adding the dump flows

Comment by Karthikeyan Krishnan [ 01/Jun/17 ]

Hi Lakshmi,

Request to collect below required log/dump outputs for further analyze the issue.

(1)
The following logs needs to be set as TRACE before configuring NAT topology.

NAT TRACE Enable:
--------------------
log:set TRACE org.opendaylight.netvirt.neutronvpn
log:set TRACE org.opendaylight.netvirt.natservice.internal
log:set TRACE org.opendaylight.netvirt.fibmanager

(2)

Please capture the following outputs before and after NAT traffic.

OVS Dump flows and Groups:
-----------------------------
sudo ovs-ofctl dump-flows -O Openflow13 br-int
sudo ovs-ofctl dump-groups -O Openflow13 br-int
sudo ovs-ofctl dump-group-stats -O Openflow13 br-int
sudo ovs-vsctl list Open_vSwitch

(3)

Please collect the following REST API call output

http://localhost:8181/restconf/config/odl-nat:external-networks/
http://localhost:8080/restconf/config/odl-nat:ext-routers/
http://localhost:8181/restconf/config/odl-nat:floating-ip-port-info
http://localhost:8181/restconf/config/odl-nat:floating-ip-info/
http://localhost:8181/restconf/operational/odl-nat:floating-ip-info/
http://localhost:8181/restconf/config/odl-fib:fibEntries/
http://localhost:8181/restconf/operational/odl-l3vpn:vpn-instance-op-data/
http://localhost:8181/restconf/operational/odl-l3vpn:learnt-vpn-vip-to-port-data/

Thanks & Regards,
Karthikeyan.

Comment by YOGA LAKSHMI SWETHA PAYYAVULA [ 01/Jun/17 ]

attached the trace enabled logs and the dump flows

Comment by YOGA LAKSHMI SWETHA PAYYAVULA [ 01/Jun/17 ]

Attachment NAT_logs.zip has been added with description: adding the dump flows

Comment by Karthikeyan Krishnan [ 05/Jun/17 ]

Working on this issue.

Comment by Karthikeyan Krishnan [ 06/Jun/17 ]

Please refer the below code review for having fix for DNAT failure in Stable/Ocata

https://git.opendaylight.org/gerrit/#/c/58176/

Comment by Karthikeyan Krishnan [ 07/Jun/17 ]

Hi Swetha,

DNAT:
DNAT- FIP issue has been fixed in ODL-Master for Stable/Ocata. Please download the latest ODL-Master (Nitrogen) distribution build and try to validate the SNAT/DNAT traffic for “flat” network.

SNAT:
Without any code changes it was working fine for our local development setup. Once your external server connectivity is proper you can able to get the below REST API output for external “flat” network. As of now ODL CSIT with Stable/Ocata has some problem in the setup. Hence we are unable to run the CSIT job for SNAT/DNAT traffic.

Sample Output:
---------------
http://localhost:8181/restconf/operational/odl-l3vpn:learnt-vpn-vip-to-port-data

{
"learnt-vpn-vip-to-port-data": {
"learnt-vpn-vip-to-port": [

{ "vpn-name": "500cfd15-833f-4a47-acac-8c23d3bf8edf", "port-fixedip": "100.100.100.1", "port-name": "64615490027595:br-ex-patch:trunk", "creation-time": "06/06/2017 10:47:10 AM", "mac-address": "0a:00:27:00:00:10" }

]
}
}

Latest ODL-Master Distribution:
----------------------------------
https://nexus.opendaylight.org/content/repositories/opendaylight.snapshot/org/opendaylight/integration/distribution-karaf/0.7.0-SNAPSHOT/distribution-karaf-0.7.0-20170606.040008-788.zip

Comment by Karthikeyan Krishnan [ 15/Jun/17 ]

Hi Swetha,

Still we are seeing problem with external-gateway IP (11.12.13.250) is not learnt by ARP. As a result in L3_FIB_TABLE (21) flow is missing to reach external gateway.

Example flow: (100.100.100.1 IP is external gateway in our tested topology)
cookie=0x8000003, duration=10.170s, table=21, n_packets=0, n_bytes=0, priority=42,ip,metadata=0x30d50/0xfffffe,nw_dst=100.100.100.1 actions=set_feld:0a:00:27:00:00:10->eth_dst,load:0x500->NXM_NX_REG6[],resubmit(,220)

We are not seeing any issue with SNAT flows as it was programming properly as expected (as per the shared logs)

cookie=0x8000004, duration=486.053s, table=21, n_packets=7, n_bytes=518, priority=10,ip,metadata=0x30d52/0xfffffe actions=goto_table:26
cookie=0x8000006, duration=485.881s, table=26, n_packets=7, n_bytes=518, priority=5,ip,metadata=0x30d52/0xfffffe actions=goto_table:46
cookie=0x81296a9, duration=48.438s, table=46, n_packets=6, n_bytes=444, idle_timeout=300, send_flow_rem priority=10,tcp,metadata=0x30d52/0xfffffe,nw_src=10.0.0.10,tp_src=39712 actions=set_field:11.12.13.9->ip_src,set_field:49152->tcp_src,set_field:fa:16:3e:34:b4:ac->eth_src,write_metadata:0x30d4e/0xffffff,goto_table:47
cookie=0x8000006, duration=485.968s, table=47, n_packets=6, n_bytes=444, priority=5,ip,metadata=0x30d4e/0xfffffe actions=load:0->NXM_OF_IN_PORT[],resubmit(,21)
(Since L3_FIB_TABLE(21) for external gateway flow is missing packets are getting dropped here)

To debug further, please do packet capture on interface “ens35” while performing external router-gw set (neutron router-gateway-set <ROUTER_NAME> <EXT_NET_NAME> --enable-snat) you should able to see ARP broadcast request and response from the external gateway. Please refer the below example snapshot for your reference.

Example Packet-Captured (100.100.100.106 is the external fixed IP used for SNAT and 100.100.100.1 is external gateway IP for tested our topology)

Also request to set the below few more modules log level as “TRACE” before doing SNAT configuration.

log:set TRACE org.opendaylight.netvirt.natservice.internal
log:set TRACE org.opendaylight.netvirt.neutronvpn
log:set TRACE org.opendaylight.netvirt.fibmanager
log:set TRACE org.opendaylight.netvirt.vpnmanager
log:set TRACE org.opendaylight.netvirt.elan

Kindly share the log information as you shared earlier after re-testing SNAT.

Comment by Karthikeyan Krishnan [ 15/Jun/17 ]

Attachment Example_ARP_Request_Response.jpg has been added with description: Example Packet-Captured for ARP Request and Response

Comment by Karthikeyan Krishnan [ 16/Jun/17 ]

Hi Swetha,

We do have CSIT job for SNAT/DNAT UC's running with openstack-Ocata in Carbon distribution.we are not seeing any issue in SNAT/DNAT traffic (TCP/UDP) is getting failed. Please refer the below link for your reference.

https://jenkins.opendaylight.org/releng/job/netvirt-csit-1node-openstack-ocata-upstream-learn-carbon/30/robot/report/log.html#s1-s1-s3

Thanks,
Karthikeyan.

Comment by YOGA LAKSHMI SWETHA PAYYAVULA [ 17/Jun/17 ]

We tested again, SNAT and DNAT is working, due to some network issues the external flows were not learnt by ARP.

Generated at Wed Feb 07 20:22:14 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.