Operating System: All
Platform: All

Request to collect below required log/dump outputs for further analyze the issue.

(1) karaf logs
The following logs needs to be set as TRACE before configuring NAT topology.

NAT TRACE Enable:
--------------------
log:set TRACE org.opendaylight.netvirt.neutronvpn
log:set TRACE org.opendaylight.netvirt.natservice.internal
log:set TRACE org.opendaylight.netvirt.fibmanager

(2)

Please capture the following outputs after initiating traffic to reach FIP.

OVS Dump flows and Groups:
-----------------------------
sudo ovs-ofctl dump-flows -O Openflow13 br-int
sudo ovs-ofctl dump-groups -O Openflow13 br-int
sudo ovs-ofctl dump-group-stats -O Openflow13 br-int
sudo ovs-vsctl list Open_vSwitch

(3) Please collect the following REST API call output

http://<ODL-IP>:8181/restconf/config/odl-nat:external-networks/
http://<ODL-IP>:8181/restconf/config/odl-nat:ext-routers/
http://<ODL-IP>:8181/restconf/config/odl-nat:floating-ip-port-info
http://<ODL-IP>:8181/restconf/config/odl-nat:floating-ip-info/
http://<ODL-IP>:8181/restconf/operational/odl-nat:floating-ip-info/
http://<ODL-IP>:8181/restconf/config/odl-fib:fibEntries/
http://<ODL-IP>:8181/restconf/operational/odl-l3vpn:vpn-instance-op-data/
http://<ODL-IP>:8181/restconf/operational/odl-l3vpn:learnt-vpn-vip-to-port-data/

here are the flows

https://github.com/manjeetbhatia/ODL_DEBUG

I also noticed the br-ex is not connected, when i dumped flow.

https://github.com/manjeetbhatia/ODL_DEBUG/blob/master/ovsvsctlshow

here is conf I used https://github.com/openstack/networking-odl/blob/master/devstack/local.conf.example

From the latest logs, I see the issue is with table=25 flow missing from DNAT which is resulting is this failure.

Ex:
cookie=0x8000004, duration=46002.999s, table=25, n_packets=82, n_bytes=8036, priority=10,ip,dl_dst=fa:16:3e:ba:63:3e,nw_dst=172.24.5.5 actions=set_field:10.0.0.11->ip_dst,write_metadata:0x30d46/0xfffffe,goto_table:27(Missing)

cookie=0x8000004, duration=71830.221s, table=26, n_packets=5, n_bytes=490, priority=10,ip,metadata=0x30d40/0xfffffe,nw_src=10.0.0.11 actions=set_field:172.24.5.5->ip_src,write_metadata:0x30d42/0xfffffe,goto_table:28
cookie=0x8000004, duration=71830.218s, table=28, n_packets=5, n_bytes=490, priority=10,ip,metadata=0x30d42/0xfffffe,nw_src=172.24.5.5 actions=resubmit(,21)

The below review has addressed this issue.

https://git.opendaylight.org/gerrit/#/c/58176/

Request to re-test the same with this fix and let us know if there are any issue.

Regarding br-ex, for external-network connectivity, we create an patch-port between br-int and br-ex and the public-interface(br-physnet1) will be added to br-ex.

https://jenkins.opendaylight.org/releng/job/netvirt-csit-1node-openstack-newton-nodl-v2-gate-stateful-nitrogen/280/robot/report/log.html#s1-s1-s3-t13-k3-k1-k2-k1-k12-k4

other_config :

neutron -v net-create external-net --router:external --provider:network_type=flat --provider:physical_network=physnet1
neutron -v subnet-create
external-net 10.10.10.0/24 --name external-subnet --gateway 10.10.10.250 --alloc
ation-pool start=10.10.10.2,end=10.10.10.249

Instead of br-ex(br-physnet1) and the public interface will be added this bridge.

Bridge "br-physnet1"
Port "br-physnet1-int-patch"
Interface "br-physnet1-int-patch"
type: patch
options:

Port "br-physnet1"
Interface "br-physnet1"
type: internal
Port control_vxlan
Interface control_vxlan
type: vxlan
options:

We assigned the public interface ip to this bridge there-after for testing.

Hi chetan,

I span devstack (stable/ocata) with carbon-snapshot-0.6 + new netvirt
I still see the issue. Here is my config https://github.com/manjeetbhatia/useful_files/blob/master/local_odl_new_virt.conf

I am using pubic:veth-pub-int as ODL_PROVIDER and I can see that patched to br-int

c839bb03-7475-4804-9dd4-436e058e2ab9
Manager "ptcp:6641:127.0.0.1"
is_connected: true
Manager "tcp:192.168.237.10:6640"
is_connected: true
Bridge br-int
Controller "tcp:192.168.237.10:6653"
is_connected: true
fail_mode: secure
Port br-int
Interface br-int
type: internal
Port veth-pub-int
Interface veth-pub-int
Port "tap6ae87e47-64"
Interface "tap6ae87e47-64"
type: internal
Port "tap70eb2776-ec"
Interface "tap70eb2776-ec"
ovs_version: "2.6.1

I tcpdumped on veth-pub-ex (public interface ) patched via (veth-pub-int)
I see arp reply with oui unknown.

will appreciate your help here.

I also tried new snapshot-0.7.0 with current master, seems like a regression there, with that dhcp lease is not happening, from logs I see dnsmasq being running.

Attachment Steps for connectivity FloatingIP.pdf has been added with description: Steps fo Floating External connectivity

Attachment local.conf has been added with description: allinone devstack(ocata) local.conf

Hi Manjeet,

I'm not much aware of configuration on openstack side, but I have tried to capture step-by-step configuration(please find attached) we did for testing DNAT Floating IP connectivity in our local set-up. Please go through it and lets us see if anything is missing.

Thanks,
Chetan

Hi chetan, I see in your conf mechanism_driver commented out, is default driver opendaylight ? i guess we need to use opendaylight as mechanism driver

Manjeet,

Please note the ODL section in chetan's local.conf.. it contains:
#ODL Settings
ODL_MODE=externalodl
ODL_MGR_IP=192.168.56.1
#ODL_MGR_IP=127.0.0.1
ODL_PORT=8080
ODL_L3=True
ODL_V2DRIVER=True
ODL_PROVIDER_MAPPINGS=${ODL_PROVIDER_MAPPINGS:-external:br-ex}

In the above , the ODL_V2DRIVER will make opendaylight_v2 available in mechanism_drivers of /etc/neutron/plugins/ml2/ml2_conf.ini after stacking.

Vivek

Hi chetan !! thanks for info and local.conf,

I tried that out, I've had replaced the HOST_IP and ODL_MGR_IP,
and stacked, it fails on br-int not become available in 600 seconds,

from ovs logs i figured out it is not able to connect

2017-06-09T07:01:39.303Z|00105|reconnect|INFO|tcp:192.168.237.11:6640: connection attempt failed (Connection refused)
2017-06-09T07:01:39.303Z|00106|reconnect|INFO|tcp:192.168.237.11:6640: waiting 8 seconds before reconnect

I see this issue on both centos 7.1 and ubuntu xenial. anything I am missing ?

Hi Manjeet,

We do create bridge br-int and br-ex manually before starting devstack.

sudo ovs-vsctl add-br br-int
sudo ovs-vsctl add-br br-ex

ok, I relied on stack script, because it take care of that part,

but what about this log trace from ovsdb-server.log

2017-06-09T07:01:39.303Z|00105|reconnect|INFO|tcp:192.168.237.11:6640: connection attempt failed (Connection refused)
2017-06-09T07:01:39.303Z|00106|reconnect|INFO|tcp:192.168.237.11:6640: waiting 8 seconds before reconnect

This error is due to ODL not up and running. Once karaf is started and feature odl-netvirt-openstack is loaded, it will be connected(Please find snapshot attached for your reference)

Attachment karaf-start.jpg has been added with description: Karaf start logs

Attachment karaf-start.gif has been added with description: karaf start logs

make sense !! thanks Im trying that out.

$ netstat -an | grep 6640
TCP 0.0.0.0:6640 0.0.0.0:0 LISTENING
TCP 192.168.56.1:6640 192.168.56.101:38554 ESTABLISHED
TCP [::]:6640 [::]:0 LISTENING

chetan.arakere@clpstpdfc592 MINGW64 /d/Open-Day-Light/ODL-Boron/netvirt/vpnservice (boron-branch)
$ netstat -an | grep 6653
TCP 0.0.0.0:6653 0.0.0.0:0 LISTENING
TCP 192.168.56.1:6653 192.168.56.101:60228 ESTABLISHED
TCP [::]:6653 [::]:0 LISTENING

chetan.arakere@clpstpdfc592 MINGW64 /d/Open-Day-Light/ODL-Boron/netvirt/vpnservice (boron-branch)
$ netstat -an | grep 6633
TCP 0.0.0.0:6633 0.0.0.0:0 LISTENING
TCP [::]:6633 [::]:0 LISTENING

Hi chetan, stack ended in success, but I don't see opendaylight got configured, there is no snapshot inside opendaylight dir, that has to be done manually as well ? like getting snapshot and doing feature install ?

I manually configured opendaylight,

I've created ovs bridges and public network as instructed on doc, it shows status active and ip=10.0.0.5 , but I don't see in dhcp lease log that ever happened, I am unable to ping from within network namespace,

here is my ovs-vstcl show now

Manager "tcp:192.168.237.11:6640"
is_connected: true
Manager "ptcp:6641:127.0.0.1"
is_connected: true
Bridge br-ex
Port br-ex-int-patch
Interface br-ex-int-patch
type: patch
options:

Port br-ex
Interface br-ex
type: internal
Bridge br-int
Controller "tcp:192.168.237.11:6653"
is_connected: true
fail_mode: secure
Port "tapbf0d84fc-c7"
Interface "tapbf0d84fc-c7"
type: internal
Port br-int
Interface br-int
type: internal
Port "tap9f238c61-5c"
Interface "tap9f238c61-5c"
Port br-ex-patch
Interface br-ex-patch
type: patch
options:

ovs_version: "2.6.1"

Hi chetan,

I used this conf (https://github.com/manjeetbhatia/useful_files/blob/master/local_conf_carbon) with carbon-snapshot-0.6 , I see floating ip is accessible, I can ping and ssh to fip. if you get chance, can you please walk over conf, to confirm everything is correct, I used public in ODL_PROVIDER_MAPPINGS because in gate jobs it will be using external network named public.

thanks for your help !!