Openstack rocky + Opendaylight SR1

By the way, floating IP can work with simulated external network.

Hi Yang,

Can you provide the following details.

1) is Controller-based or contrack Based SNAT been configured ??

2) Please provide following DS dumps from the set-up where is issue been observed and karaf logs

curl  --silent  -u  -X GET http://localhost:8181/restconf/config/odl-nat:external-networks/ |python -m json.tool

 curl  --silent  -u  -X GET http://localhost:8181/restconf/config/odl-nat:ext-routers/ |python -m json.tool

curl  --silent  -u  -X GET [http://localhost:8181/restconf/config/odl-nat:napt-switches/|python -m json.tool|http://localhost:8181/restconf/config/odl-nat:napt-switches/]

curl  --silent  -u  -X GET http://localhost:8181/restconf/config/odl-nat:intext-ip-port-map/ |python -m json.tool

curl  --silent  -u  -X GET http://localhost:8181/restconf/config/odl-nat:snatint-ip-port-map/ |python -m json.tool

 curl  --silent  -u  -X GET http://localhost:8181/restconf/config/odl-nat:floating-ip-port-info/ |python -m json.tool

curl  --silent  -u  -X GET http://localhost:8181/restconf/config/odl-nat:floating-ip-info/ |python -m json.tool

curl  --silent  -u  -X GET http://localhost:8181/restconf/config/odl-nat:router-id-name/ |python -m json.tool

curl  --silent  -u  -X GET http://localhost:8181/restconf/operational/odl-nat:intext-ip-map/ |python -m json.tool

curl  --silent  -u  -X GET http://localhost:8181/restconf/operational/odl-nat:external-ips-counter/ |python -m json.tool

curl  --silent  -u  -X GET http://localhost:8181/restconf/operational/odl-nat:floating-ip-info/ |python -m json.tool

I saw the blow error info, maybe it is helpful to you.

2019-02-28T06:06:47,391 | ERROR | org.opendaylight.yang.gen.v1.urn.opendaylight.netvirt.natservice.rev160111.napt.switches.RouterToNaptSwitch_AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0 | SnatCentralizedSwitchChangeListener | 345 - org.opendaylight.netvirt.natservice-impl - 0.7.1 | Future (eventually) failed: Error handling SNAT centralized switch update
java.lang.NullPointerException: null
at org.opendaylight.netvirt.natservice.ha.SnatCentralizedSwitchChangeListener.lambda$update$1(SnatCentralizedSwitchChangeListener.java:105) ~[?:?]
at org.opendaylight.genius.infra.ManagedNewTransactionRunnerImpl.callWithNewReadWriteTransactionAndSubmit(ManagedNewTransactionRunnerImpl.java:106) [256:org.opendaylight.genius.mdsalutil-api:0.5.1]
at org.opendaylight.netvirt.natservice.ha.SnatCentralizedSwitchChangeListener.update(SnatCentralizedSwitchChangeListener.java:98) [345:org.opendaylight.netvirt.natservice-impl:0.7.1]
at org.opendaylight.netvirt.natservice.ha.SnatCentralizedSwitchChangeListener.update(SnatCentralizedSwitchChangeListener.java:43) [345:org.opendaylight.netvirt.natservice-impl:0.7.1]
at org.opendaylight.genius.datastoreutils.AsyncDataTreeChangeListenerBase$DataTreeChangeHandler.run(AsyncDataTreeChangeListenerBase.java:176) [256:org.opendaylight.genius.mdsalutil-api:0.5.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:?]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:?]
at java.lang.Thread.run(Thread.java:748) [?:?]

It is controller-based NAT.

vagrant@odl:~/opendaylight-0.9.1$ cat etc/opendaylight/datastore/initial/config/netvirt-natservice-config.xml
<natservice-config xmlns="urn:opendaylight:netvirt:natservice:config">
<nat-mode>controller</nat-mode>
<snat-punt-timeout>5</snat-punt-timeout>
</natservice-config>
vagrant@odl:~/opendaylight-0.9.1$

stack@control:~/devstack$ ./vm_ssh vm1
qdhcp-0b9de268-2dd3-493a-91e2-f8fc71035313
10.15.1.3
Warning: Permanently added '10.15.1.3' (RSA) to the list of known hosts.
cirros@10.15.1.3's password:
$ ping 192.168.100.1
PING 192.168.100.1 (192.168.100.1): 56 data bytes
^C
— 192.168.100.1 ping statistics —
14 packets transmitted, 0 packets received, 100% packet loss
$

192.168.100.1 is gateway of my external network, it is simulated as ODL csit setup, floating IP can work.

stack@control:~/devstack$ curl --silent -u admin:admin -X GET http://192.168.0.5:8181/restconf/config/odl-nat:external-networks/ |python -m json.tool
{
"external-networks": {
"networks": [

]
}
}

stack@control:~/devstack$ curl --silent -u admin:admin -X GET http://192.168.0.5:8181/restconf/config/odl-nat:ext-routers/ |python -m json.tool
{
"ext-routers": {
"routers": [
{
"enable-snat": true,
"ext_gw_mac_address": "fa:16:3e:73:81:72",
"external-ips": [

],
"network-id": "031f4894-2437-42d4-b380-c575fc854b03",
"router-name": "fdbae3de-7bc7-4a7d-acf5-911bec787487",
"subnet-ids": [
"2e7b7e8c-8c82-456b-a8e5-d71d23121260",
"b7d4023a-bd4e-460e-9afe-30e16b79d842"
]
}
]
}
}
stack@control:~/devstack$ curl --silent -u admin:admin -X GET http://192.168.0.5:8181/restconf/config/odl-nat:napt-switches/ |python -m json.tool
{
"napt-switches": {
"router-to-napt-switch": [

,

,

,

,

]
}
}
stack@control:~/devstack$ curl --silent -u admin:admin -X GET http://192.168.0.5:8181/restconf/config/odl-nat:intext-ip-port-map/ |python -m json.tool
{
"intext-ip-port-map": {
"ip-port-mapping": [
{
"intext-ip-protocol-type": [

],
"router-id": 103063
}
]
}
}

stack@control:~/devstack$ curl --silent -u admin:admin -X GET http://192.168.0.5:8181/restconf/config/odl-nat:snatint-ip-port-map/ |python -m json.tool
{
"snatint-ip-port-map": {
"intip-port-map": [
{
"ip-port": [
{
"int-ip-proto-type": [

],
"internal-ip": "10.15.1.2"
}
],
"router-id": 103063
}
]
}
}

stack@control:~/devstack$ curl --silent -u admin:admin -X GET http://192.168.0.5:8181/restconf/config/odl-nat:floating-ip-port-info/ |python -m json.tool
{
"floating-ip-port-info": {
"floating-ip-id-to-port-mapping": [

]
}
}

stack@control:~/devstack$ curl --silent -u admin:admin -X GET http://192.168.0.5:8181/restconf/config/odl-nat:floating-ip-info/ |python -m json.tool
{
"floating-ip-info": {}
}

stack@control:~/devstack$ curl --silent -u admin:admin -X GET http://192.168.0.5:8181/restconf/config/odl-nat:router-id-name/ |python -m json.tool
{
"router-id-name": {
"routerIds": [

,

,

,

,

,

]
}
}

stack@control:~/devstack$ curl --silent -u admin:admin -X GET http://192.168.0.5:8181/restconf/operational/odl-nat:intext-ip-map/ |python -m json.tool
{
"intext-ip-map": {
"ip-mapping": [
{
"ip-map": [

,

],
"segment-id": 103063
}
]
}
}

stack@control:~/devstack$ curl --silent -u admin:admin -X GET http://192.168.0.5:8181/restconf/operational/odl-nat:external-ips-counter/ |python -m json.tool
{
"external-ips-counter": {
"external-counters": [
{
"external-ip-counter": [

],
"segment-id": 103063
},
{
"external-ip-counter": [

],
"segment-id": 103010
},
{
"external-ip-counter": [

],
"segment-id": 103052
},
{
"external-ip-counter": [

],
"segment-id": 103037
},
{
"external-ip-counter": [

],
"segment-id": 103022
}
]
}
}

stack@control:~/devstack$ curl --silent -u admin:admin -X GET http://192.168.0.5:8181/restconf/operational/odl-nat:floating-ip-info/ |python -m json.tool
{
"floating-ip-info": {
"router-ports": [
{
"external-network-id": "1b7f57dc-12cb-413a-b444-05253e46e23f",
"ports": [

],
"router-id": "7027fad7-ee0d-4c5f-9001-f7d3645a1d52"
},
{
"external-network-id": "945b9f26-178e-4456-af02-c7cb5a2b316e",
"ports": [

],
"router-id": "7f632150-be8c-4e30-ae7d-fa5753fbf8fe"
}
]
}
}

Contoller Based SNAT doesn't support ICMP currently. Only TCP and UDP is supported. Since I see ping to external-gw, ICMP packets will be dropped by the following flow.

Ex: 

TCP:cookie=0x81286c5, duration=914.453s, table=46, n_packets=1, n_bytes=74, priority=5,tcp,metadata=0x30d78/0xfffffe actions=CONTROLLER:65535,learn(table=46,hard_timeout=5,priority=7,cookie=0x81286c5,eth_type=0x800,nw_proto=6,NXM_OF_IP_SRC[],NXM_OF_IP_DST[],NXM_OF_TCP_SRC[],NXM_OF_TCP_DST[],OXM_OF_METADATA[1..23])

ICMP:

cookie=0x8000008, duration=914.453s, table=46, n_packets=6, n_bytes=588, priority=0,icmp,metadata=0x30d78/0xfffffe actions=drop

UDP:

cookie=0x81286c5, duration=914.453s, table=46, n_packets=1, n_bytes=59, priority=5,udp,metadata=0x30d78/0xfffffe actions=CONTROLLER:65535,learn(table=46,hard_timeout=5,priority=7,cookie=0x81286c5,eth_type=0x800,nw_proto=17,NXM_OF_IP_SRC[],NXM_OF_IP_DST[],NXM_OF_UDP_SRC[],NXM_OF_UDP_DST[],OXM_OF_METADATA[1..23])

You can configure SNAT in conntrack mode for ICMP SNAT translation

Ref: https://docs.opendaylight.org/en/stable-carbon/submodules/netvirt/docs/specs/conntrack-based-snat.html

Got it, http works, thank you so much. I'll check if ping is ok for conntrack-based nat.

$ curl http://192.168.100.1:8888/
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>Directory listing for /</title>
</head>
<body>
<h1>Directory listing for /</h1>
<hr>
<ul>
<li><a href=".bash_history">.bash_history</a></li>
<li><a href=".bash_logout">.bash_logout</a></li>
<li><a href=".bashrc">.bashrc</a></li>
<li><a href=".cache/">.cache/</a></li>
<li><a href=".novaclient/">.novaclient/</a></li>
<li><a href=".profile">.profile</a></li>
<li><a href=".ssh/">.ssh/</a></li>
<li><a href=".viminfo">.viminfo</a></li>
<li><a href="devstack.tar.bz2">devstack.tar.bz2</a></li>
<li><a href="neutron.tar.bz2">neutron.tar.bz2</a></li>
<li><a href="nova.tar.bz2">nova.tar.bz2</a></li>
</ul>
<hr>
</body>
</html>
$

Hi, Chetan

After I changed to conntrack, snat can't work, ssh, http and ping 192.168.100.1 all don't work, so I propose this bug should be reopened.

$ ping 192.168.100.1
PING 192.168.100.1 (192.168.100.1): 56 data bytes
^C
— 192.168.100.1 ping statistics —
5 packets transmitted, 0 packets received, 100% packet loss
$ ssh yangyi@192.168.100.1
^C
$ wget http://192.168.100.1:8888/
Connecting to 192.168.100.1:8888 (192.168.100.1:8888)
^C
$

After I used conntrack, the below rests are wrong:

stack@control:~/devstack$ curl --silent -u admin:admin -X GET http://192.168.0.5:8181/restconf/operational/odl-nat:intext-ip-map/ |python -m json.tool
{
"errors": {
"error": [

]
}
}
stack@control:~/devstack$ curl --silent -u admin:admin -X GET http://192.168.0.5:8181/restconf/operational/odl-nat:external-ips-counter/ |python -m json.tool
{
"errors": {
"error": [

]
}
}
stack@control:~/devstack$ curl --silent -u admin:admin -X GET http://192.168.0.5:8181/restconf/config/odl-nat:intext-ip-port-map/ |python -m json.tool
{
"errors": {
"error": [

]
}
}
stack@control:~/devstack$

I saw the below error karaf log, are they really errors?

2019-03-06T03:42:32,784 | ERROR | DataTreeEventCallbackRegistrar-Timeouter-0 | NeutronPortChangeListener | 346 - org.opendaylight.netvirt.neutronvpn-impl - 0.7.2 | GwPort 79ef4012-5eef-40f3-95d3-95085f2543a6 added without Router
2019-03-06T03:42:34,317 | ERROR | DataTreeEventCallbackRegistrar-Timeouter-0 | NeutronPortChangeListener | 346 - org.opendaylight.netvirt.neutronvpn-impl - 0.7.2 | GwPort 0ce62d79-0ba7-47bd-a569-cdcaf6d75384 added without Router