OpenDaylight JIRA

OpenDaylight JIRA https://jira.opendaylight.org This file is an XML representation of an issue en-us

8.20.10

820010

22-06-2022

[NETVIRT-1568] snat can't work in Openstack rocky + ODL Fluorine SR1 https://jira.opendaylight.org/browse/NETVIRT-1568 netvirt <p>after enable snat by "openstack router set --external-gateway extnet --enable-snat extvr, VM can't ping external IP no matter I use simulated external network or real external network.</p> <p>Openstack rocky + Opendaylight SR1</p> NETVIRT-1568

snat can't work in Openstack rocky + ODL Fluorine SR1

Bug Highest Resolved Done Chetan Arakere Gowdru Yi Yang netvirt Thu, 28 Feb 2019 14:40:33 +0000 Wed, 6 Mar 2019 06:26:49 +0000 Fri, 1 Mar 2019 07:38:49 +0000 Fluorine-SR1 natservice 0 3 <p>By the way, floating IP can work with simulated external network.</p> <p>Hi Yang,</p> <p>Can you provide the following details.</p> <p>1) is Controller-based or contrack Based SNAT been configured ??</p> <p>2) Please provide following DS dumps from the set-up where is issue been observed and karaf logs</p> <p>curl --silent -u -X GET <a href="http://localhost:8181/restconf/config/odl-nat:external-networks/" class="external-link" target="_blank" rel="nofollow noopener">http://localhost:8181/restconf/config/odl-nat:external-networks/</a> |python -m json.tool</p> <p> curl --silent -u -X GET <a href="http://localhost:8181/restconf/config/odl-nat:ext-routers/" class="external-link" target="_blank" rel="nofollow noopener">http://localhost:8181/restconf/config/odl-nat:ext-routers/</a> |python -m json.tool</p> <p>curl --silent -u -X GET <span class="error">[http://localhost:8181/restconf/config/odl-nat:napt-switches/|python -m json.tool|http://localhost:8181/restconf/config/odl-nat:napt-switches/]</span></p> <p>curl --silent -u -X GET <a href="http://localhost:8181/restconf/config/odl-nat:intext-ip-port-map/" class="external-link" target="_blank" rel="nofollow noopener">http://localhost:8181/restconf/config/odl-nat:intext-ip-port-map/</a> |python -m json.tool</p> <p>curl --silent -u -X GET <a href="http://localhost:8181/restconf/config/odl-nat:snatint-ip-port-map/" class="external-link" target="_blank" rel="nofollow noopener">http://localhost:8181/restconf/config/odl-nat:snatint-ip-port-map/</a> |python -m json.tool</p> <p> curl --silent -u -X GET <a href="http://localhost:8181/restconf/config/odl-nat:floating-ip-port-info/" class="external-link" target="_blank" rel="nofollow noopener">http://localhost:8181/restconf/config/odl-nat:floating-ip-port-info/</a> |python -m json.tool</p> <p>curl --silent -u -X GET <a href="http://localhost:8181/restconf/config/odl-nat:floating-ip-info/" class="external-link" target="_blank" rel="nofollow noopener">http://localhost:8181/restconf/config/odl-nat:floating-ip-info/</a> |python -m json.tool</p> <p>curl --silent -u -X GET <a href="http://localhost:8181/restconf/config/odl-nat:router-id-name/" class="external-link" target="_blank" rel="nofollow noopener">http://localhost:8181/restconf/config/odl-nat:router-id-name/</a> |python -m json.tool</p> <p>curl --silent -u -X GET <a href="http://localhost:8181/restconf/operational/odl-nat:intext-ip-map/" class="external-link" target="_blank" rel="nofollow noopener">http://localhost:8181/restconf/operational/odl-nat:intext-ip-map/</a> |python -m json.tool</p> <p>curl --silent -u -X GET <a href="http://localhost:8181/restconf/operational/odl-nat:external-ips-counter/" class="external-link" target="_blank" rel="nofollow noopener">http://localhost:8181/restconf/operational/odl-nat:external-ips-counter/</a> |python -m json.tool</p> <p>curl --silent -u -X GET <a href="http://localhost:8181/restconf/operational/odl-nat:floating-ip-info/" class="external-link" target="_blank" rel="nofollow noopener">http://localhost:8181/restconf/operational/odl-nat:floating-ip-info/</a> |python -m json.tool</p> <p>I saw the blow error info, maybe it is helpful to you.</p> <p>2019-02-28T06:06:47,391 | ERROR | org.opendaylight.yang.gen.v1.urn.opendaylight.netvirt.natservice.rev160111.napt.switches.RouterToNaptSwitch_AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0 | SnatCentralizedSwitchChangeListener | 345 - org.opendaylight.netvirt.natservice-impl - 0.7.1 | Future (eventually) failed: Error handling SNAT centralized switch update<br/> java.lang.NullPointerException: null<br/> at org.opendaylight.netvirt.natservice.ha.SnatCentralizedSwitchChangeListener.lambda$update$1(SnatCentralizedSwitchChangeListener.java:105) ~<span class="error">[?:?]</span><br/> at org.opendaylight.genius.infra.ManagedNewTransactionRunnerImpl.callWithNewReadWriteTransactionAndSubmit(ManagedNewTransactionRunnerImpl.java:106) <span class="error">[256:org.opendaylight.genius.mdsalutil-api:0.5.1]</span><br/> at org.opendaylight.netvirt.natservice.ha.SnatCentralizedSwitchChangeListener.update(SnatCentralizedSwitchChangeListener.java:98) <span class="error">[345:org.opendaylight.netvirt.natservice-impl:0.7.1]</span><br/> at org.opendaylight.netvirt.natservice.ha.SnatCentralizedSwitchChangeListener.update(SnatCentralizedSwitchChangeListener.java:43) <span class="error">[345:org.opendaylight.netvirt.natservice-impl:0.7.1]</span><br/> at org.opendaylight.genius.datastoreutils.AsyncDataTreeChangeListenerBase$DataTreeChangeHandler.run(AsyncDataTreeChangeListenerBase.java:176) <span class="error">[256:org.opendaylight.genius.mdsalutil-api:0.5.1]</span><br/> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) <span class="error">[?:?]</span></p> <p>at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) <span class="error">[?:?]</span><br/> at java.lang.Thread.run(Thread.java:748) <span class="error">[?:?]</span></p> <p>It is controller-based NAT.</p> <p>vagrant@odl:~/opendaylight-0.9.1$ cat etc/opendaylight/datastore/initial/config/netvirt-natservice-config.xml<br/> <natservice-config xmlns="urn:opendaylight:netvirt:natservice:config"><br/> <nat-mode>controller</nat-mode><br/> <snat-punt-timeout>5</snat-punt-timeout><br/> </natservice-config><br/> vagrant@odl:~/opendaylight-0.9.1$</p> <p> </p> <p>stack@control:~/devstack$ ./vm_ssh vm1<br/> qdhcp-0b9de268-2dd3-493a-91e2-f8fc71035313<br/> 10.15.1.3<br/> Warning: Permanently added '10.15.1.3' (RSA) to the list of known hosts.<br/> cirros@10.15.1.3's password:<br/> $ ping 192.168.100.1<br/> PING 192.168.100.1 (192.168.100.1): 56 data bytes<br/> ^C<br/> — 192.168.100.1 ping statistics —<br/> 14 packets transmitted, 0 packets received, 100% packet loss<br/> $</p> <p> </p> <p>192.168.100.1 is gateway of my external network, it is simulated as ODL csit setup, floating IP can work.</p> <p>stack@control:~/devstack$ curl --silent -u admin:admin -X GET <a href="http://192.168.0.5:8181/restconf/config/odl-nat:external-networks/" class="external-link" target="_blank" rel="nofollow noopener">http://192.168.0.5:8181/restconf/config/odl-nat:external-networks/</a> |python -m json.tool<br/> {<br/> "external-networks": {<br/> "networks": [</p> { "id": "031f4894-2437-42d4-b380-c575fc854b03", "provider-network-type": "FLAT", "router-ids": [ "fdbae3de-7bc7-4a7d-acf5-911bec787487" ], "vpnid": "031f4894-2437-42d4-b380-c575fc854b03" } <p> ]<br/> }<br/> }</p> <p>stack@control:~/devstack$ curl --silent -u admin:admin -X GET <a href="http://192.168.0.5:8181/restconf/config/odl-nat:ext-routers/" class="external-link" target="_blank" rel="nofollow noopener">http://192.168.0.5:8181/restconf/config/odl-nat:ext-routers/</a> |python -m json.tool<br/> {<br/> "ext-routers": {<br/> "routers": [<br/> {<br/> "enable-snat": true,<br/> "ext_gw_mac_address": "fa:16:3e:73:81:72",<br/> "external-ips": [</p> { "ip-address": "192.168.100.200", "subnet-id": "eb23897f-3d21-4beb-9bf0-9c6da6a9ebbf" } <p> ],<br/> "network-id": "031f4894-2437-42d4-b380-c575fc854b03",<br/> "router-name": "fdbae3de-7bc7-4a7d-acf5-911bec787487",<br/> "subnet-ids": [<br/> "2e7b7e8c-8c82-456b-a8e5-d71d23121260",<br/> "b7d4023a-bd4e-460e-9afe-30e16b79d842"<br/> ]<br/> }<br/> ]<br/> }<br/> }<br/> stack@control:~/devstack$ curl --silent -u admin:admin -X GET <a href="http://192.168.0.5:8181/restconf/config/odl-nat:napt-switches/" class="external-link" target="_blank" rel="nofollow noopener">http://192.168.0.5:8181/restconf/config/odl-nat:napt-switches/</a> |python -m json.tool<br/> {<br/> "napt-switches": {<br/> "router-to-napt-switch": [<br/> </p> { "primary-switch-id": 0, "router-name": "be9057dd-01d2-4f66-96cb-f1dc6dc39e18" } <p>,<br/> </p> { "primary-switch-id": 0, "router-name": "7027fad7-ee0d-4c5f-9001-f7d3645a1d52" } <p>,<br/> </p> { "primary-switch-id": 0, "router-name": "6532d936-74bd-4be5-adc7-e029ed1916cd" } <p>,<br/> </p> { "primary-switch-id": 0, "router-name": "7f632150-be8c-4e30-ae7d-fa5753fbf8fe" } <p>,</p> { "primary-switch-id": 128449969988000, "router-name": "fdbae3de-7bc7-4a7d-acf5-911bec787487" } <p> ]<br/> }<br/> }<br/> stack@control:~/devstack$ curl --silent -u admin:admin -X GET <a href="http://192.168.0.5:8181/restconf/config/odl-nat:intext-ip-port-map/" class="external-link" target="_blank" rel="nofollow noopener">http://192.168.0.5:8181/restconf/config/odl-nat:intext-ip-port-map/</a> |python -m json.tool<br/> {<br/> "intext-ip-port-map": {<br/> "ip-port-mapping": [<br/> {<br/> "intext-ip-protocol-type": [</p> { "protocol": "UDP" } <p> ],<br/> "router-id": 103063<br/> }<br/> ]<br/> }<br/> }</p> <p>stack@control:~/devstack$ curl --silent -u admin:admin -X GET <a href="http://192.168.0.5:8181/restconf/config/odl-nat:snatint-ip-port-map/" class="external-link" target="_blank" rel="nofollow noopener">http://192.168.0.5:8181/restconf/config/odl-nat:snatint-ip-port-map/</a> |python -m json.tool<br/> {<br/> "snatint-ip-port-map": {<br/> "intip-port-map": [<br/> {<br/> "ip-port": [<br/> {<br/> "int-ip-proto-type": [</p> { "protocol": "UDP" } <p> ],<br/> "internal-ip": "10.15.1.2"<br/> }<br/> ],<br/> "router-id": 103063<br/> }<br/> ]<br/> }<br/> }</p> <p>stack@control:~/devstack$ curl --silent -u admin:admin -X GET <a href="http://192.168.0.5:8181/restconf/config/odl-nat:floating-ip-port-info/" class="external-link" target="_blank" rel="nofollow noopener">http://192.168.0.5:8181/restconf/config/odl-nat:floating-ip-port-info/</a> |python -m json.tool<br/> {<br/> "floating-ip-port-info": {<br/> "floating-ip-id-to-port-mapping": [</p> { "floating-ip-deleted": true, "floating-ip-id": "a521a38e-e923-41dd-a92d-fc780f97cc12", "floating-ip-port-id": "144e61be-e70b-4e64-9aab-3e72a141e52a", "floating-ip-port-mac-address": "fa:16:3e:8a:72:4c", "floating-ip-port-subnet-id": "d5af3578-c1fa-423b-a680-a8cc80e1ac32" } <p> ]<br/> }<br/> }</p> <p>stack@control:~/devstack$ curl --silent -u admin:admin -X GET <a href="http://192.168.0.5:8181/restconf/config/odl-nat:floating-ip-info/" class="external-link" target="_blank" rel="nofollow noopener">http://192.168.0.5:8181/restconf/config/odl-nat:floating-ip-info/</a> |python -m json.tool<br/> {<br/> "floating-ip-info": {}<br/> }</p> <p>stack@control:~/devstack$ curl --silent -u admin:admin -X GET <a href="http://192.168.0.5:8181/restconf/config/odl-nat:router-id-name/" class="external-link" target="_blank" rel="nofollow noopener">http://192.168.0.5:8181/restconf/config/odl-nat:router-id-name/</a> |python -m json.tool<br/> {<br/> "router-id-name": {<br/> "routerIds": [<br/> </p> { "router-id": 103052, "router-name": "be9057dd-01d2-4f66-96cb-f1dc6dc39e18" } <p>,<br/> </p> { "router-id": 103003, "router-name": "902624c0-aaf5-4656-bf63-6258de0fd70c" } <p>,<br/> </p> { "router-id": 103022, "router-name": "7027fad7-ee0d-4c5f-9001-f7d3645a1d52" } <p>,<br/> </p> { "router-id": 103037, "router-name": "7f632150-be8c-4e30-ae7d-fa5753fbf8fe" } <p>,<br/> </p> { "router-id": 103063, "router-name": "fdbae3de-7bc7-4a7d-acf5-911bec787487" } <p>,</p> { "router-id": 103010, "router-name": "6532d936-74bd-4be5-adc7-e029ed1916cd" } <p> ]<br/> }<br/> }</p> <p>stack@control:~/devstack$ curl --silent -u admin:admin -X GET <a href="http://192.168.0.5:8181/restconf/operational/odl-nat:intext-ip-map/" class="external-link" target="_blank" rel="nofollow noopener">http://192.168.0.5:8181/restconf/operational/odl-nat:intext-ip-map/</a> |python -m json.tool<br/> {<br/> "intext-ip-map": {<br/> "ip-mapping": [<br/> {<br/> "ip-map": [<br/> </p> { "external-ip": "192.168.100.200/32", "internal-ip": "10.16.2.0/24", "label": 103068 } <p>,</p> { "external-ip": "192.168.100.200/32", "internal-ip": "10.15.1.0/24", "label": 103068 } <p> ],<br/> "segment-id": 103063<br/> }<br/> ]<br/> }<br/> }</p> <p>stack@control:~/devstack$ curl --silent -u admin:admin -X GET <a href="http://192.168.0.5:8181/restconf/operational/odl-nat:external-ips-counter/" class="external-link" target="_blank" rel="nofollow noopener">http://192.168.0.5:8181/restconf/operational/odl-nat:external-ips-counter/</a> |python -m json.tool<br/> {<br/> "external-ips-counter": {<br/> "external-counters": [<br/> {<br/> "external-ip-counter": [</p> { "counter": 2, "external-ip": "192.168.100.200/32" } <p> ],<br/> "segment-id": 103063<br/> },<br/> {<br/> "external-ip-counter": [</p> { "counter": 0, "external-ip": "192.168.100.209/32" } <p> ],<br/> "segment-id": 103010<br/> },<br/> {<br/> "external-ip-counter": [</p> { "counter": 0, "external-ip": "192.168.100.207/32" } <p> ],<br/> "segment-id": 103052<br/> },<br/> {<br/> "external-ip-counter": [</p> { "counter": 0, "external-ip": "192.168.100.200/32" } <p> ],<br/> "segment-id": 103037<br/> },<br/> {<br/> "external-ip-counter": [</p> { "counter": 0, "external-ip": "192.168.100.200/32" } <p> ],<br/> "segment-id": 103022<br/> }<br/> ]<br/> }<br/> }</p> <p>stack@control:~/devstack$ curl --silent -u admin:admin -X GET <a href="http://192.168.0.5:8181/restconf/operational/odl-nat:floating-ip-info/" class="external-link" target="_blank" rel="nofollow noopener">http://192.168.0.5:8181/restconf/operational/odl-nat:floating-ip-info/</a> |python -m json.tool<br/> {<br/> "floating-ip-info": {<br/> "router-ports": [<br/> {<br/> "external-network-id": "1b7f57dc-12cb-413a-b444-05253e46e23f",<br/> "ports": [</p> { "port-name": "e956fef4-4bd7-42a9-87b5-b5a996b895cc" } <p> ],<br/> "router-id": "7027fad7-ee0d-4c5f-9001-f7d3645a1d52"<br/> },<br/> {<br/> "external-network-id": "945b9f26-178e-4456-af02-c7cb5a2b316e",<br/> "ports": [</p> { "port-name": "50c56cba-7cc8-4b49-ae16-1620630d7ea8" } <p> ],<br/> "router-id": "7f632150-be8c-4e30-ae7d-fa5753fbf8fe"<br/> }<br/> ]<br/> }<br/> }</p> <p>Contoller Based SNAT doesn't support ICMP currently. Only TCP and UDP is supported. Since I see ping to external-gw, ICMP packets will be dropped by the following flow.</p> <p>Ex: </p> <p>TCP:cookie=0x81286c5, duration=914.453s, table=46, n_packets=1, n_bytes=74, priority=5,tcp,metadata=0x30d78/0xfffffe actions=CONTROLLER:65535,learn(table=46,hard_timeout=5,priority=7,cookie=0x81286c5,eth_type=0x800,nw_proto=6,NXM_OF_IP_SRC[],NXM_OF_IP_DST[],NXM_OF_TCP_SRC[],NXM_OF_TCP_DST[],OXM_OF_METADATA<span class="error">[1..23]</span>)</p> <p>ICMP:</p> <p><b>cookie=0x8000008, duration=914.453s, table=46, n_packets=6, n_bytes=588, priority=0,icmp,metadata=0x30d78/0xfffffe actions=drop</b></p> <p>UDP:</p> <p>cookie=0x81286c5, duration=914.453s, table=46, n_packets=1, n_bytes=59, priority=5,udp,metadata=0x30d78/0xfffffe actions=CONTROLLER:65535,learn(table=46,hard_timeout=5,priority=7,cookie=0x81286c5,eth_type=0x800,nw_proto=17,NXM_OF_IP_SRC[],NXM_OF_IP_DST[],NXM_OF_UDP_SRC[],NXM_OF_UDP_DST[],OXM_OF_METADATA<span class="error">[1..23]</span>)</p> <p> </p> <p>You can configure SNAT in conntrack mode for ICMP SNAT translation</p> <p>Ref: <a href="https://docs.opendaylight.org/en/stable-carbon/submodules/netvirt/docs/specs/conntrack-based-snat.html" class="external-link" target="_blank" rel="nofollow noopener">https://docs.opendaylight.org/en/stable-carbon/submodules/netvirt/docs/specs/conntrack-based-snat.html</a></p> <p> </p> <p>Got it, http works, thank you so much. I'll check if ping is ok for conntrack-based nat.</p> <p> </p> <p>$ curl <a href="http://192.168.100.1:8888/" class="external-link" target="_blank" rel="nofollow noopener">http://192.168.100.1:8888/</a><br/> <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd"><br/> <html><br/> <head><br/> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"><br/> <title>Directory listing for /</title><br/> </head><br/> <body><br/> <h1>Directory listing for /</h1><br/> <hr><br/> <ul><br/> <li><a href=".bash_history">.bash_history</a></li><br/> <li><a href=".bash_logout">.bash_logout</a></li><br/> <li><a href=".bashrc">.bashrc</a></li><br/> <li><a href=".cache/">.cache/</a></li><br/> <li><a href=".novaclient/">.novaclient/</a></li><br/> <li><a href=".profile">.profile</a></li><br/> <li><a href=".ssh/">.ssh/</a></li><br/> <li><a href=".viminfo">.viminfo</a></li><br/> <li><a href="devstack.tar.bz2">devstack.tar.bz2</a></li><br/> <li><a href="neutron.tar.bz2">neutron.tar.bz2</a></li><br/> <li><a href="nova.tar.bz2">nova.tar.bz2</a></li><br/> </ul><br/> <hr><br/> </body><br/> </html><br/> $</p> <p>Hi, Chetan</p> <p>After I changed to conntrack, snat can't work, ssh, http and ping 192.168.100.1 all don't work, so I propose this bug should be reopened.</p> <p>$ ping 192.168.100.1<br/> PING 192.168.100.1 (192.168.100.1): 56 data bytes<br/> ^C<br/> — 192.168.100.1 ping statistics —<br/> 5 packets transmitted, 0 packets received, 100% packet loss<br/> $ ssh yangyi@192.168.100.1<br/> ^C<br/> $ wget <a href="http://192.168.100.1:8888/" class="external-link" target="_blank" rel="nofollow noopener">http://192.168.100.1:8888/</a><br/> Connecting to 192.168.100.1:8888 (192.168.100.1:8888)<br/> ^C<br/> $</p> <p>After I used conntrack, the below rests are wrong:</p> <p> </p> <p>stack@control:~/devstack$ curl --silent -u admin:admin -X GET <a href="http://192.168.0.5:8181/restconf/operational/odl-nat:intext-ip-map/" class="external-link" target="_blank" rel="nofollow noopener">http://192.168.0.5:8181/restconf/operational/odl-nat:intext-ip-map/</a> |python -m json.tool<br/> {<br/> "errors": {<br/> "error": [</p> { "error-message": "Request could not be completed because the relevant data model content does not exist ", "error-tag": "data-missing", "error-type": "application" } <p> ]<br/> }<br/> }<br/> stack@control:~/devstack$ curl --silent -u admin:admin -X GET <a href="http://192.168.0.5:8181/restconf/operational/odl-nat:external-ips-counter/" class="external-link" target="_blank" rel="nofollow noopener">http://192.168.0.5:8181/restconf/operational/odl-nat:external-ips-counter/</a> |python -m json.tool<br/> {<br/> "errors": {<br/> "error": [</p> { "error-message": "Request could not be completed because the relevant data model content does not exist ", "error-tag": "data-missing", "error-type": "application" } <p> ]<br/> }<br/> }<br/> stack@control:~/devstack$ curl --silent -u admin:admin -X GET <a href="http://192.168.0.5:8181/restconf/config/odl-nat:intext-ip-port-map/" class="external-link" target="_blank" rel="nofollow noopener">http://192.168.0.5:8181/restconf/config/odl-nat:intext-ip-port-map/</a> |python -m json.tool<br/> {<br/> "errors": {<br/> "error": [</p> { "error-message": "Request could not be completed because the relevant data model content does not exist ", "error-tag": "data-missing", "error-type": "application" } <p> ]<br/> }<br/> }<br/> stack@control:~/devstack$</p> <p>I saw the below error karaf log, are they really errors?</p> <p> </p> <p>2019-03-06T03:42:32,784 | ERROR | DataTreeEventCallbackRegistrar-Timeouter-0 | NeutronPortChangeListener | 346 - org.opendaylight.netvirt.neutronvpn-impl - 0.7.2 | GwPort 79ef4012-5eef-40f3-95d3-95085f2543a6 added without Router<br/> 2019-03-06T03:42:34,317 | ERROR | DataTreeEventCallbackRegistrar-Timeouter-0 | NeutronPortChangeListener | 346 - org.opendaylight.netvirt.neutronvpn-impl - 0.7.2 | GwPort 0ce62d79-0ba7-47bd-a569-cdcaf6d75384 added without Router</p> Development Priority Rank 0|i03mv3: