OpenDaylight JIRA https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 [NETVIRT-1080] PNF failed after FIP detached https://jira.opendaylight.org/browse/NETVIRT-1080 netvirt <p>The reproduction steps .<br/> 1.External NW creation<br/> 2.Internal NW creation<br/> 3.Router creation and GW/IF setting<br/> 4.VM creation<br/> 5.SNAT confirmation<br/> OK<br/> 6.FIP attach<br/> 7.DNAT confirmation<br/> OK<br/> 8.FIP detach<br/> 9.SNAT confirmation<br/> NG</p> <p>ODL:Nitrogen-SR1 3 nodes<br/> OpenStack: Pike 3 nodes (1 controller, 2 compute)</p> NETVIRT-1080 PNF failed after FIP detached Bug Medium Resolved Done Vinh Nguyen Hari Prasidh Tue, 6 Feb 2018 12:30:39 +0000 Tue, 15 May 2018 14:18:28 +0000 Tue, 15 May 2018 14:18:17 +0000 Nitrogen Oxygen Oxygen-SR2 Fluorine General 0 7 <p>Some investigations as below:</p> <p>1) Could not reproduce the issue using Nitrogen SR2. Please re-test with the SR2 or later</p> <p>2) The attached OVS_info.zip doesn't seem to relate to the setup in log_full.html.gz</p> <p>3) In  the OVS_info.zip setup, the external gateway doesn't seem to be up-and-running.</p> <p>    The external network group entry for all three nodes drop external traffic:</p> <p>      cookie=0x8000004, duration=7288.519s, table=21, n_packets=1910, n_bytes=187180, priority=10,ip,metadata=0x324b2/0xfffffe actions=group:225000</p> <p>      group_id=225000,type=all,bucket=actions=drop</p> <p>This indicates the external network group installer failed to detect mac address of the  external gateway.</p> <p> </p> <p>4) In the scenario that captured in log_full.html.gz, I don't see the above issue in 3).</p> <p>   Could we retry the scenario and send me the latest log?</p> <p>If you look on below failed test scenario from the log.html. You can find above issue.<br/> Ping External Network PNF from Vm Instance 1 (non-critical).</p> <p>And recent CSIT jobs are not having the test scenarios what I've mentioned in issue reproduction steps.</p> <p>Issue can be reproducible in sandbox jobs .</p> <p><a href="https://logs.opendaylight.org/sandbox/vex-yul-odl-jenkins-2/netvirt-csit-1node-openstack-pike-upstream-stateful-snat-conntrack-nitrogen/15/robot-plugin/log_full.html.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/vex-yul-odl-jenkins-2/netvirt-csit-1node-openstack-pike-upstream-stateful-snat-conntrack-nitrogen/15/robot-plugin/log_full.html.gz</a></p> <p><a href="https://logs.opendaylight.org/sandbox/vex-yul-odl-jenkins-2/netvirt-csit-1node-openstack-pike-upstream-stateful-snat-conntrack-nitrogen/17/robot-plugin/log_full.html.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/vex-yul-odl-jenkins-2/netvirt-csit-1node-openstack-pike-upstream-stateful-snat-conntrack-nitrogen/17/robot-plugin/log_full.html.gz</a></p> <p>please find the attachments for logs <span class="nobr"><a href="https://jira.opendaylight.org/secure/attachment/14444/14444_SandboxJobs.zip" title="SandboxJobs.zip attached to NETVIRT-1080">SandboxJobs.zip<sup><img class="rendericon" src="https://jira.opendaylight.org/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/></sup></a></span></p> <p>This is also hitting oxygen: <a href="https://jenkins.opendaylight.org/releng/me/my-views/view/netvirt-oxygen-queens/job/netvirt-csit-1node-openstack-queens-upstream-stateful-oxygen/207/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/me/my-views/view/netvirt-oxygen-queens/job/netvirt-csit-1node-openstack-queens-upstream-stateful-oxygen/207/</a></p> <p>This issue is not related to the PNF/SNAT issue in recent CSIT. The issue is found in sandbox where extra tests are added to the end of the current external-network test cases. The additional test cases are:</p> <p> </p> <ul> <li>Delete the FIP for VM instance1: PASS</li> <li>SNAT TCP connection to External Gateway From VM Instance1 : PASS</li> <li>SNAT UDP connection to External Gateway From VM Instance1 : PASS</li> <li>Ping External Network PNF from Vm Instance 1: FAIL</li> </ul> <p>Here, PNF ping fails when the FIP is deleted. We would expect PNF scenario continues to work via SNAT</p> <p> </p> <p>Update title to 'PNF failed after FIP Detached'<br/> Reason: Based on the attched CSIT report, the SNAT TCP/UDP connection verification passed, <br/> the failure was in PNF verification after FIP detached</p> <p>Investigation:</p> <p>Three nodes, control, compute1, compute2</p> <p>1.External NW creation<br/> 2.Internal NW creation<br/> 3.Router creation and GW/IF setting<br/> 4.VMs creation: VM1 on compute1 node, VM2 on compute2 ndoe</p> <p>The PNF SubnetRoute flow entries are installed for ALL 3 nodes</p> <p>cookie=0x8000003, duration=339.441s, table=21, n_packets=0, n_bytes=0, priority=34,ip,metadata=0x30d42/0xfffffe,nw_dst=10.10.10.0/24 actions=write_metadata:0x138c030d42/0xfffffffffe,goto_table:22<br/> cookie=0x8000004, duration=339.441s, table=22, n_packets=0, n_bytes=0, priority=42,ip,metadata=0x30d42/0xfffffe,nw_dst=10.10.10.255 actions=drop<br/> cookie=0x8000004, duration=902.422s, table=22, n_packets=0, n_bytes=0, priority=0 actions=CONTROLLER:65535</p> <p>5.SNAT confirmation<br/> 6.FIPs attach<br/> 7.DNAT confirmation<br/> 8.FIPs for VMs detach<br/> 9.VM pings external PNF instances: FAILED</p> <p>Problem:</p> <p>The PNF subnetRoute flow entries are removed on the OVS node that hosted the VM after detaching the FIP (in step 8).<br/> Hence traffic from VM on that OVS node to the PNF instance is no longer possible.</p> <p>The PNF subnetRoute flow entries are removed on the dpn when the FIP port is the last port for the VPN on that dpn:</p> <p><a href="https://github.com/opendaylight/netvirt/blob/master/vpnmanager/impl/src/main/java/org/opendaylight/netvirt/vpnmanager/VpnFootprintService.java#L352" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/opendaylight/netvirt/blob/master/vpnmanager/impl/src/main/java/org/opendaylight/netvirt/vpnmanager/VpnFootprintService.java#L352</a></p> <p>The VpnToDpnList for the external subnet after FIP is attached (step 6):<br/> {<br/> "vpn-id": 100012,<br/> "vpn-instance-name": "ddf97de4-0a2d-48a8-b7d3-af8ffdae6761",<br/> "vpn-state": "created",<br/> "vpn-to-dpn-list": [<br/> {<br/> "dpn-state": "active",<br/> "dpnId": 8796751999625,<br/> "ip-addresses": [<br/> </p> { "ip-address": "192.168.56.18/32", "ip-address-source": "ExternalFixedIP" } <p>,</p> { "ip-address": "192.168.56.13/32", "ip-address-source": "FloatingIP" } <p> ]<br/> },<br/> {<br/> "dpn-state": "active",<br/> "dpnId": 8796748560798,<br/> "ip-addresses": [</p> { "ip-address": "192.168.56.17/32", "ip-address-source": "FloatingIP" } <p> ]<br/> }<br/> ],<br/> "vrf-id": "ddf97de4-0a2d-48a8-b7d3-af8ffdae6761"<br/> }</p> <p>Notes: </p> <ul class="alternate" type="square"> <li>Two compute nodes, dpnid: 8796751999625, 8796748560798</li> <li>Ports on 8796751999625:<br/> + 192.168.56.18: router external GW interface <br/> + 192.168.56.13: FIP for VM1</li> <li>Ports on 8796748560798:<br/> + 192.168.56.17: FIP for VM2</li> </ul> <p>The VpnToDpnList for the external subnet after FIP is deleted (step 8):</p> <p>{<br/> "vpn-id": 100012,<br/> "vpn-instance-name": "ddf97de4-0a2d-48a8-b7d3-af8ffdae6761",<br/> "vpn-state": "created",<br/> "vpn-to-dpn-list": [<br/> {<br/> "dpn-state": "active",<br/> "dpnId": 8796751999625,<br/> "ip-addresses": [</p> { "ip-address": "192.168.56.18/32", "ip-address-source": "ExternalFixedIP" } <p> ]<br/> },</p> { "dpn-state": "inactive", "dpnId": 8796748560798 } <p> ],<br/> "vrf-id": "ddf97de4-0a2d-48a8-b7d3-af8ffdae6761"<br/> }</p> <p>After detaching the FIP on 8796748560798 the vpn-to-dpn-list for external subnet vpn is empty,<br/> fibManager.cleanUpDpnForVpn is called to clean up the PNF flow entries.</p> <p><a href="https://github.com/opendaylight/netvirt/blob/master/vpnmanager/impl/src/main/java/org/opendaylight/netvirt/vpnmanager/VpnFootprintService.java#L354" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/opendaylight/netvirt/blob/master/vpnmanager/impl/src/main/java/org/opendaylight/netvirt/vpnmanager/VpnFootprintService.java#L354</a></p> <p>Suggested solution:</p> <ul class="alternate" type="square"> <li>Method FibManager.cleanUpDpnForVpn cleans up flow entries associating with the one VPN such as<br/> SubnetRoute, BroadCast,etc. For internal VPN, these flow entries are created for internal VPN <br/> when at least one VPN interfaces exists on the VPN and should be removed when the last VPN <br/> interface are removed.</li> <li>For external subnet VPN, the flow entries mentioned above are created when the subnet is created.</li> <li>Therefore, when deleting last VPN interface on external subnet VPN, simply remove VpnToDpnList<br/> associated with the VPN. The cleanup DPN for external subnet VPN will be done when the external<br/> subnet is deleted.</li> </ul> <p>Patch: </p> <p><a href="https://git.opendaylight.org/gerrit/#/c/69102/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/69102/</a></p> <p> </p> <p>CSIT verification pending.</p> <p>reading the commit message in the <a href="https://git.opendaylight.org/gerrit/#/c/69102/" class="external-link" target="_blank" rel="nofollow noopener">patch</a> makes sense for why we<br/> could lose connectivity (flow removed), but I think the connectivity does eventually return. What is making that<br/> happen?</p> <p>commit message:</p> <blockquote><p>Problem:<br/> Deleting last FIP port on dpn also deleting the PNF flow<br/> entries ion the OVS node.<br/> Solution:<br/> Don't invoke fibManager.cleanUpDpnForVpn (which removes<br/> the PNF flows) when last port on external subnet vpn is<br/> deleted on the dpn.</p></blockquote> <p>sorry the previous analysis is incorrect.</p> <p>Revised analysis:</p> <p>I found that the problem ocurs only when:</p> <ul class="alternate" type="square"> <li>NAT conntract mode is used.</li> <li>The deleted FIP VM is on the NAPT switch.</li> </ul> <p>The vpn-to-dpn-list for the external subnet on each compute dpn contains single IP address -<br/> the FIP. The external router GW interface is contained in the vpn-to-dpn-list on the control node.</p> <p>If the FIP on the NAPT switch is deleted, since it is the last address on the<br/> vpn-to-dpn-list. fibManager.cleanUpDpnForVpn is invoked and the PNF flows will be<br/> removed from the dpn as a result.</p> <p>NAPT controller mode doesn't have this issue because the external router GW interface IP is<br/> contained in the vpn-to-dpn-list of the NAPT switch. Thus deleting the last FIP will not<br/> invoke fibManager.cleanUpDpnForVpn since the router GW interface IP still exists in the dpn.</p> <p>The following is the vpn-instance-op-data-entry for the external subnet when FIPs added:</p> <p>Conntrack mode:<br/> From the log for SandboxJobs/job1, the dpnid of nodes are:</p> <ul class="alternate" type="square"> <li>control: 223071002466895, 189fdbb1-eab1-4108-9b2a-bff343503552: external router gw interface</li> <li>compute1: 73535277218113 - NAPT switch, FIP: 10.10.10.13</li> <li>compute2: 116882536471118 - non NAPT switch, FIP: 10.10.10.4</li> </ul> <p> </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>{   "vpn-id": 100001,   "vpn-instance-name": "aa5d4ad7-8f17-4efe-b0cf-bec0ccc2c086",   "vpn-state": "created",   "vpn-to-dpn-list": [     {        "dpn-state": "active",        "dpnId": 223071002466895,        "vpn-interfaces": [           {             "interface-name": "189fdbb1-eab1-4108-9b2a-bff343503552"           }       ]    },    {       "dpn-state": "active",       "dpnId": 73535277218113,       "ip-addresses": [           {             "ip-address": "10.10.10.13/32",             "ip-address-source": "FloatingIP"           }        ]     },     {       "dpn-state": "active",       "dpnId": 116882536471118,       "ip-addresses": [           {             "ip-address": "10.10.10.4/32",             "ip-address-source": "FloatingIP"           }        ]     }   ],   "vrf-id": "aa5d4ad7-8f17-4efe-b0cf-bec0ccc2c086" }, The following is the after FIP for VM1 is deleted: {   "vpn-id": 100001,   "vpn-instance-name": "aa5d4ad7-8f17-4efe-b0cf-bec0ccc2c086",   "vpn-state": "created",   "vpn-to-dpn-list": [     {       "dpn-state": "active",       "dpnId": 223071002466895,       "vpn-interfaces": [           {            "interface-name": "189fdbb1-eab1-4108-9b2a-bff343503552"          }       ]     },     {       "dpn-state": "inactive",       "dpnId": 73535277218113     } ,     {       "dpn-state": "active",       "dpnId": 116882536471118,       "ip-addresses": [         {           "ip-address": "10.10.10.4/32",           "ip-address-source": "FloatingIP"         }       ]    }   ],   "vrf-id": "aa5d4ad7-8f17-4efe-b0cf-bec0ccc2c086" }, For NAPT controller mode: FIPs added: {   "vpn-id": 100012,   "vpn-instance-name": "ddf97de4-0a2d-48a8-b7d3-af8ffdae6761",   "vpn-state": "created",   "vpn-to-dpn-list": [     {       "dpn-state": "active",       "dpnId": 8796751999625,       "ip-addresses": [         {           "ip-address": "192.168.56.18/32",           "ip-address-source": "ExternalFixedIP"         } ,         {          "ip-address": "192.168.56.13/32",          "ip-address-source": "FloatingIP"        }      ]   },   {       "dpn-state": "active",        "dpnId": 8796748560798,        "ip-addresses": [           {            "ip-address": "192.168.56.17/32",            "ip-address-source": "FloatingIP" }          ]       }   ],   "vrf-id": "ddf97de4-0a2d-48a8-b7d3-af8ffdae6761" } FIP on the NAPT switch (8796751999625) deleted: {   "vpn-id": 100012,   "vpn-instance-name": "ddf97de4-0a2d-48a8-b7d3-af8ffdae6761",   "vpn-state": "created",   "vpn-to-dpn-list": [     {       "dpn-state": "active",       "dpnId": 8796751999625,       "ip-addresses": [         {           "ip-address": "192.168.56.18/32",           "ip-address-source": "ExternalFixedIP"         }       ]     },     {       "dpn-state": "active",       "dpnId": 8796748560798,       "ip-addresses": [         {           "ip-address": "192.168.56.17/32",           "ip-address-source": "FloatingIP" }       ]     }   ],   "vrf-id": "ddf97de4-0a2d-48a8-b7d3-af8ffdae6761" </pre> </div></div> <p> }</p> Development Rank 0|i03a13: