https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/NETVIRT-509 netvirt <p>Steps:<br/> 1. l3vpn associated with router which has two subnet and corresponding VMs.<br/> 2. Datapath test works fine.<br/> 3. Dissociate l3vpn from router.<br/> 4. Remove interface from router and delete router.<br/> 5. Associate l3vpn with network <br/> 6. Verified fib entry and flow table 21, which has corresponding entry.<br/> 7. But datapath test failed between networks and its 100% packet loss.<br/> 8. Also tested OVS restart , which also has same behavior like 100% packet loss.<br/> 9 But when i did one VM instance restart on one of the network, the datapath works fine between networks.</p> <p>Attached the ODL Sandbox log which has ODL and OVS dump.</p> <p>Issue observed on both carbon and boron.</p> <p>Operating System: All<br/> Platform: All</p> NETVIRT-509

Dissociates l3vpn from router and then Associates with network has 100% packet loss

Bug Medium Resolved Won't Do Aswin Suryanarayanan Suvitha Balu Fri, 3 Mar 2017 08:31:27 +0000 Thu, 3 May 2018 14:37:15 +0000 Thu, 14 Dec 2017 17:58:28 +0000 Boron General 0 3 <p>Sandbox log: <a href="https://logs.opendaylight.org/sandbox/jenkins091/netvirt-csit-1node-openstack-newton-nodl-v2-upstream-learn-carbon/15/archives/log.html.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/netvirt-csit-1node-openstack-newton-nodl-v2-upstream-learn-carbon/15/archives/log.html.gz</a></p> <p>Hi Hanumant,</p> <p>This issue may be clearly related to the fix for <a href="https://jira.opendaylight.org/browse/NETVIRT-452" title="Vpn Interface not deleted from operational data store" class="issue-link" data-issue-key="NETVIRT-452"><del>NETVIRT-452</del></a> here:<br/> <a href="https://git.opendaylight.org/gerrit/52655" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/52655</a></p> <p>Can you do a check in sandbox flows/fibEntries and let me know?</p> <p>Vivek</p> <p>Please find our analysis :</p> <p>Ping 20.1.1.10/32(VM_IP_NET20) from 10.1.1.9/32(VM_IP_NET10) is being done.</p> <ul> <li>MAC Address : 10.1.1.9/32 fa:16:3e:34:de:4a<br/> 20.1.1.10/32 fa:16:3e:c3:ec:b2</li> <li>On a source DPN (i.e where VM_IP_NET10 is hosted) , we see below packet path</li> <li>Here neutron router is not present , instead neutron network is being associated to L3VPN. Hence when ARP is resolved for 10.1.1.1 (gateway IP) , mac-addr fe:16:3e:34:de:4a is returned in the ARP responder table=81</li> </ul> <p>table0 (in_port=2 actions=write_metadata:0x30000000000) ==&gt; <br/> table17 (metadata=0x30000000000, actions=write_metadata:0x6000030000000000) ==&gt; <br/> table40 (here it doesn't match any flow , so goes to default flow i.e actions=resubmit(,41),resubmit(,42) ==&gt;</p> <p>table41 (doesn't match any entry here)<br/> table42 (here it matches below entry)</p> <p>table=42, n_packets=122, n_bytes=11728, priority=61010,ip,metadata=0x30000000000/0xfffff0000000000 actions=learn(table=252,idle_timeout=300,priority=61010,delete_learned,cookie=0x6900000,eth_type=0x800,NXM_OF_IP_SRC[]=NXM_OF_IP_DST[],NXM_OF_IP_DST[]=NXM_OF_IP_SRC[],NXM_OF_IP_PROTO[],load:0x1-&gt;NXM_NX_REG5<span class="error">&#91;0..7&#93;</span>),resubmit(,17) ==&gt;</p> <p>table17 ==&gt; table19 . On table19 it must have matched one of the below 2 entries , but it didn't . Packet count shows 0.</p> <p>cookie=0x8000009, duration=26.309s, table=19, n_packets=0, n_bytes=0, priority=20,metadata=0x222f2/0xfffffffe,dl_dst=fe:16:3e:34:de:4a actions=goto_table:21 cookie=0x8000009, duration=26.119s, table=19, n_packets=0, n_bytes=0, priority=20,metadata=0x222f2/0xfffffffe,dl_dst=fe:16:3e:8a:e2:a9 actions=goto_table:21</p> <p>We suspect packet destMAC is tampered with , hence its not able to match the dl_dst in table=19.</p> <p>Hi Hanumant,</p> <p>Thanks for the analysis.</p> <p>It looks like with the steps put by submitter here, the VM would be completely unaware that it went out of a router-based-vpn and re-entered into a network-based-vpn. The VM might be holding the old MAC ARP resolved for the gateway-ip-address of 10.1.1.1. And so it might have used the same old gateway-mac-address incorrectly to send the IP Packets.</p> <p>Hi Suvitha,</p> <p>Can we please check with wireshark if the VM attempts ARPing after it sees IP Packet losses and through that ARPing it gets the new mac-address now applied for 10.1.1.1 which is fe:xx:xx:xx:xx rather than the router-interface mac-address.</p> <p>Vivek</p> <p>(In reply to Vivekanandan Narasimhan from comment #4)<br/> &gt; Hi Hanumant,<br/> &gt; <br/> &gt; Thanks for the analysis.<br/> &gt; <br/> &gt; It looks like with the steps put by submitter here, the VM would be<br/> &gt; completely unaware that it went out of a router-based-vpn and re-entered<br/> &gt; into a network-based-vpn. The VM might be holding the old MAC ARP resolved<br/> &gt; for the gateway-ip-address of 10.1.1.1. And so it might have used the same<br/> &gt; old gateway-mac-address incorrectly to send the IP Packets.<br/> &gt; <br/> &gt; Hi Suvitha,<br/> &gt; <br/> &gt; Can we please check with wireshark if the VM attempts ARPing after it sees<br/> &gt; IP Packet losses and through that ARPing it gets the new mac-address now<br/> &gt; applied for 10.1.1.1 which is fe:xx:xx:xx:xx rather than the<br/> &gt; router-interface mac-address.<br/> &gt; <br/> &gt; Vivek</p> <p>right now, we don't have a way to do pcaps in these openstack instances,<br/> but I had started to work on that a while back. I gave up and abandoned<br/> it when it sat for too long without work. Hanumant, do you have the <br/> cycles to finish it up? it's here:</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/45441/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/45441/</a></p> <p>This is the second time in maybe 6 months that we've wanted this, which<br/> isn't a lot, but it surely would be helpful sooner or later.</p> <p>JamO</p> <p>(In reply to Jamo Luhrsen from comment #5)<br/> &gt; (In reply to Vivekanandan Narasimhan from comment #4)<br/> &gt; &gt; Hi Hanumant,<br/> &gt; &gt; <br/> &gt; &gt; Thanks for the analysis.<br/> &gt; &gt; <br/> &gt; &gt; It looks like with the steps put by submitter here, the VM would be<br/> &gt; &gt; completely unaware that it went out of a router-based-vpn and re-entered<br/> &gt; &gt; into a network-based-vpn. The VM might be holding the old MAC ARP resolved<br/> &gt; &gt; for the gateway-ip-address of 10.1.1.1. And so it might have used the same<br/> &gt; &gt; old gateway-mac-address incorrectly to send the IP Packets.<br/> &gt; &gt; <br/> &gt; &gt; Hi Suvitha,<br/> &gt; &gt; <br/> &gt; &gt; Can we please check with wireshark if the VM attempts ARPing after it sees<br/> &gt; &gt; IP Packet losses and through that ARPing it gets the new mac-address now<br/> &gt; &gt; applied for 10.1.1.1 which is fe:xx:xx:xx:xx rather than the<br/> &gt; &gt; router-interface mac-address.<br/> &gt; &gt; <br/> &gt; &gt; Vivek<br/> &gt; <br/> &gt; right now, we don't have a way to do pcaps in these openstack instances,<br/> &gt; but I had started to work on that a while back. I gave up and abandoned<br/> &gt; it when it sat for too long without work. Hanumant, do you have the <br/> &gt; cycles to finish it up? it's here:<br/> &gt; <br/> &gt; <a href="https://git.opendaylight.org/gerrit/#/c/45441/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/45441/</a><br/> &gt; <br/> &gt; This is the second time in maybe 6 months that we've wanted this, which<br/> &gt; isn't a lot, but it surely would be helpful sooner or later.<br/> &gt; <br/> &gt; JamO</p> <p>I meant, Suvitha, not Hanumant <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Trying to reproduce this in local setup. Went through the pipeline, everything looked ok.</p> <p>Sure Jamo, i can explore on this.</p> <p>Hi Hari,</p> <p>Do we have any updates on this?</p> <p>Vivek</p> <p>ETA - 5th May 2017</p> <p>Hi Suvitha,</p> <p>I tried to reproduce this bug locally, and wasn't able to. PLease could you try it again and let me know if you still this issue.</p> <p>Regards<br/> Hari</p> <p>Log from Sandbox:</p> <p> <a href="https://logs.opendaylight.org/sandbox/jenkins091/netvirt-csit-1node-openstack-newton-upstream-learn-boron/1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/netvirt-csit-1node-openstack-newton-upstream-learn-boron/1</a></p> <p><a href="https://logs.opendaylight.org/sandbox/jenkins091/netvirt-csit-1node-openstack-newton-upstream-learn-carbon/2" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/netvirt-csit-1node-openstack-newton-upstream-learn-carbon/2</a></p> <p>Hi Suvitha,</p> <p>Thank you for running it again. The difference between the csit and my setup is that i don't configure ACL's. I ran the test without ACL's enabled. This seems to be the root cause. I need to investigate this further.</p> <p>I am putting the ETA as 10th May 2017 as i need to investigate this further. There is nothing to be done from L3VPN side.</p> <p>Regards<br/> Hari</p> <p>HI Suvitha,</p> <p>I have tried to recreate this manually. with manual steps it is working. I have tried this multiple times and don’t see an issue.</p> <p>Please can you add some delay of like 2-3 sec after router disassociation and also introduce a delay of 2-3 seconds after adding networks to L3VPN and before you ping.</p> <p>Can you try this an let me know.</p> <p>Regards<br/> Hari</p> <p>HI Suvitha,<br/> In the csit script, did you put a delay and check. Also in the script after network association, the ping count is 3. Can you make it 20 and test again. We saw some delay in ACL learning tables which introduces a delay. It takes time for ACL learning tables.<br/> Please can you check and revert back.</p> <p>Regards<br/> Hari</p> <p>(In reply to Hari Krishna from comment #15)<br/> &gt; HI Suvitha,<br/> &gt; In the csit script, did you put a delay and check. Also in the script after<br/> &gt; network association, the ping count is 3. Can you make it 20 and test again.<br/> &gt; We saw some delay in ACL learning tables which introduces a delay. It takes<br/> &gt; time for ACL learning tables.<br/> &gt; Please can you check and revert back.<br/> &gt; <br/> &gt; Regards<br/> &gt; Hari</p> <p>Hari,</p> <p>how long of a delay do you suggest? I don't think this problem is <br/> happening every time, so we would need to explain why sometimes there<br/> is a delay and other times not.</p> <p>I think the idea of increasing ping count to 20 for debugging is best, <br/> as that will give us an easy way to know how long the delay is.</p> <p>Suvitha,</p> <p>any chance you know of a failure for this bug in an releng job? the <br/> sandbox is purged on a weekly basis so the logs we have links for in<br/> this bug are not working any more.</p> <p>Thanks,<br/> JamO</p> <p>Hi Jamo/Suvitha,</p> <p>In our local setup what I have seen is after disassociate from router and associating to networks. The ACL learn tables namely<br/> 212,213,214 are not learning in a timely manner. There could be two issues.<br/> One as soon as networks are associated they would take some time to learn. That is why I suggested 2-3 seconds delay after associating networks<br/> and before trying the ping.<br/> Secondly what I have noticed is, the very first time ping is done, there is packet loss. This packet loss is between 10-15 packets before the ACL learn tables<br/> get populated. The CSIT test case only tried with packet count of 3, which would fail initially. That is why I have recommended to Suvitha to increase the<br/> packet count to 20 and test. <br/> But please not once the ACL learn tables are populated, no matter how many time you disassociate/associate networks, the traffic flow becomes normal.<br/> It’s only the first time that there is traffic loss.</p> <p>I am following up with the ACL team separately to find out, if the above two observations are valid and what is the expectation from the ACL learn tables.</p> <p>Regards<br/> Hari</p> <p>Hi Som </p> <p>the steps to reproduce this issue.</p> <p>#Create Neutron networks<br/> #List the networks <br/> #Create neutron subnets<br/> #add allow rules<br/> #Create neutron ports<br/> #Create nova VM's <br/> #Check ELAN data traffic within the networks<br/> #create tunnel<br/> #Create routers<br/> #Add interface to router<br/> #Check L2 path with router<br/> #Create L3VPN<br/> #Associate to router<br/> #dissassociate router<br/> #Remove subnets from router<br/> #delete router<br/> #Associate networks to l3vpn</p> <p>I have anyway sent you the log files, I am assigning this bug to you to have a look.</p> <p>Regards<br/> Hari</p> <p>Hi Som</p> <p>In our local setup what I have seen is after disassociate from router and associating to networks. The ACL learn tables namely<br/> 212,213,214 are not learning in a timely manner. There could be two issues.<br/> One as soon as networks are associated they would take some time to learn. That is why I suggested 2-3 seconds delay after associating networks<br/> and before trying the ping.<br/> Secondly what I have noticed is, the very first time ping is done, there is packet loss. This packet loss is between 10-15 packets before the ACL learn tables<br/> get populated. The CSIT test case only tried with packet count of 3, which would fail initially. That is why I have recommended to increase the<br/> packet count to 20 and test. <br/> But not once the ACL learn tables are populated, no matter how many time you disassociate/associate networks, the traffic flow becomes normal.<br/> It’s only the first time that there is traffic loss.</p> <p>Yes this testing is done one latest nitrogen. <br/> I am also attaching the log files.</p> <p>Regards<br/> Hari</p> <p>Hi Slava,</p> <p>As this issue is related to ACL in Learn mode, I am assigning this one to you. <br/> Please redirect to the right person, if needed.</p> <p>Thanks,<br/> Som</p> <p>Learn mode is deprecated.</p> Development External issue ID 7893 External issue URL ODL SR Target Milestone Rank 0|i01s8n: