OpenDaylight JIRA

OpenDaylight JIRA https://jira.opendaylight.org This file is an XML representation of an issue en-us

8.20.10

820010

22-06-2022

[NETVIRT-1144] PNF failure: Ping External Network PNF from Vm Instance 1 https://jira.opendaylight.org/browse/NETVIRT-1144 netvirt <p><a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-queens-upstream-stateful-oxygen/210/odl_1/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-queens-upstream-stateful-oxygen/210/odl_1/odl1_karaf.log.gz</a></p> NETVIRT-1144

PNF failure: Ping External Network PNF from Vm Instance 1

Bug High Resolved Duplicate Kiran N Upadhyaya Sam Hague csit:exception patch_merged Fri, 9 Mar 2018 02:03:20 +0000 Mon, 3 Dec 2018 13:52:31 +0000 Mon, 3 Dec 2018 12:09:47 +0000 Oxygen Fluorine General 0 6 0 minutes 6 hours <p>Hi Kiran,</p> <p> </p> <p>I raised a review(<a href="https://git.opendaylight.org/gerrit/#/c/69263/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/69263/</a>). Now this Exception is masked and table=21->22(10.10.10.0/24) been programmed on all computes, but the PNF use-case is still failing and this looks to be different issue.</p> <p> </p> <p>On analysis on dump, I see the FIB rule(for destination 10.10.10.253) is not getting hit even programmed, but the subsequent packets were also been getting punted to CSC</p> <p> </p> <p><a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-queens-gate-stateful-oxygen/54/robot-plugin/log_03_external_network.html.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-queens-gate-stateful-oxygen/54/robot-plugin/log_03_external_network.html.gz</a></p> <p> </p> <p><b>Before PNF Test-Case Execution</b></p> <p> </p> <p>cookie=0x8000003, duration=103.572s, table=21, <b>n_packets=0</b>, n_bytes=0, priority=34,ip,metadata=0x30d76/0xfffffe,nw_dst=10.10.10.0/24 actions=write_metadata:0x1393030d76/0xfffffffffe,goto_table:22</p> <p>cookie=0x8000004, duration=76.026s, table=25, <b>n_packets=6</b>, n_bytes=588, priority=10,ip,dl_dst=fa:16:3e:79:e3:6a,nw_dst=10.10.10.10 actions=set_field:41.0.0.7->ip_dst,write_metadata:0x30d7c/0xfffffe,goto_table:27</p> <p>cookie=0x8000004, duration=76.026s, table=26*, n_packets=6*, n_bytes=588, priority=10,ip,metadata=0x30d7c/0xfffffe,nw_src=41.0.0.7 actions=set_field:10.10.10.10->ip_src,write_metadata:0x30d76/0xfffffe,goto_table:28</p> <p>cookie=0x8000004, duration=76.026s, table=27, <b>n_packets=6</b>, n_bytes=588, priority=10,ip,metadata=0x30d7c/0xfffffe,nw_dst=41.0.0.7 actions=resubmit(,21)</p> <p>cookie=0x8000004, duration=76.026s, table=28, <b>n_packets=6</b>, n_bytes=588, priority=10,ip,metadata=0x30d76/0xfffffe,nw_src=10.10.10.10 actions=set_field:fa:16:3e:79:e3:6a->eth_src,resubmit(,21)</p> <p> </p> <p><b>After PNF Test-Case Execution (3 packet sent to 10.10.10.253 via ping)</b></p> <p> </p> <p>cookie=0x8000003, duration=12.320s, table=21, <b>n_packets=0</b>, n_bytes=0, priority=42,ip,metadata=0x30d76/0xfffffe,nw_dst=10.10.10.253 actions=set_field:f6:b3:8f:89:e1:2c->eth_dst,load:0x1d00->NXM_NX_REG6[],resubmit(,220) (Flow to reach destination been programmed, but not hit)</p> <p>cookie=0x8000003, duration=128.005s, table=21, <b>n_packets=3</b>, n_bytes=294, priority=34,ip,metadata=0x30d76/0xfffffe,nw_dst=10.10.10.0/24 actions=write_metadata:0x1393030d76/0xfffffffffe,goto_table:22 (all 3 packets been punted to CSC)</p> <p>cookie=0x8000004, duration=2450.898s, table=22, n_packets=3, n_bytes=294, priority=0 actions=CONTROLLER:65535</p> <p>cookie=0x8000004, duration=100.459s, table=25, n_packets=6, n_bytes=588, priority=10,ip,dl_dst=fa:16:3e:79:e3:6a,nw_dst=10.10.10.10 actions=set_field:41.0.0.7->ip_dst,write_metadata:0x30d7c/0xfffffe,goto_table:27</p> <p>cookie=0x8000004, duration=100.459s, table=26, n_packets=9, n_bytes=882, priority=10,ip,metadata=0x30d7c/0xfffffe,nw_src=41.0.0.7 actions=set_field:10.10.10.10->ip_src,write_metadata:0x30d76/0xfffffe,goto_table:28</p> <p> cookie=0x8000004, duration=100.459s, table=27, n_packets=6, n_bytes=588, priority=10,ip,metadata=0x30d7c/0xfffffe,nw_dst=41.0.0.7 actions=resubmit(,21)</p> <p>cookie=0x8000004, duration=100.459s, table=28, n_packets=9, n_bytes=882, priority=10,ip,metadata=0x30d76/0xfffffe,nw_src=10.10.10.10 actions=set_field:fa:16:3e:79:e3:6a->eth_src,resubmit(,21)</p> <p> </p> <p>When compared with pass case, only first packet punted to CSC and subsequent packet been hit the actual FIB rule(looks like the first packet is lost here too)</p> <p><span class="error">[1]</span><a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-queens-gate-stateful-oxygen/11/robot-plugin/log_03_external_network.html.gz#s1-t17-k4-k1-k2-k1-k12-k4" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-queens-gate-stateful-oxygen/11/robot-plugin/log_03_external_network.html.gz#s1-t17-k4-k1-k2-k1-k12-k4</a></p> <p> </p> <p>cookie=0x8000003, duration=112.184s, table=21, <b>n_packets=1</b>, n_bytes=98, priority=34,ip,metadata=0x30d76/0xfffffe,nw_dst=10.10.10.0/24 actions=write_metadata:0x1393030d76/0xfffffffffe,goto_table:22</p> <p>cookie=0x8000003, duration=5.254s, table=21, <b>n_packets=2</b>, n_bytes=196, priority=42,ip,metadata=0x30d76/0xfffffe,nw_dst=10.10.10.253 actions=set_field:7a:1e:56:99:cb:cc->eth_dst,load:0x1e00->NXM_NX_REG6[],resubmit(,220)</p> <p> </p> <p>Request to let me know your inputs on this.</p> <p> </p> <p>Thanks in Advance,</p> <p>Chetan</p> <p>Hi Sam,</p> <p> </p> <p>I don’t see the current PNF issue is related to any of these bug list(but might be due to delayed installation of FIB flow for dest-10.10.10.253)</p> <p> </p> <p>@Kiran,</p> <p> </p> <p>From the karaf logs, I see by the time FIB flow is processed, all the 3 ICMP packet been punted to CSC.</p> <p> </p> <p><b>3 ping packets punted to CSC</b></p> <p><b>2018-03-08T15:49:30,875</b> | INFO | pool-50-thread-1 | SubnetRoutePacketInHandler | 367 - org.opendaylight.netvirt.vpnmanager-impl - 0.6.0.SNAPSHOT | SUBNETROUTE: onPacketReceived: Processing IPv4 Packet received with Source IP 10.10.10.10 and Target IP 10.10.10.253 and vpnId 100027</p> <p><b>2018-03-08T15:49:31,877</b> | INFO | pool-50-thread-1 | SubnetRoutePacketInHandler | 367 - org.opendaylight.netvirt.vpnmanager-impl - 0.6.0.SNAPSHOT | SUBNETROUTE: onPacketReceived: Processing IPv4 Packet received with Source IP 10.10.10.10 and Target IP 10.10.10.253 and vpnId 100027</p> <p><b>2018-03-08T15:49:32,878</b> | INFO | pool-50-thread-1 | SubnetRoutePacketInHandler | 367 - org.opendaylight.netvirt.vpnmanager-impl - 0.6.0.SNAPSHOT | SUBNETROUTE: onPacketReceived: Processing IPv4 Packet received with Source IP 10.10.10.10 and Target IP 10.10.10.253 and vpnId 100027</p> <p> </p> <p><b>ARP response obtained immediated in 30 sec</b></p> <p><b>2018-03-08T15:49:30,884</b> | INFO | pool-50-thread-1 | ArpNotificationHandler | 367 - org.opendaylight.netvirt.vpnmanager-impl - 0.6.0.SNAPSHOT | Received ARP for sender MAC F6:B3:8F:89:E1:2C and sender IP 10.10.10.253 via interface 229136718462639:br-physnet1-pa:trunk</p> <p><b>2018-03-08T15:49:30,884</b> | INFO | pool-50-thread-1 | ArpNotificationHandler | 367 - org.opendaylight.netvirt.vpnmanager-impl - 0.6.0.SNAPSHOT | ARP being processed for Source IP 10.10.10.253</p> <p> </p> <p><b>Creation of FIB entry for this IP(10.10.10.253) is been done after in 32 sec</b></p> <p><b>2018-03-08T15:49:32,898</b> | INFO | ForkJoinPool-1-worker-0 | FibUtil | 350 - org.opendaylight.netvirt.fibmanager-impl - 0.6.0.SNAPSHOT | addOrUpdateFibEntry: Created/Updated vrfEntry for rd d1075ac5-6277-4e63-bc18-13330d751f73 prefix 10.10.10.253/32 nexthop [] label 0 l3vni 0 origin LOCAL encapType null</p> <p><b>2018-03-08T15:49:32,899</b> | INFO | ForkJoinPool-1-worker-0 | VpnInterfaceManager | 367 - org.opendaylight.netvirt.vpnmanager-impl - 0.6.0.SNAPSHOT | update: new Adjacency 10.10.10.253/32 with nextHop null label null subnet Uuid <span class="error">[_value=d1075ac5-6277-4e63-bc18-13330d751f73]</span> added to vpn interface 229136718462639:br-physnet1-pa:trunk on vpn dd4397b6-25c6-4532-84f6-cabc71760846 dpnId 229136718462639</p> <p> </p> <p>So, I suspect due to delayed installation of FIB flows for PNF, all the packets(3) been punted to CSC and been lost.</p> <p> </p> <p><a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-queens-gate-stateful-oxygen/54/robot-plugin/log_03_external_network.html.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-queens-gate-stateful-oxygen/54/robot-plugin/log_03_external_network.html.gz</a></p> <p><a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-queens-gate-stateful-oxygen/54/odl_1/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-queens-gate-stateful-oxygen/54/odl_1/odl1_karaf.log.gz</a></p> <p> </p> <p>Thanks,</p> <p>Chetan</p> <p>The fix <a href="https://git.opendaylight.org/gerrit/#/c/69263/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/69263/</a> is handling the OptimisticLockFailedException.</p> <p>Post this fix, two CSIT jobs been ranned and this exception not observed.</p> <p>But, the PNF UC is still failed during these run due to delayed installation of actual FIB flow for PNF(10.10.10.253) as commented above.</p> <p>This needs to be investigated.</p> <p> </p> <p>Thanks for the patch <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=xcheara" class="user-hover" rel="xcheara">xcheara</a>. Are you driving progress on resolving the remaining failure?</p> <h5><a name="kiran.n.upadhyaya%40ericsson.commailto%3Akiran.n.upadhyaya%40ericsson.com%C2%A0isfurtherinvestigatingontheissueofdelayedinstallationofFIBflow."></a><span class="nobr"><a href="mailto:kiran.n.upadhyaya@ericsson.com" class="external-link" target="_blank" rel="nofollow noopener">kiran.n.upadhyaya@ericsson.com<sup><img class="rendericon" src="https://jira.opendaylight.org/images/icons/mail_small.gif" height="12" width="13" align="absmiddle" alt="" border="0"/></sup></a></span> <b>is further investigating on the issue of delayed installation of FIB flow.</b></h5> <p><a href="https://git.opendaylight.org/gerrit/#/c/69263/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/69263/</a> ready for merge</p> <p>Patch is merged, thanks <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=shague" class="user-hover" rel="shague">shague</a>.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jluhrsen" class="user-hover" rel="jluhrsen">jluhrsen</a> et al are worried that something may still be going wrong under the hood, may a perf regression (used to take 1 second, now takes 3 or more). Three or four times worse perf could be a big deal if it's code that might be very active in some use case and we're just not hitting that code much in our CSIT.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=ecelgp" class="user-hover" rel="ecelgp">ecelgp</a> - To check on perf regressions in flow programming. In quick check he doesn't see regression from Nitrogen to Oxygen for flow programming time.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=shague" class="user-hover" rel="shague">shague</a> - To work with <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=kiranvasudeva" class="user-hover" rel="kiranvasudeva">kiranvasudeva</a> to try to clarify exactly what logic path is taking so long, seems to be narrowly scoped problem.</p> <p>Discussion about if this should be a blocker is ongoing. Will do more digging and see how it looks when we know more.</p> <p>FIxed the two second delay in PNF route learning here:</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/69429/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/69429/</a></p> <p> </p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=eupakir" class="user-hover" rel="eupakir">eupakir</a> - Awesome, thanks for tracking down the root cause.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=shague" class="user-hover" rel="shague">shague</a> - Please review 69429.</p> <p>this is still happening.</p> <p><a href="https://jenkins.opendaylight.org/releng/job/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-snat-conntrack-fluorine/90/robot/openstack/03%20external%20network/Initial%20Ping%20To%20External%20Network%20PNF%20from%20Vm%20Instance%201/" class="external-link" target="_blank" rel="nofollow noopener">examples here </a></p> <p>Hi Kiran/all,</p> <p>From the logs, we notice that the remote FIB flow for ip 10.10.10.253 is not getting installed in failure case though the packet is punted to controller, ping has failed due to this</p> <p>Please find below the flows in both success and failure cases</p> <p>Success case:</p> <p>cookie=0x8000003, duration=96.736s, table=21, n_packets=4, n_bytes=392, priority=34,ip,metadata=0x30d70/0xfffffe,nw_dst=10.10.10.0/24 actions=write_metadata:0x1393030d70/0xfffffffffe,goto_table:22</p> <p>cookie=0x8000010, duration=2781.341s, table=22, n_packets=2, n_bytes=196, priority=0,ip actions=CONTROLLER:65535,learn(table=22,hard_timeout=10,priority=10,cookie=0x8000010,eth_type=0x800,NXM_OF_IP_DST[],OXM_OF_METADATA<span class="error">[1..23]</span>)</p> <p>cookie=0x8000003, duration=11.138s, table=21, n_packets=2, n_bytes=196, priority=42,ip,metadata=0x30d70/0xfffffe,nw_dst=10.10.10.253 actions=set_field:2a:44:f6:95:27:78->eth_dst,load:0x1000->NXM_NX_REG6[],resubmit(,220)</p> <p>Failure case:</p> <p>cookie=0x8000003, duration=74.976s, table=21, n_packets=3, n_bytes=294, priority=34,ip,metadata=0x30d70/0xfffffe,nw_dst=10.10.10.0/24 actions=write_metadata:0x1393030d70/0xfffffffffe,goto_table:22</p> <p>cookie=0x8000010, duration=2759.581s, table=22, n_packets=1, n_bytes=98, priority=0,ip actions=CONTROLLER:65535,learn(table=22,hard_timeout=10,priority=10,cookie=0x8000010,eth_type=0x800,NXM_OF_IP_DST[],OXM_OF_METADATA<span class="error">[1..23]</span>)</p> <p>Regards,<br/> Zakir</p> <p>Reassigned Bug to Kiran for further debugging on Subnet Route Handling Feature.</p> <p>anything going on with this? we see it periodically still. Just saw it in our new <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/545/tim-ovsdb-netvirt-csit-1node-0cmb-1ctl-2cmp-apex-queens-gate-snat-conntrack-neon/59/robot-plugin/log_full.html.gz" class="external-link" target="_blank" rel="nofollow noopener">apex gate job </a><br/> which we want to make 100% stable so we can turn that in to a voting gate.</p> <p>Not yet <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jluhrsen" class="user-hover" rel="jluhrsen">jluhrsen</a>. I haven't been able to further debug it.</p> Relates NETVIRT-1399 Development Rank 0|i03bg7: