https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/NETVIRT-1372 netvirt <p>The test fails when ti checks for the gw mac's in the table 19 flows. fa:16:3e:1e:c9:e6 in this test run is the one that failed to be removed. This test fails intermittently every other day or so.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> cookie=0x8220015, duration=7053.025s, table=19, n_packets=62, n_bytes=2604, priority=100,arp,arp_op=1 actions=resubmit(,17) cookie=0x8220016, duration=7053.025s, table=19, n_packets=28, n_bytes=1176, priority=100,arp,arp_op=2 actions=resubmit(,17) cookie=0x8000009, duration=49.935s, table=19, n_packets=0, n_bytes=0, priority=20,metadata=0x30e0c/0xfffffe,dl_dst=fa:16:3e:1e:c9:e6 actions=goto_table:21 cookie=0x1080000, duration=7053.025s, table=19, n_packets=3513, n_bytes=468116, priority=0 actions=resubmit(,17) cookie=0x88880000, duration=7052.683s, table=195, n_packets=193, n_bytes=8274, priority=0,arp actions=CONTROLLER:65535,learn(table=195,hard_timeout=5,priority=10,cookie=0x88880000,eth_type=0x806,NXM_OF_ARP_SPA[],NXM_OF_ARP_TPA[],OXM_OF_METADATA[24..39],load:0x1-&gt;NXM_NX_REG4[0..7]),learn(table=196,hard_timeout=5,priority=10,cookie=0x88880000,eth_type=0x806,NXM_OF_ARP_TPA[]=NXM_OF_ARP_SPA[],NXM_OF_ARP_SPA[]=NXM_OF_ARP_TPA[],OXM_OF_METADATA[24..39],load:0x1-&gt;NXM_NX_REG4[8..15]) </pre> </div></div> NETVIRT-1372

Fail: vpn basic ipv6.Delete Router And Router Interfaces With L3VPN: Verify GWMAC Flow Entry Removed From Flow Table' failed

Bug High Resolved Cannot Reproduce Somashekar B Sam Hague csit:exception Mon, 16 Jul 2018 11:27:59 +0000 Tue, 16 Oct 2018 11:14:29 +0000 Tue, 16 Oct 2018 11:14:29 +0000 Fluorine-SR1 ipv6service 0 4 <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=shague" class="user-hover" rel="shague">shague</a>, can you put a link to the robot log with this failure?</p> <p>This issue is not observed on the recent builds.<br/> I have verified builds from 76-111 which is from 27-July-18 to 2-Aug-18.<br/> This testcase has failed in builds 79, 88, 90, 91, 94 but for different reasons, mainly because of new exceptions during it's execution. But it has not failed for the reason specified in this bug.</p> <p>I see this issue in RH OSP CI recently, DEBUG karaf logs + robot log attached to: <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1616288" class="external-link" target="_blank" rel="nofollow noopener">https://bugzilla.redhat.com/show_bug.cgi?id=1616288</a></p> <p>Saw this issue in upstream Netvirt CSIT again:</p> <p><a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-snat-conntrack-fluorine/52/robot-plugin/log_08_vpn_basic.html.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-snat-conntrack-fluorine/52/robot-plugin/log_08_vpn_basic.html.gz</a></p> <p>it's still happening, re-opening</p> <p>I am not able to reproduce this issue.</p> <p>Observation:<br/> ----------------<br/> From the steps in CSIT, I could find that the stale flows in table 19 occur when router is associated with L3VPN.</p> <p>When subnets are added to router, for every router interface (dl_dst=MAC), table 19 flow is programmed with router metadata.<br/> When router is associated with L3VPN, the existing flows with router metadata will be deleted and a new set of flows with VPN metadata will be programmed for all the router interfaces.<br/> In this case, one flow with router metadata is left over as stale on 2 out of 3 DPNs. <br/> Now, when router interfaces are deleted, corresponding flows with VPN metadata will be deleted leaving router metadata flow as stale.<br/> Later, router is dissociated with L3VPN.</p> <p>In case, if we dissociate router from L3VPN and later delete router interfaces, we may not hit this stale issue.<br/> I don't recommend to change the CSIT steps, just want to make a note of my observation here.</p> <p>Logs with TRACE enabled might help a bit here.</p> <p>the last time this failed in 1node was over two weeks ago, but maybe it's just more frequent in the 3node jobs, which is where<br/> <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=thapar" class="user-hover" rel="thapar">thapar</a> saw it.</p> <p>what modules would we need TRACE enabled for? One thing we can try is running this suite many times in a row<br/> in the sandbox to see if it can be reproduced more commonly.</p> <p>Please enable TRACE for below modules:</p> <p>org.opendaylight.netvirt.neutronvpn<br/> org.opendaylight.netvirt.vpnmanager<br/> org.opendaylight.netvirt.fibmanager</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jluhrsen" class="user-hover" rel="jluhrsen">jluhrsen</a>, did you get a chance to get the TRACE logs for this issue?</p> <p>I've started a run with just this test suite and trace logs:</p> <p><a href="https://jenkins.opendaylight.org/sandbox/user/thapar/my-views/view/MyView/job/thapar-netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-snat-conntrack-neon/1/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/sandbox/user/thapar/my-views/view/MyView/job/thapar-netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-snat-conntrack-neon/1/</a></p> <p>Will share the results once available and rerun the suite if don't hit the issue.</p> <p>Could you explain the workaround you mentioned? Do you think changing order can 'avoid' this issue? If yes, I'd like to try that too as backup plan.</p> <p>First run didn't see the issue and I messed up the debugmap. Started a new run with correct debugmap.</p> <p>Update: Second run passed but no logs. I think TRACE generates way too many log entries than our jenkins can manage. Can we get more directed traces to reduce amount of logs?</p> <p><a href="https://jenkins.opendaylight.org/sandbox/user/thapar/my-views/view/MyView/job/thapar-netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-snat-conntrack-neon/2/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/sandbox/user/thapar/my-views/view/MyView/job/thapar-netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-snat-conntrack-neon/2/</a></p> <p>Hi Vishal,</p> <p>Try setting "org.opendaylight.netvirt.neutronvpn.NeutronHostConfigChangeListener" to INFO level.</p> <p>Hi Vishal,</p> <p>Thanks for running the test suite.</p> <p>From CSIT logs, issue has happened when router is associated with L3VPN. <br/> For one of the router interface (dl_dst=fa:16:3e:0c:a0:75), table 19 flow having router metadata is not getting deleted in 2 out of 3 DPNs (refer <span class="error">&#91;2&#93;</span>).<br/> So, when we perform router dissociation, it should come back to same stage as <span class="error">&#91;1&#93;</span> which was before router assocation with BGPVPN.<br/> Now if we delete router interfaces (removing subnets from router), these table 19 flows should get deleted.<br/> There should not be any stale flows in table 19.</p> <p><span class="error">&#91;1&#93;</span> Before router assocation with BGPVPN:</p> <p>cookie=0x8220015, duration=1035.396s, table=19, n_packets=45, n_bytes=1890, priority=100,arp,arp_op=1 actions=resubmit(,17) <br/> cookie=0x8220016, duration=1035.396s, table=19, n_packets=22, n_bytes=924, priority=100,arp,arp_op=2 actions=resubmit(,17) <br/> cookie=0x8000009, duration=173.732s, table=19, n_packets=0, n_bytes=0, priority=20,metadata=0x33c9e/0xfffffe,dl_dst=fa:16:3e:fa:a9:90 actions=goto_table:21 <br/> cookie=0x8000009, duration=169.363s, table=19, n_packets=0, n_bytes=0, priority=20,metadata=0x33c9e/0xfffffe,dl_dst=fa:16:3e:0c:a0:75 actions=goto_table:21 <br/> cookie=0x1080000, duration=1035.396s, table=19, n_packets=5503, n_bytes=569678, priority=0 actions=resubmit(,17)</p> <p><span class="error">&#91;2&#93;</span> After router assocation with BGPVPN:</p> <p>cookie=0x8220015, duration=1109.108s, table=19, n_packets=45, n_bytes=1890, priority=100,arp,arp_op=1 actions=resubmit(,17) <br/> cookie=0x8220016, duration=1109.108s, table=19, n_packets=22, n_bytes=924, priority=100,arp,arp_op=2 actions=resubmit(,17) <br/> cookie=0x8000009, duration=243.075s, table=19, n_packets=0, n_bytes=0, priority=20,metadata=0x33c9e/0xfffffe,dl_dst=fa:16:3e:0c:a0:75 actions=goto_table:21 <br/> cookie=0x8000009, duration=44.144s, table=19, n_packets=0, n_bytes=0, priority=20,metadata=0x33cb6/0xfffffe,dl_dst=fa:16:3e:fa:a9:90 actions=goto_table:21 <br/> cookie=0x8000009, duration=41.477s, table=19, n_packets=0, n_bytes=0, priority=20,metadata=0x33cb6/0xfffffe,dl_dst=fa:16:3e:0c:a0:75 actions=goto_table:21 <br/> cookie=0x1080000, duration=1109.108s, table=19, n_packets=5764, n_bytes=593666, priority=0 actions=resubmit(,17)<br/>    </p> <p>Hi Som,</p> <p>Could you clarify this? Set all other TRACE and this to INFO?</p> <p>Yes, Vishal. </p> <p>trying this here:<br/> <a href="https://jenkins.opendaylight.org/sandbox/user/thapar/my-views/view/MyView/job/thapar-netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-snat-conntrack-neon/4/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/sandbox/user/thapar/my-views/view/MyView/job/thapar-netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-snat-conntrack-neon/4/</a></p> <p>And we're not hitting the issue with TRACE enabled <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/sad.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Should we close this bug for now and re-open again if it's reproducible later?</p> <p>Yes, let's go ahead and close it for now.</p> Development Rank 0|i03grr: