https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/NETVIRT-537 netvirt <p><a href="https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-newton-nodl-v2-upstream-transparent-carbon/413/archives/log.html.gz#s1-s2-s1-t10" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-newton-nodl-v2-upstream-transparent-carbon/413/archives/log.html.gz#s1-s2-s1-t10</a></p> <p>Operating System: All<br/> Platform: All</p> NETVIRT-537

CSIT Sporadic failures - TestNetworkAdvancedServerOps.test_server_connectivity_pause_unpause

Bug Highest Resolved Cannot Reproduce Chetan Arakere Gowdru Jamo Luhrsen csit:failures csit:snat-conntrack csit:sporadic Wed, 15 Mar 2017 18:22:15 +0000 Fri, 6 Apr 2018 00:02:28 +0000 Fri, 6 Apr 2018 00:02:28 +0000 Carbon General 0 4 <p>Hi Chetan,</p> <p>I know you're still into debugging tempest tests failures. It seems that most issues around tempest tests are resolved, and I have a feeling the rest of the bug fixes has contributed to that (fixes for table=21 rules, DHCP, etc).</p> <p>However, I can still see "total meldowns" here and there (Jamo has reported a few), like this report:<br/> <a href="https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-newton-nodl-v2-upstream-learn-carbon/209/archives/log.html" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-newton-nodl-v2-upstream-learn-carbon/209/archives/log.html</a></p> <p>Just wanted to update you on my analysis so far just in case you'll run into debugging those (we're entering a holiday this week, so won't be able to continue).</p> <p>The tests are basically failing on the inability to connect to VM FIP.<br/> During the times the test tries to connect to the VM, I’m constantly seeing the ARP counters in the ODL increasing:<br/> 2017-04-09 17:36:40,781 | INFO | CountersThread | CountersDumperThread | 294 - org.opendaylight.infrautils.counters-impl - 1.1.0.SNAPSHOT | auc.arp_req_rcv_notification: +4, auc.arp_req_rcv: +4<br/> 2017-04-09 17:36:41,781 | INFO | CountersThread | CountersDumperThread | 294 - org.opendaylight.infrautils.counters-impl - 1.1.0.SNAPSHOT | auc.arp_req_rcv_notification: +3, auc.arp_req_rcv: +3<br/> 2017-04-09 17:36:42,781 | INFO | CountersThread | CountersDumperThread | 294 - org.opendaylight.infrautils.counters-impl - 1.1.0.SNAPSHOT | auc.arp_req_rcv_notification: +3, auc.arp_req_rcv: +3</p> <p>My feeling is that one of relevant ARP responder rules is missing, and therefore when the connection to the VM is attempted ARP requests are constantly sent and punted to the controller.<br/> It's impossible to tell from the counters whether those ARP requests are directed to the FIP address OR to the VM router interface, but my guess is those are directed to the VM router interface (because there are prints on the FIP ARP responder rules being installed).</p> <p>I guess adding more logs around the installation of ARP responder rules for router interfaces might help here, or maybe even adding temporary INFO prints for ARP requests punted to the controller until we resolve those tempest issues.</p> <p>Figured out this might save you some time in case you're looking into those failures this week.</p> <p>Have a nice week,<br/> Koby</p> <p>Hi Koby/All,</p> <p>In order further investigate on this issue, I have added info logs as suggested.</p> <p>1) Added info logs for Arp Responder Install flow(to print flow-name which will have the FIP/VM router interface ip ex: "Arp:tbl_81:lport_2:gw_192.168.225.5")<br/> 2) Change log from trace to info when ARP requests are punted to controller.</p> <p>I hope these info logs are good enough to identify if there are ARP responder rules got missed resulting in this failure.</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/54593/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/54593/</a></p> <p>Thanks,<br/> Chetan</p> <p>tempest failures appear to mostly have been resolved. We can re-open any<br/> of the tempest bugs if/when they appear again in CSIT.</p> <p><a href="https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-newton-nodl-v2-upstream-stateful-carbon/657/log.html.gz#s1-s2-s1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-newton-nodl-v2-upstream-stateful-carbon/657/log.html.gz#s1-s2-s1</a></p> <p>seen in our snat-conntrack job with Oxygen:</p> <p><a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-queens-upstream-stateful-snat-conntrack-oxygen/214/" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-queens-upstream-stateful-snat-conntrack-oxygen/214/</a></p> Development External issue ID 7995 External issue URL Rank 0|i01sev: Status Whiteboard csit:sporadic_failures