https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/NETVIRT-1022 netvirt <p>From time to time we find some jobs that really go south and have lots of failures. When<br/> debugging and looking at the karaf.log there are many red flags, but one is the repeat<br/> logging of this message:</p> <p>2017-11-21 00:15:51,069 | INFO | pool-43-thread-1 | NaptPacketInHandler | 336 - org.opendaylight.netvirt.natservice-impl - 0.6.0.SNAPSHOT | onPacketReceived : Retry Packet IN Queue Size : 0</p> <p>in this job <span class="error">&#91;0&#93;</span> that message was printed over 900 times</p> <p>something fundamental is broken in the environment and maybe the above log<br/> message is benign. It does not appear very much in the jobs that are passing and<br/> healthy.</p> <p>here is the karaf.log <span class="error">&#91;1&#93;</span> for that job <span class="error">&#91;0&#93;</span>. The log also has a lot of NullPointerExceptions,<br/> TransactionCommitFailedException, OptimisticLockFailedException, and UnsupportedOperationException.</p> <p>more debugging and analysis is needed to figure out what is really broken.</p> <p><span class="error">&#91;0&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-ocata-upstream-stateful-oxygen/397/log_connectivity.html.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-ocata-upstream-stateful-oxygen/397/log_connectivity.html.gz</a><br/> <span class="error">&#91;1&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-ocata-upstream-stateful-oxygen/397/odl_1/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-ocata-upstream-stateful-oxygen/397/odl_1/odl1_karaf.log.gz</a></p> NETVIRT-1022

CSIT Sporadic failures - major functionality breakage

Bug Highest Verified Done Arunprakash D Jamo Luhrsen csit:exception csit:failures Tue, 21 Nov 2017 18:40:57 +0000 Sat, 10 Feb 2018 00:14:24 +0000 Sat, 10 Feb 2018 00:14:14 +0000 Carbon-SR3 Carbon-SR3 General 0 7 <p>Hi Chetan / Karthikeyan ,</p> <p>Please put the appropriate log level for the one which is appearing 900 times. It seems to be flooding the karaf logs.</p> <p>Please analyze other exceptions as well.</p> <p> </p> <p>btw, I already have this <a href="https://git.opendaylight.org/gerrit/#/c/63958/" class="external-link" target="_blank" rel="nofollow noopener">patch</a> which changes debug level from INFO to DEBUG</p> <p>Reason for failed TCs of FloatingIP Ping and SNAT TCP traffic:</p> <p>================================================</p> <p>Due to VM instance-2 is not getting IP address FloatingIP (DNAT) and SNAT (TCP) traffics are getting failed.</p> <p>Regarding SNAT packet is getting looped because of internal VM IP is not aware of the ODL-Controller. Any way packet will be getting looped by maximum for 4 Secs (Within 4 Secs if the SNAT session flow is not getting installed on the OVS switch, packet will be dropped by ODL-Controller). This logging(which is changed from DEBUG to INFO with patch - <a href="https://git.opendaylight.org/gerrit/#/c/63958/)" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/63958/)</a> will not impact any functionality.</p> <p> </p> <p>With Jamo's <a href="https://git.opendaylight.org/gerrit/#/c/63958/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/63958/</a> merged, we don't see the repeating log, but now what do we do with this bug?</p> <p>Karthikeyan, are you saying the issue is related to the issue where we don't get dhcp address in some subnets: <a href="https://jira.opendaylight.org/browse/NETVIRT-1038?" class="external-link" rel="nofollow">https://jira.opendaylight.org/browse/NETVIRT-1038?</a> That issue is still being worked.</p> <p>Still we have been observed VM Instance 1 and 2 didn't get IP address due to DHCP and the corresponding TC also been failed. Hence corresponding VMs NAT use cases also been failed.</p> <p>TC: Check Vm Instances Have Ip Address<br/> --------------------------------------------------------<br/> <a href="https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-ocata-upstream-stateful-oxygen/524/log_03_external_network.html.gz#s1-t6" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-ocata-upstream-stateful-oxygen/524/log_03_external_network.html.gz#s1-t6</a></p> <p>Latest CSIT Log:<br/> ------------------------<br/> <a href="https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-ocata-upstream-stateful-oxygen/524/" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-ocata-upstream-stateful-oxygen/524/</a></p> <p>This issue is dependent on DHCP failure issue.</p> <p><a href="https://jira.opendaylight.org/browse/NETVIRT-1038" class="external-link" rel="nofollow">https://jira.opendaylight.org/browse/NETVIRT-1038</a></p> <p><a href="https://jira.opendaylight.org/browse/NETVIRT-1022" title="CSIT Sporadic failures - major functionality breakage" class="issue-link" data-issue-key="NETVIRT-1022"><del>NETVIRT-1022</del></a> Analysis:<br/> ===================<br/> Have analysed VM Instance 2 failure use cases and observed that VM instance 2 did not responded to any SNAT/DNAT request because of missing IP address of the VM due to DHCP failure known issue (<a href="https://jira.opendaylight.org/browse/NETVIRT-1038" class="external-link" rel="nofollow">https://jira.opendaylight.org/browse/NETVIRT-1038</a>)</p> <p>Please refer the below details for passed (VM Instance 1) and failed (VM Instance 2) dump-flows/groups analysis based on the packet flow. Based on the below analysis if VM instance 2 IP address has been assigned properly through DHCP means both SNAT and DNAT traffic should work as expected.</p> <p> Failed TC: <a href="https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-ocata-upstream-stateful-oxygen/524/log_03_external_network.html.gz#s1-t15" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-ocata-upstream-stateful-oxygen/524/log_03_external_network.html.gz#s1-t15</a></p> <p> VM Instance 2 Fixed(local) IP: 41.0.0.9 (Actually IP is not been assigned to VM Instance 2)<br/> VM Instance 2 Floating-IP: 10.10.10.13</p> <p> FIP Ping Request has reached to VM Instance 2<br/> --------------------------------------------------------------------<br/> cookie=0x8000003, duration=55.458s, table=21, n_packets=3, n_bytes=294, priority=42,ip,metadata=0x30d46/0xfffffe,nw_dst=10.10.10.13 actions=set_field:fa:16:3e:94:58:18-&gt;eth_dst,goto_table:25<br/> cookie=0x8000004, duration=55.443s, table=25, n_packets=3, n_bytes=294, priority=10,ip,dl_dst=fa:16:3e:94:58:18,nw_dst=10.10.10.13 actions=set_field:41.0.0.9-&gt;ip_dst,write_metadata:0x30d54/0xfffffe,goto_table:27<br/> cookie=0x8000004, duration=55.443s, table=27, n_packets=3, n_bytes=294, priority=10,ip,metadata=0x30d54/0xfffffe,nw_dst=41.0.0.9 actions=resubmit(,21)<br/> cookie=0x8000003, duration=92.427s, table=21, n_packets=3, n_bytes=294, priority=42,ip,metadata=0x30d54/0xfffffe,nw_dst=41.0.0.9 actions=group:150002</p> <p>No Response from VM Instance 2: (FIP Response)<br/> -----------------------------------------------------------------------<br/> cookie=0x8000004, duration=55.443s, table=26, n_packets=0, n_bytes=0, priority=10,ip,metadata=0x30d54/0xfffffe,nw_src=41.0.0.9 actions=set_field:10.10.10.13-&gt;ip_src,write_metadata:0x30d46/0xfffffe,goto_table:28<br/> cookie=0x8000004, duration=55.443s, table=28, n_packets=0, n_bytes=0, priority=10,ip,metadata=0x30d46/0xfffffe,nw_src=10.10.10.13 actions=set_field:fa:16:3e:94:58:18-&gt;eth_src,resubmit(,21)<br/> cookie=0x8000004, duration=109.905s, table=21, n_packets=0, n_bytes=0, priority=10,ip,metadata=0x30d46/0xfffffe actions=group:225002</p> <p>group_id=225002,duration=111.080s,ref_count=1,packet_count=0,byte_count=0,bucket0:packet_count=0,byte_count=0</p> <p> Passed TC: <a href="https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-ocata-upstream-stateful-oxygen/524/log_03_external_network.html.gz#s1-t14" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-ocata-upstream-stateful-oxygen/524/log_03_external_network.html.gz#s1-t14</a><br/> VM Instance 1 Fixed(local) IP: 41.0.0.6<br/> VM Instance 1 Floating-IP: 10.10.10.5</p> <p>FIP Ping Request has reached to VM Instance 1<br/> ----------------------------------------------------------------------<br/> cookie=0x8000003, duration=38.717s, table=21, n_packets=3, n_bytes=294, priority=42,ip,metadata=0x30d46/0xfffffe,nw_dst=10.10.10.5 actions=set_field:fa:16:3e:e9:c1:4e-&gt;eth_dst,goto_table:25<br/> cookie=0x8000004, duration=38.704s, table=25, n_packets=3, n_bytes=294, priority=10,ip,dl_dst=fa:16:3e:e9:c1:4e,nw_dst=10.10.10.5 actions=set_field:41.0.0.6-&gt;ip_dst,write_metadata:0x30d54/0xfffffe,goto_table:27<br/> cookie=0x8000004, duration=38.704s, table=27, n_packets=3, n_bytes=294, priority=10,ip,metadata=0x30d54/0xfffffe,nw_dst=41.0.0.6 actions=resubmit(,21)<br/> cookie=0x8000003, duration=70.233s, table=21, n_packets=3, n_bytes=294, priority=42,ip,metadata=0x30d54/0xfffffe,nw_dst=41.0.0.6 actions=group:150007</p> <p>Response from VM Instance 1: (FIP Response)<br/> ---------------------------------------------------------------<br/> cookie=0x8000004, duration=38.704s, table=26, n_packets=3, n_bytes=294, priority=10,ip,metadata=0x30d54/0xfffffe,nw_src=41.0.0.6 actions=set_field:10.10.10.5-&gt;ip_src,write_metadata:0x30d46/0xfffffe,goto_table:28<br/> cookie=0x8000004, duration=38.704s, table=28, n_packets=3, n_bytes=294, priority=10,ip,metadata=0x30d46/0xfffffe,nw_src=10.10.10.5 actions=set_field:fa:16:3e:e9:c1:4e-&gt;eth_src,resubmit(,21)<br/> cookie=0x8000004, duration=87.466s, table=21, n_packets=3, n_bytes=294, priority=10,ip,metadata=0x30d46/0xfffffe actions=group:225002</p> <p>group_id=225002,duration=88.877s,ref_count=1,packet_count=3,byte_count=294,bucket0:packet_count=3,byte_count=294</p> <p>As per the local.conf and ml2.conf.ini configured information, CSIT job is running with Openstack DHCP service (q-dhcp) for allocating IP address to openstack compute VM instances.</p> <p> Also based on the compute-1 and compute-2 flows, we have not seen DHCP related flows (table 60 -&gt; Punt to Controller) been installed (If ODL DHCP is enabled). So, i requested someone from DHCP to have a look into openstack DHCP service (q-dhcp) issue for further analysis. Since there is no role of ODL-DHCP in this problem scenario, we don’t suspect any flow related issue is causing the problem for allocating IP address to the VM instances.</p> <p>Compute-1 DHCP flow: (Only default flow is existing)</p> <p>cookie=0x6800000, duration=2716.022s, table=60, n_packets=0, n_bytes=0, priority=0 actions=resubmit(,17)</p> <p>Ref link: <a href="https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-ocata-upstream-stateful-oxygen/524/log_03_external_network.html.gz#s1-t15-k3-k1-k2-k1-k15-k4" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-ocata-upstream-stateful-oxygen/524/log_03_external_network.html.gz#s1-t15-k3-k1-k2-k1-k15-k4</a></p> <p>Compute-2 DHCP flow: (Only default flow is existing)</p> <p>cookie=0x6800000, duration=2704.584s, table=60, n_packets=0, n_bytes=0, priority=0 actions=resubmit(,17)</p> <p>Ref link: <a href="https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-ocata-upstream-stateful-oxygen/524/log_03_external_network.html.gz#s1-t15-k3-k1-k3-k1-k15-k4" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-ocata-upstream-stateful-oxygen/524/log_03_external_network.html.gz#s1-t15-k3-k1-k3-k1-k15-k4</a></p> <p>some <a href="https://lists.opendaylight.org/pipermail/openflowplugin-dev/2018-January/007950.html" class="external-link" target="_blank" rel="nofollow noopener">discussion</a> is happening over email as well.</p> <p>Do we need to move this bug to the openflowplugin project?</p> <p>I took the liberty to re-assign this to Arun, as that is who is currently debugging<br/> things.</p> <p>Do we have ETA on resolution?  We need to consider unlocking the Carbon branch to allow other updates to come in.</p> <p> </p> <p>The tunnel port on one of the controller/compute node is getting missed in the inventory.</p> <p> From the logs, we can see the tunnel port addition was received at openflowplugin and the same has been submitted to inventory  by openflowplugin, but however there is an ERROR while submitting the transaction.</p> <p>This leads to the port being missed to add into the inventory. </p> <p>The exception shows the flow stats message is being added/removed from the inventory, but with stats disabled it is not expected to happen.</p> <p>Currently analysis further with few more logs enabled in openflowplugin stable/carbon branch. But the review is failing for unavailability of snapshot version in nexus.</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/67771/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/67771/</a></p> <p>Also, we are unable to reproduce the issue locally and has to be dependent on CSIT to fix the issue and it might take sometime.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=Arunprakash" class="user-hover" rel="Arunprakash">Arunprakash</a> I used the distro created from your <a href="http://example.com/" class="external-link" target="_blank" rel="nofollow noopener">debug patch</a> to run the job with</p> <p>TRACE logging and packet captures. Here are those logs:</p> <p>  <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/8/jamo-netvirt-csit-1node-openstack-ocata-upstream-stateful-carbon/16/" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/8/jamo-netvirt-csit-1node-openstack-ocata-upstream-stateful-carbon/16/</a></p> <p>Found out the root cause and raised new patch to address the same.</p> <p><span class="error">&#91;1&#93;</span>  <a href="https://git.opendaylight.org/gerrit/#/c/67801/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/67801/</a></p> <p>CSIT output <span class="error">&#91;2&#93;</span> with the patch <span class="error">&#91;1&#93;</span></p> <p><span class="error">&#91;2&#93;</span> <a href="https://jenkins.opendaylight.org/releng/job/netvirt-csit-1node-openstack-ocata-gate-stateful-carbon/266/console" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/job/netvirt-csit-1node-openstack-ocata-gate-stateful-carbon/266/console</a></p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jluhrsen" class="user-hover" rel="jluhrsen">jluhrsen</a>, please verify the csit results with patch <span class="error">&#91;1&#93;</span>.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=klou" class="user-hover" rel="klou">klou</a>, based on the confirmation from Jamo, we can merge the patch <span class="error">&#91;1&#93;</span>.</p> <p>Awesome <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=Arunprakash" class="user-hover" rel="Arunprakash">Arunprakash</a></p> <p>I'll be running the distro from your 67801 patch in the sandbox to get more runs in to build our confidence in<br/> the fix.</p> <p> </p> <p>Arun, good stuff!</p> <p>Job <span class="error">&#91;1&#93;</span> looked good, one tempest failure. Fired off <span class="error">&#91;2&#93;</span> and <span class="error">&#91;3&#93;</span> to get more coverage. But looking good.</p> <p><span class="error">&#91;1&#93;</span> <a href="https://jenkins.opendaylight.org/releng/job/netvirt-csit-1node-openstack-ocata-gate-stateful-carbon/267" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/job/netvirt-csit-1node-openstack-ocata-gate-stateful-carbon/26</a>6</p> <p><span class="error">&#91;2&#93;</span> <a href="https://jenkins.opendaylight.org/releng/job/openflowplugin-patch-test-netvirt-carbon/48/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/job/openflowplugin-patch-test-netvirt-carbon/48/</a></p> <p><span class="error">&#91;3&#93;</span> <a href="https://jenkins.opendaylight.org/releng/job/netvirt-csit-1node-openstack-ocata-gate-stateful-carbon/267" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/job/netvirt-csit-1node-openstack-ocata-gate-stateful-carbon/267</a></p> <p> </p> <p> </p> <p>There are still tons of table 50 flow exceptions is latest csit:</p> <p><a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-ocata-upstream-stateful-carbon/316/odl_1/odl1_exception.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-ocata-upstream-stateful-carbon/316/odl_1/odl1_exception.log.gz</a></p> <p>But looks like the merge for <a href="https://git.opendaylight.org/gerrit/#/c/67801/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/67801/</a> did not happen, so need to get that merge in.</p> <p>Hi Sam,</p> <p>Deletion logic for flow from operational inventory on flow removed message doesn’t exist in nitrogen and oxygen, so we don’t need this review in nitrogen and oxygen.</p> <p>Regards,</p> <p>Arun</p> <p><b>From:</b> Sam Hague <span class="nobr"><a href="mailto:shague@redhat.com]" class="external-link" target="_blank" rel="nofollow noopener">mailto:[shague@redhat.com^{<img class="rendericon" src="https://jira.opendaylight.org/images/icons/mail_small.gif" height="12" width="13" align="absmiddle" alt="" border="0"/>}</a></span> <br/> <b>Sent:</b> Wednesday, February 07, 2018 11:02 PM<br/> <b>To:</b> D Arunprakash &lt;<span class="nobr"><a href="mailto:d.arunprakash@ericsson.com" class="external-link" target="_blank" rel="nofollow noopener">d.arunprakash@ericsson.com^{<img class="rendericon" src="https://jira.opendaylight.org/images/icons/mail_small.gif" height="12" width="13" align="absmiddle" alt="" border="0"/>}</a></span>&gt;; Anil Vishnoi &lt;<span class="nobr"><a href="mailto:vishnoianil@gmail.com" class="external-link" target="_blank" rel="nofollow noopener">vishnoianil@gmail.com^{<img class="rendericon" src="https://jira.opendaylight.org/images/icons/mail_small.gif" height="12" width="13" align="absmiddle" alt="" border="0"/>}</a></span>&gt;; Jamo Luhrsen &lt;<span class="nobr"><a href="mailto:jluhrsen@redhat.com" class="external-link" target="_blank" rel="nofollow noopener">jluhrsen@redhat.com^{<img class="rendericon" src="https://jira.opendaylight.org/images/icons/mail_small.gif" height="12" width="13" align="absmiddle" alt="" border="0"/>}</a></span>&gt;; openflowplugin-dev &lt;<span class="nobr"><a href="mailto:openflowplugin-dev@lists.opendaylight.org" class="external-link" target="_blank" rel="nofollow noopener">openflowplugin-dev@lists.opendaylight.org^{<img class="rendericon" src="https://jira.opendaylight.org/images/icons/mail_small.gif" height="12" width="13" align="absmiddle" alt="" border="0"/>}</a></span>&gt;; odl netvirt dev &lt;<span class="nobr"><a href="mailto:netvirt-dev@lists.opendaylight.org" class="external-link" target="_blank" rel="nofollow noopener">netvirt-dev@lists.opendaylight.org^{<img class="rendericon" src="https://jira.opendaylight.org/images/icons/mail_small.gif" height="12" width="13" align="absmiddle" alt="" border="0"/>}</a></span>&gt;<br/> <b>Subject:</b> <a href="https://git.opendaylight.org/gerrit/#/c/67801/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/67801/</a> - why is it not a problem in nitrogen and oxygen?</p> <p>Arun, Anil,</p> <p>the patch below was merged to carbon. Are we good that it is not needed on nitrogen or oxygen?</p> <p>Thanks, Sam</p> <p>Can we mark this issue as resolved now?  Thanks!</p> Relates NETVIRT-1038 Development Rank 0|i038d3: