https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/NETVIRT-873 netvirt <p><a href="https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-3node-openstack-ocata-upstream-stateful-nitrogen/148/log.html.gz#s1-s1-s3-t13" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-3node-openstack-ocata-upstream-stateful-nitrogen/148/log.html.gz#s1-s1-s3-t13</a></p> <p>These failures are not there in our 1node (non-HA) job, but are the<br/> same tests. It should also be noted that these failures are happening<br/> before any HA related activities (e.g., taking a controller in/out of<br/> the cluster) are done.</p> <p>My first suggestion would be to compare flow table entries of the<br/> control/compute nodes in this job vs a passing example in the 1node<br/> job:</p> <p><a href="https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-ocata-upstream-stateful-nitrogen/152/" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-ocata-upstream-stateful-nitrogen/152/</a></p> <p>Operating System: All<br/> Platform: All</p> NETVIRT-873

CSIT 3node failures - External Network Suite - FIP connectivity failures

Bug Medium Resolved Won't Do Chetan Arakere Gowdru Jamo Luhrsen csit:failures Tue, 29 Aug 2017 23:07:20 +0000 Tue, 23 Oct 2018 16:22:33 +0000 Tue, 23 Oct 2018 16:22:33 +0000 Nitrogen General 0 3 <p>Compared Build #148 (Failed FIP 4 TCs) and Build #149 (Passed FIP 4 TCs) and below are the observations.</p> <p>Build #148:<br/> ===========</p> <p>Table 25-&gt;27<br/> cookie=0x8000004, duration=101.387s, table=25, n_packets=0, n_bytes=0, priority=10,ip,dl_dst=fa:16:3e:b3:81:8f,nw_dst=10.10.10.10 actions=set_field:90.0.0.6-&gt;ip_dst,write_metadata:0x324c0/0xfffffe,goto_table:27</p> <p>Table 21-&gt;25 (Here instead of pointing to table 25, it is pointing as remote tunnel out port). This is the root cause of this problem. Local FIP flow is missing here.</p> <p>cookie=0x8000003, duration=101.222s, table=21, n_packets=6, n_bytes=588, priority=42,ip,metadata=0x324b6/0xfffffe,nw_dst=10.10.10.10 actions=set_field:fa:16:3e:b3:81:8f-&gt;eth_dst,load:0x700-&gt;NXM_NX_REG6[],resubmit(,220)</p> <p>Log Reference Link:<br/> ==================<br/> <a href="https://jenkins.opendaylight.org/releng/user/jluhrsen/my-views/view/netvirt%20csit/job/netvirt-csit-3node-openstack-ocata-upstream-stateful-nitrogen/148/robot/report/log.html#s1-s1-s3-t15" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/user/jluhrsen/my-views/view/netvirt%20csit/job/netvirt-csit-3node-openstack-ocata-upstream-stateful-nitrogen/148/robot/report/log.html#s1-s1-s3-t15</a></p> <p>Build #149<br/> ==========</p> <p>Table 21-&gt;25 (Local FIP entry properly got installed)<br/> cookie=0x8000003, duration=65.479s, table=21, n_packets=6, n_bytes=588, priority=42,ip,metadata=0x30d44/0xfffffe,nw_dst=10.10.10.13 actions=set_field:fa:16:3e:b3:95:35-&gt;eth_dst,goto_table:25</p> <p>Table 25-&gt;27<br/> cookie=0x8000004, duration=65.511s, table=25, n_packets=6, n_bytes=588, priority=10,ip,dl_dst=fa:16:3e:b3:95:35,nw_dst=10.10.10.13 actions=set_field:90.0.0.8-&gt;ip_dst,write_metadata:0x30d4e/0xfffffe,goto_table:27</p> <p>Log Reference Link:<br/> =======================<br/> <a href="https://jenkins.opendaylight.org/releng/user/jluhrsen/my-views/view/netvirt%20csit/job/netvirt-csit-3node-openstack-ocata-upstream-stateful-nitrogen/149/robot/report/log.html#s1-s1-s3-t15" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/user/jluhrsen/my-views/view/netvirt%20csit/job/netvirt-csit-3node-openstack-ocata-upstream-stateful-nitrogen/149/robot/report/log.html#s1-s1-s3-t15</a></p> <p>We will observe next upcoming CSIT 3 node job for confirming the issue consistence.</p> <p>I see some of these that might be related:</p> <p>2017-09-01 17:23:01,300 | INFO | 2]-nio2-thread-5 | ServerSession | 130 - org.apache.sshd.core - 0.14.0 | Server session created from /10.29.4.11:52656<br/> 2017-09-01 17:23:01,356 | WARN | 2]-nio2-thread-3 | ServerSession | 130 - org.apache.sshd.core - 0.14.0 | Exception caught<br/> java.io.IOException: Connection reset by peer<br/> at sun.nio.ch.FileDispatcherImpl.read0(Native Method)<span class="error">&#91;:1.8.0_141&#93;</span><br/> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)<span class="error">&#91;:1.8.0_141&#93;</span><br/> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)<span class="error">&#91;:1.8.0_141&#93;</span><br/> at sun.nio.ch.IOUtil.read(IOUtil.java:197)<span class="error">&#91;:1.8.0_141&#93;</span><br/> at sun.nio.ch.UnixAsynchronousSocketChannelImpl.finishRead(UnixAsynchronousSocketChannelImpl.java:387)<span class="error">&#91;:1.8.0_141&#93;</span><br/> at sun.nio.ch.UnixAsynchronousSocketChannelImpl.finish(UnixAsynchronousSocketChannelImpl.java:191)<span class="error">&#91;:1.8.0_141&#93;</span><br/> at sun.nio.ch.UnixAsynchronousSocketChannelImpl.onEvent(UnixAsynchronousSocketChannelImpl.java:213)<span class="error">&#91;:1.8.0_141&#93;</span><br/> at sun.nio.ch.EPollPort$EventHandlerTask.run(EPollPort.java:293)<span class="error">&#91;:1.8.0_141&#93;</span><br/> at java.lang.Thread.run(Thread.java:748)<span class="error">&#91;:1.8.0_141&#93;</span></p> <p>Have added few info logs in Fibmanager-Impl and raised the below review in stable/nitrogen and ran 3-node cluster CSIT job for 4 times and looks like issue is not been observed after info logs been added into the code. Still trying to identify the RCA of this issue. Will post the result once identified the RCA of this issue. </p> <p><a href="https://git.opendaylight.org/gerrit/#/c/63081/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/63081/</a></p> <p>Stable/Nitrogen 3 Node Cluster Job:<br/> ------------------------------------<br/> <a href="https://jenkins.opendaylight.org/releng/job/netvirt-csit-3node-openstack-ocata-gate-stateful-nitrogen/6/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/job/netvirt-csit-3node-openstack-ocata-gate-stateful-nitrogen/6/</a></p> <p>Note: 3-Node cluster CSIT job on Master (Oxygen) --&gt; Issue is not re-producible.</p> <p>Master (Oxygen) 3 Node Cluster Job:<br/> ------------------------------------<br/> <a href="https://jenkins.opendaylight.org/releng/job/netvirt-csit-3node-openstack-ocata-gate-stateful-oxygen/1/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/job/netvirt-csit-3node-openstack-ocata-gate-stateful-oxygen/1/</a></p> <p>Bug closed and others will be opened after the recent clustering efforts make it easier to focus on different issues.</p> Development External issue ID 9074 External issue URL Rank 0|i01uhj: Status Whiteboard csit:sporadic_failures