https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/GENIUS-254 genius <p>In approximately 1 in 30 jobs, the suite setups is failing because the odltools check<br/> to verify default tunnels are up is not passing.</p> <p>example output:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> 2018-12-09 08:33:29,241 | ERR | common.rest_client | 0052 | 404 Client Error: Not Found <span class="code-keyword">for</span> url: http:<span class="code-comment">//192.0.2.19:8081/restconf/config/itm-state:dpn-teps-state </span>Analysing transport-zone:<span class="code-keyword">default</span>-transport-zone ..TerminationPoint tun9c586aac495 between 192.0.2.19 and 192.0.2.5 missing from config ..Interface tuna642e00a5c1 is down between 192.0.2.5 and 192.0.2.19 </pre> </div></div> <p>Examples:<br/> <a href="https://logs.opendaylight.org/sandbox/vex-yul-odl-jenkins-2/jamo-netvirt-csit-1node-0cmb-1ctl-2cmp-apex-queens-gate-snat-conntrack-oxygen/27/robot-plugin/log_full.html" class="external-link" target="_blank" rel="nofollow noopener">one</a></p> <p><a href="https://logs.opendaylight.org/sandbox/vex-yul-odl-jenkins-2/jamo-netvirt-csit-1node-0cmb-1ctl-2cmp-apex-queens-gate-snat-conntrack-oxygen/48/robot-plugin/log_full.html" class="external-link" target="_blank" rel="nofollow noopener">two </a> </p> <p><a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/554/jamo-netvirt-csit-1node-0cmb-1ctl-2cmp-apex-queens-gate-snat-conntrack-neon/31/robot-plugin/log_full.html" class="external-link" target="_blank" rel="nofollow noopener">three </a></p> <p><span class="error">&#91;Note that jobs 1 and 2 will be purged in a week, but infra is stopping me from copying logs to the logs server at the moment&#93;</span></p> GENIUS-254

default tunnels are down and/or missing from config

Bug Medium Verified Done nidhi adhvaryu Jamo Luhrsen apex:gate csit:failures Mon, 10 Dec 2018 18:05:55 +0000 Thu, 13 Jun 2019 04:45:24 +0000 Thu, 13 Jun 2019 04:45:13 +0000 Sodium General 0 4 0 minutes 1 day <p><a href="https://lists.opendaylight.org/pipermail/genius-dev/2018-December/003504.html" class="external-link" target="_blank" rel="nofollow noopener">email thread </a> discussing this problem</p> <p><a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/559/jamo-netvirt-csit-1node-0cmb-1ctl-2cmp-apex-queens-gate-snat-conntrack-oxygen/74/robot-plugin/log_full.html" class="external-link" target="_blank" rel="nofollow noopener">another example </a></p> <p>Since this is NETVIRT Issue, will be fixed as <a href="https://jira.opendaylight.org/browse/NETVIRT-1541" class="external-link" rel="nofollow">https://jira.opendaylight.org/browse/NETVIRT-1541</a></p> <p>Saw a different variation of the issue with bug in genius</p> <p>While testing <a href="https://jira.opendaylight.org/browse/NETVIRT-1541" title=" Default Tunnels deleted from config" class="issue-link" data-issue-key="NETVIRT-1541"><del>NETVIRT-1541</del></a> fixes, ran into another variation of this bug in d/s testing, this time in Genius.</p> <p> tunea6b6a44e1b is not created. This is what sequence of log entries looks like:</p> <p>2019-01-11T07:46:30,224 | INFO | nioEventLoopGroup-9-9 | opendaylight.openflowplugin.impl.device.initialization.OF13DeviceInitializer | Static node openflow:9444498845632 info: OFPMPPORTDESC collected<br/> 2019-01-11T07:46:30,226 | DEBUG | ForkJoinPool-1-worker-5 | org.opendaylight.genius.interfacemanager.listeners.InterfaceTopologyStateListener | Bridge entry not found in config DS for dpn: 154706290356175<br/> 2019-01-11T07:46:30,226 | INFO | ForkJoinPool-1-worker-5 | genius.interfacemanager.renderer.ovs.confighelpers.OvsInterfaceConfigAddHelper | adding tunnel configuration for interface tunea6b6a44e1b<br/> 2019-01-11T07:46:30,227 | DEBUG | ForkJoinPool-1-worker-5 | genius.interfacemanager.renderer.ovs.confighelpers.OvsInterfaceConfigAddHelper | creating bridge interfaceEntry in ConfigDS 154706290356175</p> <p>No further references to this tunnel after this.</p> <p>This is what they look like for a tunnel that gets created:</p> <p>2019-01-11T07:46:30,208 | INFO | ForkJoinPool-1-worker-2 | genius.interfacemanager.renderer.ovs.confighelpers.OvsInterfaceConfigAddHelper | adding tunnel configuration for interface tun4fab841ac4e<br/> 2019-01-11T07:46:30,208 | DEBUG | ForkJoinPool-1-worker-2 | genius.interfacemanager.renderer.ovs.confighelpers.OvsInterfaceConfigAddHelper | creating bridge interfaceEntry in ConfigDS 9444498845632<br/> <font color="#14892c">2019-01-11T07:46:30,209 | DEBUG | ForkJoinPool-1-worker-2 | genius.interfacemanager.renderer.ovs.confighelpers.OvsInterfaceConfigAddHelper | creating bridge interface on dpn 9444498845632</font><br/> 2019-01-11T07:46:30,209 | DEBUG | ForkJoinPool-1-worker-2 | genius.interfacemanager.renderer.ovs.utilities.SouthboundUtils | adding tunnel port tun4fab841ac4e to bridge InstanceIdentifier</p> <p>The highlighted log entry in second is missing from first.</p> <p>Looking at the code, following check is failing</p> <p> LOG.debug("creating bridge interfaceEntry in ConfigDS {}", dpId);<br/> interfaceMetaUtils.createBridgeInterfaceEntryInConfigDS(dpId, interfaceNew.getName());</p> <p> // create bridge on switch, if switch is connected<br/> BridgeRefEntry bridgeRefEntry = interfaceMetaUtils.getBridgeRefEntryFromOperDS(dpId);<br/> <font color="#d04437"> if (bridgeRefEntry != null &amp;&amp; bridgeRefEntry.getBridgeReference() != null) {</font><br/> LOG.debug("creating bridge interface on dpn {}", dpId);</p> <p>This means bridgeRefEntry is missing, even though it was created couple log entries before the one above:</p> <p>2019-01-11T07:46:30,216 | DEBUG | ForkJoinPool-1-worker-5 | org.opendaylight.genius.interfacemanager.commons.InterfaceMetaUtils | Creating bridge ref entry for dpn: 154706290356175 bridge: </p> <p>For reference, this is log entry for bridgeRefEntry for the working case:</p> <p>2019-01-11T07:46:29,351 | DEBUG | ForkJoinPool-1-worker-4 | org.opendaylight.genius.interfacemanager.commons.InterfaceMetaUtils | Creating bridge ref entry for dpn: 9444498845632 bridge: </p> <p>In working there is a gap of about 1s between two. In failure case it is less than 10ms.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=k.faseela" class="user-hover" rel="k.faseela">k.faseela</a> Issue is tunnel missing from OVSDB config. From log entries I shared you can see ITM never pushes the tunnel to OVSDB config and have pointed out what I think is the race condition. Any suggestions on fix?</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=thapar" class="user-hover" rel="thapar">thapar</a> : Got the issue, let me think about an easy way to fix this.</p> <p>great <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/> what's the TL;DR on the root cause?</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jluhrsen" class="user-hover" rel="jluhrsen">jluhrsen</a> Race condition. Genius stores BridgeRefEntry that maps a bridge to its corresponding DPNID. It gets populated when bridge is created or switch first connects, whichever is latter. Tunnels creation trigger is also same but different part of code. Due to extra steps in tunnel creation, we mostly have this entry in place before we get to point where we try creating tunnel port. But in this case by the time we read it is not present yet. It might have something to do with changes done as part of tx runner that we're hitting this race more frequently.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=thapar" class="user-hover" rel="thapar">thapar</a> : Here is an overall idea of the fix I want to try out.</p> <p>                         FYI, not tested yet.</p> <p>                          <a href="https://git.opendaylight.org/gerrit/79553" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/79553</a></p> <p> </p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=thapar" class="user-hover" rel="thapar">thapar</a>/<a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jluhrsen" class="user-hover" rel="jluhrsen">jluhrsen</a> : Is it possible to run the patch against netvirt CSIT couple of times to see if the issue is not reproducible now? Have done local testing to make sure the patch is not breaking anything, but couldn't really hit the race condition timings to see if that problem is solved.</p> <p>this gerrit keyword should trigger the 3node job: "test-genius-cluster-netvirt"<br/> I triggered it once just now.</p> <p>For ITM also we have added a similar fix.</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/81980/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/81980/</a></p> <p>The fix for this issue has been merged here, <a href="https://git.opendaylight.org/gerrit/#/c/81980/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/81980/</a> for ITM.</p> Development Rank 0|i03ldz: