https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/OVSDB-472 ovsdb <p>sporadically in u/s CSIT jobs we are getting failures where instances are not able to<br/> get their ip address. It looks like it might be that not all vxlan tunnels are getting<br/> created.</p> <p>in <a href="https://logs.opendaylight.org/sandbox/vex-yul-odl-jenkins-2/jamo-netvirt-csit-1node-0cmb-1ctl-2cmp-apex-queens-gate-snat-conntrack-oxygen/1/robot-plugin/log_full.html.gz#s1-s1-k2-k1-k1-k2-k1-k3-k1-k9-k4" class="external-link" target="_blank" rel="nofollow noopener">this failure</a> it seems that compute #2 does not<br/> have the tunnel to the control node. but, note that it does exist on the control node.<br/> somewhere this creation is failing.</p> <p><a href="https://logs.opendaylight.org/sandbox/vex-yul-odl-jenkins-2/jamo-netvirt-csit-1node-0cmb-1ctl-2cmp-apex-queens-gate-snat-conntrack-oxygen/1" class="external-link" target="_blank" rel="nofollow noopener">all logs under this directory</a></p> <p><a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/490/jamo-netvirt-csit-1node-0cmb-1ctl-2cmp-apex-queens-gate-snat-conntrack-oxygen/2" class="external-link" target="_blank" rel="nofollow noopener">an identical passing job for comparison</a></p> OVSDB-472

not all tunnels getting created

Bug Highest Resolved Done Tim Rozet Jamo Luhrsen csit:failures Tue, 30 Oct 2018 20:58:50 +0000 Wed, 28 Nov 2018 21:05:41 +0000 Wed, 28 Nov 2018 21:05:41 +0000 Oxygen-SR4 Fluorine-SR2 Neon Southbound.Open_vSwitch 0 5 <p>is anyone looking at this?</p> <p>here's <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-itm-direct-tunnels-fluorine/119/" class="external-link" target="_blank" rel="nofollow noopener">another example </a> in our Fluorine SR1 release candidate:</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jluhrsen" class="user-hover" rel="jluhrsen">jluhrsen</a> : Will take a look at this today.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jluhrsen" class="user-hover" rel="jluhrsen">jluhrsen</a>, <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=shague" class="user-hover" rel="shague">shague</a> : Could you please check if the failing job link above is proper? It shows a passing job.</p> <p>And I do see total 6 tunnels across control, compute-1 and compute-2 in the link.</p> <p>And I assume that you are talking about br-int bridge on OVS. Please note that Genius ITM is managing only tunnels which are created with "tun" prefix. If you are talking about "br-datacentre" tunnels, it is not even connected to controller via openflow, I don't know who creates those tunnels</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jluhrsen" class="user-hover" rel="jluhrsen">jluhrsen</a> : Looking at <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-itm-direct-tunnels-fluorine/119/robot-plugin/log_full.html.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-itm-direct-tunnels-fluorine/119/robot-plugin/log_full.html.gz</a> validate Deployment failed with 404 Error :</p> <p>${output} = 2018-11-08 18:35:27,926 | ERR | common.rest_client | 0052 | 404 Client Error: Not Found for url: <a href="http://10.30.170.77:8181/restconf/config/ietf-interfaces:interfaces" class="external-link" target="_blank" rel="nofollow noopener">http://10.30.170.77:8181/restconf/config/ietf-interfaces:interfaces</a></p> <p>This says URL not found, are we checking this once ODL and restconf is fully UP, or anything wrong in the script?</p> <p>Yes, the services are up, at least whatever the diagstatus check is looking for - they are checked just before the tunnels are checked: <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-itm-direct-tunnels-fluorine/119/robot-plugin/log_full.html.gz#s1-s1-k1-k1-k9-k2-k1-k1-k1-k1-k1-k1-k1-k1-k1-k1-k4" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-itm-direct-tunnels-fluorine/119/robot-plugin/log_full.html.gz#s1-s1-k1-k1-k9-k2-k1-k1-k1-k1-k1-k1-k1-k1-k1-k1-k4</a></p> <p>But this last link is for the itm-direct-tunnels case - which it does look like the odltools check isn't right. I will push a patch to ignore the check for itm-direct-tunnels or use a different method for it.</p> <p>The original job linked in the description does not have the issue described, so I think we have the wrong link there. We will look for a better link.</p> <p>What about the link to the failing job I added yesterday? Is that not this bug?</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jluhrsen" class="user-hover" rel="jluhrsen">jluhrsen</a> : That was an itm-direct-tunnels job, where the validate tunnels script has to be modified, as the datastore URL being checked is not valid for ITM direct tunnels.</p> <p>No, the itm-direct-tunnels csit is checking for tunnels using odltools - but odltools does not support that check today, so the validate check there is always failing regardless of the tunnel state. Further down you can see that the tunnels are there.</p> <p>We need to find a non itm-direct-tunnels CSIT run that has the issue. The one linked in the description does not have the issue so I think we posted the wrong link there. I looked in the recent csits briefly and did not find a tunnel failure job so far.</p> <p>Also need to tweak the validate for the itm-direct-tunnels csit to use a different validation or get Vishal's changes in to add the support to odltools. For now, I think I will just pass the check when itm-direct-tunnels are configured so we don't falsely fail the suite setup.</p> <p>how is itm-direct-tunnels passing sometimes? shouldn't it always be blowing up like the link I posted?</p> <p>how about <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/504/jamo-netvirt-csit-1node-0cmb-1ctl-2cmp-apex-queens-gate-snat-conntrack-oxygen/42/robot-plugin/log_full.html.gz#s1-s1-k1-k1-k9-k2-k2" class="external-link" target="_blank" rel="nofollow noopener">this guy</a> ?</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jluhrsen" class="user-hover" rel="jluhrsen">jluhrsen</a>, <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=shague" class="user-hover" rel="shague">shague</a> : The last link given by Jamo has tunnel as "DOWN". But would have been nice if we had the TearDown Dumps for this step, as I cannot easily debug without knowing whether tunnel was present there on the switch or not.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jluhrsen" class="user-hover" rel="jluhrsen">jluhrsen</a> : I cannot even find karaf logs in the folder in this case. If validation step fails, those will not be captured?</p> <p>I chatted with Faseela on IRC, but for the record, <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/504/jamo-netvirt-csit-1node-0cmb-1ctl-2cmp-apex-queens-gate-snat-conntrack-oxygen/42/robot-plugin/log_full.html.gz#s1-s1-k2-k1" class="external-link" target="_blank" rel="nofollow noopener">test teardown debugs</a> and <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/504/jamo-netvirt-csit-1node-0cmb-1ctl-2cmp-apex-queens-gate-snat-conntrack-oxygen/42/controller_sos/sosreport-controllerreport-20181108184901/opt/opendaylight/data/log/karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">karaf logs</a></p> <p>there was a bug in the validate code that was letting some pass - remember the FAIL or False thing? Previously the check was just "${status}" == "FAIL", but this odltools was returning False so it was validated. I changed that recently to check both FAIL and False.</p> <p>More details of the discussion at <a href="https://lists.opendaylight.org/pipermail/genius-dev/2018-November/003429.html" class="external-link" target="_blank" rel="nofollow noopener">https://lists.opendaylight.org/pipermail/genius-dev/2018-November/003429.html</a></p> <p>This seems to be not a GENIUS bug. <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jluhrsen" class="user-hover" rel="jluhrsen">jluhrsen</a>/ <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=shague" class="user-hover" rel="shague">shague</a> : Move to appropriate project that handles the script probably?</p> <p>I don't believe it yet <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>I've replied on that email thread, but we can discuss things here too. What should we do next<br/> to get to the bottom of this.</p> <p>BTW, there is not an easy way yet to enable TRACE debugs with a jenkins parameter<br/> since this is an apex job. We need to account for that somehow. I guess I can add the<br/> debugging with a csit patch, if we need it.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jluhrsen" class="user-hover" rel="jluhrsen">jluhrsen</a> : From the current INFO logs itself, it is clear that the tunnel deletion is not triggered from GENIUS.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=thapar" class="user-hover" rel="thapar">thapar</a> : Do you have any clue on the auto-bridge creation logic, that can go wrong here? If so, should we move the Jira to Netvirt?</p> <p>Just an FYI that, currently I am not doing anything for this issue.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jluhrsen" class="user-hover" rel="jluhrsen">jluhrsen</a> TRACES for org.opendaylight.genius.interfacemanager &amp; org.opendaylight.genius.itm (along with whatever <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=thapar" class="user-hover" rel="thapar">thapar</a> is going to ask for)</p> <p>TRACE logs for org.opendaylight.netvirt.elan.internal.ElanBridgeManager</p> <p>ITM may not be needed if interfaces present in IFM config. Too much logging will cause problems. Would prefer if go with DEBUG for interfacemanager than TRACE.</p> <p>Asked for ITM, as I was wondering whether auto tunnels code along with itm direct tunnels somehow kicks in and delete tunnels on the switch<span class="error">&#91;ofcourse this is not itm direct tunnels job that failed, but what if the check has gone wrong somewhere&#93;</span></p> <p>interfacemanager has enough info logs, and none of them are showing up in this delete case, so may be interfacemanager logs are not needed.</p> <p>Could be if we're switching between the two but I think we're not.</p> Development Priority Rank 0|i03k27: