is anyone looking at this?

here's another example in our Fluorine SR1 release candidate:

jluhrsen : Will take a look at this today.

jluhrsen, shague : Could you please check if the failing job link above is proper? It shows a passing job.

And I do see total 6 tunnels across control, compute-1 and compute-2 in the link.

And I assume that you are talking about br-int bridge on OVS. Please note that Genius ITM is managing only tunnels which are created with "tun" prefix. If you are talking about "br-datacentre" tunnels, it is not even connected to controller via openflow, I don't know who creates those tunnels

jluhrsen : Looking at https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-itm-direct-tunnels-fluorine/119/robot-plugin/log_full.html.gz validate Deployment failed with 404 Error :

${output} = 2018-11-08 18:35:27,926 | ERR | common.rest_client | 0052 | 404 Client Error: Not Found for url: http://10.30.170.77:8181/restconf/config/ietf-interfaces:interfaces

This says URL not found, are we checking this once ODL and restconf is fully UP, or anything wrong in the script?

Yes, the services are up, at least whatever the diagstatus check is looking for - they are checked just before the tunnels are checked: https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-itm-direct-tunnels-fluorine/119/robot-plugin/log_full.html.gz#s1-s1-k1-k1-k9-k2-k1-k1-k1-k1-k1-k1-k1-k1-k1-k1-k4

But this last link is for the itm-direct-tunnels case - which it does look like the odltools check isn't right. I will push a patch to ignore the check for itm-direct-tunnels or use a different method for it.

The original job linked in the description does not have the issue described, so I think we have the wrong link there. We will look for a better link.

What about the link to the failing job I added yesterday? Is that not this bug?

jluhrsen : That was an itm-direct-tunnels job, where the validate tunnels script has to be modified, as the datastore URL being checked is not valid for ITM direct tunnels.

No, the itm-direct-tunnels csit is checking for tunnels using odltools - but odltools does not support that check today, so the validate check there is always failing regardless of the tunnel state. Further down you can see that the tunnels are there.

We need to find a non itm-direct-tunnels CSIT run that has the issue. The one linked in the description does not have the issue so I think we posted the wrong link there. I looked in the recent csits briefly and did not find a tunnel failure job so far.

Also need to tweak the validate for the itm-direct-tunnels csit to use a different validation or get Vishal's changes in to add the support to odltools. For now, I think I will just pass the check when itm-direct-tunnels are configured so we don't falsely fail the suite setup.

how is itm-direct-tunnels passing sometimes? shouldn't it always be blowing up like the link I posted?

how about this guy ?

jluhrsen, shague : The last link given by Jamo has tunnel as "DOWN". But would have been nice if we had the TearDown Dumps for this step, as I cannot easily debug without knowing whether tunnel was present there on the switch or not.

jluhrsen : I cannot even find karaf logs in the folder in this case. If validation step fails, those will not be captured?

I chatted with Faseela on IRC, but for the record, test teardown debugs and karaf logs

there was a bug in the validate code that was letting some pass - remember the FAIL or False thing? Previously the check was just "${status}" == "FAIL", but this odltools was returning False so it was validated. I changed that recently to check both FAIL and False.

More details of the discussion at https://lists.opendaylight.org/pipermail/genius-dev/2018-November/003429.html

This seems to be not a GENIUS bug. jluhrsen/ shague : Move to appropriate project that handles the script probably?

I don't believe it yet

I've replied on that email thread, but we can discuss things here too. What should we do next
to get to the bottom of this.

BTW, there is not an easy way yet to enable TRACE debugs with a jenkins parameter
since this is an apex job. We need to account for that somehow. I guess I can add the
debugging with a csit patch, if we need it.

jluhrsen : From the current INFO logs itself, it is clear that the tunnel deletion is not triggered from GENIUS.

thapar : Do you have any clue on the auto-bridge creation logic, that can go wrong here? If so, should we move the Jira to Netvirt?

Just an FYI that, currently I am not doing anything for this issue.

jluhrsen TRACES for org.opendaylight.genius.interfacemanager & org.opendaylight.genius.itm (along with whatever thapar is going to ask for)

TRACE logs for org.opendaylight.netvirt.elan.internal.ElanBridgeManager

ITM may not be needed if interfaces present in IFM config. Too much logging will cause problems. Would prefer if go with DEBUG for interfacemanager than TRACE.

Asked for ITM, as I was wondering whether auto tunnels code along with itm direct tunnels somehow kicks in and delete tunnels on the switch[ofcourse this is not itm direct tunnels job that failed, but what if the check has gone wrong somewhere]

interfacemanager has enough info logs, and none of them are showing up in this delete case, so may be interfacemanager logs are not needed.

Could be if we're switching between the two but I think we're not.