Can you please provide exact reproduction steps?
Hi Tom,
This is the controller csit clustering job.
The job stops ODL on all nodes, with a kill -9 on karaf pid.
Then, the job starts ODL on all nodes, with ../bin/start.
When ODL starts, we see the exception, and restconf fails with error 500 from then on.
Here is a link to the job with the exception. This will make it a little easier to see all the karaf logs, etc.
I think there's more detail in there. You mentioned changing tell-based setting - I assume that happens after kill -9. Does it delete the data dir before restarting? Is this reproducible every time or intermittent? Is it reproducible with single node?
I think I see the problem - I see both restconf-nb-bierman02-noauth and restconf-nb-bierman02-auth features being installed. It should be one or the other although we should put in a guard to at least avoid the ISE and emit a warning in Bierman02WebRegistrarImpl if the web context has already been created.
Great, if you put a link to the patch when you have it ready, I should be able to run it through csit to see how it looks.
Well CSIT should not install both features 
so I'd suggest to fix that to unblock. Seeing that it's a mis-configuration then I think we can lower severity/priority.
featuresBoot = odl-integration-compatible-with-all,odl-jolokia,odl-restconf-noauth,odl-clustering-test-app, 1a22a137-5311-4fe4-b894-bff5147a7819
where is the misconfig? maybe int/dist has something wrong with the compatible-with-all? otherwise, the misconfig is in the cluster-test-app or restconf-noauth or odl-jolokia features.
I suspect odl-integration-compatible-with-all installs odl-restconf. It's not wrong - it's just that installing odl-restconf-noauth in addition causes the issue. Like I said, I'll push a patch to guard against it but you'll get non-deterministic behavior across runs, ie one run uses auth, the next doesn't. Maybe that's why we're still seeing time outs - we think it's using noauth but it's really not. It looks like we need the equivalent of odl-integration-compatible-with-all that installs noauth. Or take restconf out of odl-integration-compatible-with-all and install the desired version separately.
Patch to guard against multiple registrations: https://git.opendaylight.org/gerrit/#/c/73488/
I assume odl-integration-compatible-with-all installs the "world" (at least the managed world). If so and this is a controller CSIT why are we installing all that? All you should need odl-clustering-test-app, odl-jolokia and odl-restconf(-noauth).
Thanks Tom. I'm just queued a csit job that removes the odl-netconf-noauth from the features list.
The odl-integration-compatible-with-all feature is installed as part of clustering config, I'll have to dig some more or get input from Jamo on how best to adjust that one.
ok, so we got a good theory here, and I know vpickard is testing this in the sandbox now, but running the job without
the noauth feature being installed. The protection patch c/73488 seems like a good idea as well. Need it cherry picked
to oxygen too, I guess.
We can do that but that patch really shouldn't be needed for CSIT - as I've mentioned you don't want to install both auth and no-auth - it will be non-deterministic which one will take effect on each run. You should be able to just remove odl-integration-compatible-with-all as I've mentioned. odl-clustering-test-app installs mdsal etc - that's all I install, along with restconf, when I'm testing with the cars stuff.
the point of compatible-with-all is to load all the features we have determined should be able to
run together without breaking things.
I wasn't implying the patch is needed for CSIT. If this is really what's happening, the patch to protect
/restconf from being totally dead is nice to have. We will now also know that putting noauth on the
same job as running compatible-with-all is wrong.
btw, I don't understand how this explains the sporadic nature of this problem. If compatible-with-all
is bringing in -auth and we are also always trying to install -noauth, then why doesn't the issue show
up every time?
It repro'ed every time I installed both - same exact error - so the theory is correct - based on how pax web works it is deterministic wrt failure. Are you sure you're really seeing it sporadically? Has there ever been a case where you saw it occur and didn't install both.
I thought compatible-with-all was for dist-check jobs... isn't necessary for CSIT?
ok, good that it's easy to reproduce. vpickard, I assumed this was one of the sporadic failures. Is this happening
every time? Seems that the end result of restconf being totally dead would make every test case fail.
compatible-with-all is what is used in every CSIT job you see with all in it. The jobs with only in it do not
install that.
Maybe it's a legacy thing. I mean if a CSIT is testing clustering with cars, why install eg lispflowmapping, sfc et al? What if one of those unused features has an orthogonal issue that affects the clustering tests?
I took a closer look at the logs for this CSIT run. ODL is stopped and started a number of times in the CSIT.
From what I see, one of the features will end up successfully registering, and the other one will fail. And it isn't the same one that wins every time.
I think we are/should be using the bierman02-auth version, right? So, if that is the one that fails to register, then 500 from then on until ODL is restarted in next test case down the line.
Right - that's what I mentioned before - it's non-deterministic which one gets in first. We should be using auth. In fact no-auth really should not even exist anymore - it was kept for legacy from the days before we even had authentication. Devs wanted to keep it for convenience (including me ). You may notice the new rfc8040 restconf feature no longer has a no-auth option - I removed it recently.
That said, the reason for using no-auth now in CSIT was to try to isolate the restconf failures in https://jira.opendaylight.org/browse/CONTROLLER-1838. Which then led to this issue b/c CSIT was installing both which is a no-no.
I'll lower the priority.
I think that's the point, to find (and fix) the orthogonal issues that come up. If it's deemed some kind of
unresolvable conflict, then we remove the feature from the compatible-with-all list.
we used to have both only and all jobs, so we could more easily know if the problems were
coming like this, but we were overloading jenkins, so where we could we removed the only
jobs. this is one such case.
So, it's non-deterministic, but tpantelis can hit it every time which sounds deterministic
maybe the CSIT timing is random enough that it doesn't happen every time...
Also, tpantelis this is not the same job or same feature set from CONTROLLER-1838. let's
not tie them together.
vpickard, once we known tihs never comes once we don't have -noauth, we can close this
bug, I think.
tpantelis, I wonder if we should hide this -noauth feature somehow, since it's strictly a
dev tool?
Maybe I'm not understanding something here... but I thought it's the purpose of the distribution-check jobs which run for every gerrit patch to test feature compatibility across all the projects. Hence why compatible-with-all was created - distribution-check jobs install and verify all the features come up.
If all that is correct (which I think it is), why do project CSITs need to duplicate what distribution-check jobs already do?
no, that's not correct. compatible-with-all was there from the start and used in CSIT. distribution-check came
later and took advantage of it. And, distribution-check just loads features. CSIT executes functional/system
level tests against those features.
example, l2switch, if loaded along side of netvirt will break things. l2switch is not compatible with other
openflow-y applications. However, l2switch will load just fine with along side netvirt. Thus, we ended
up creating those buckets (compatible or not) so we can make sure we can keep track of these kinds of
things.
anyway, none of this really matters as it applies to this jira.
this jira is about us investigating and figuring out (we hope) that we should not be using -noauth
(anywhere, I guess) in our CSIT jobs.
I mean it's non-deterministic which feature will install and which will fail, ie first one in wins. The failure (w/o my patch) is deterministic.
Not sure how you hide it. It might not just be a dev tool - it's been released for many versions so someone else may be using it out there so we technically can't/shouldn't just remove it from the release w/o going thru the deprecation/EOL cycle. That process can certainly be started if someone wants to drive it...
Here is the patch that removed odl-netconf-noauth from the clustering csit job. Patch has been merged.