https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-1844 controller <p>This bundle is failing to start in the controller clustering job <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/controller-csit-3node-clustering-all-fluorine/127/robot-plugin/log.html.gz#s1-s45" class="external-link" target="_blank" rel="nofollow noopener">127</a>, after ODL is restarted with Tell Based False.</p> <p> </p> <p>From then on, RPC failures occur with error 500.</p> <p> </p> <p> </p> <p>Snippet from odl1_karaf.log</p> <p>====================== </p> <p>2018-06-25T13:04:34,012 | ERROR | Blueprint Extender: 2 | BlueprintContainerImpl | 75 - org.apache.aries.blueprint.core - 1.8.3 | Unable to start blueprint container for bundle org.opendaylight.netconf.restconf-nb-bierman02-auth/1.8.0<br/> org.osgi.service.blueprint.container.ComponentDefinitionException: Error when instantiating bean .component-1 of class org.opendaylight.restconf.nb.bierman02.web.auth.WebInitializer<br/> at org.apache.aries.blueprint.container.BeanRecipe.wrapAsCompDefEx(BeanRecipe.java:361) <span class="error">&#91;75:org.apache.aries.blueprint.core:1.8.3&#93;</span><br/> at org.apache.aries.blueprint.container.BeanRecipe.getInstanceFromType(BeanRecipe.java:351) <span class="error">&#91;75:org.apache.aries.blueprint.core:1.8.3&#93;</span><br/> at org.apache.aries.blueprint.container.BeanRecipe.getInstance(BeanRecipe.java:282) <span class="error">&#91;75:org.apache.aries.blueprint.core:1.8.3&#93;</span><br/> at org.apache.aries.blueprint.container.BeanRecipe.internalCreate2(BeanRecipe.java:830) <span class="error">&#91;75:org.apache.aries.blueprint.core:1.8.3&#93;</span><br/> at org.apache.aries.blueprint.container.BeanRecipe.internalCreate(BeanRecipe.java:811) <span class="error">&#91;75:org.apache.aries.blueprint.core:1.8.3&#93;</span><br/> at org.apache.aries.blueprint.di.AbstractRecipe$1.call(AbstractRecipe.java:79) <span class="error">&#91;75:org.apache.aries.blueprint.core:1.8.3&#93;</span><br/> at java.util.concurrent.FutureTask.run(FutureTask.java:266) <span class="error">&#91;?:?&#93;</span><br/> at org.apache.aries.blueprint.di.AbstractRecipe.create(AbstractRecipe.java:88) <span class="error">&#91;75:org.apache.aries.blueprint.core:1.8.3&#93;</span><br/> at org.apache.aries.blueprint.container.BlueprintRepository.createInstances(BlueprintRepository.java:255) <span class="error">&#91;75:org.apache.aries.blueprint.core:1.8.3&#93;</span><br/> at org.apache.aries.blueprint.container.BlueprintRepository.createAll(BlueprintRepository.java:186) <span class="error">&#91;75:org.apache.aries.blueprint.core:1.8.3&#93;</span><br/> at org.apache.aries.blueprint.container.BlueprintContainerImpl.instantiateEagerComponents(BlueprintContainerImpl.java:704) <span class="error">&#91;75:org.apache.aries.blueprint.core:1.8.3&#93;</span><br/> at org.apache.aries.blueprint.container.BlueprintContainerImpl.doRun(BlueprintContainerImpl.java:410) <span class="error">&#91;75:org.apache.aries.blueprint.core:1.8.3&#93;</span><br/> at org.apache.aries.blueprint.container.BlueprintContainerImpl.run(BlueprintContainerImpl.java:275) <span class="error">&#91;75:org.apache.aries.blueprint.core:1.8.3&#93;</span><br/> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) <span class="error">&#91;?:?&#93;</span><br/> at java.util.concurrent.FutureTask.run(FutureTask.java:266) <span class="error">&#91;?:?&#93;</span><br/> at org.apache.aries.blueprint.container.ExecutorServiceWrapper.run(ExecutorServiceWrapper.java:106) <span class="error">&#91;75:org.apache.aries.blueprint.core:1.8.3&#93;</span><br/> at org.apache.aries.blueprint.utils.threading.impl.DiscardableRunnable.run(DiscardableRunnable.java:48) <span class="error">&#91;75:org.apache.aries.blueprint.core:1.8.3&#93;</span><br/> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) <span class="error">&#91;?:?&#93;</span><br/> at java.util.concurrent.FutureTask.run(FutureTask.java:266) <span class="error">&#91;?:?&#93;</span><br/> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) <span class="error">&#91;?:?&#93;</span><br/> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) <span class="error">&#91;?:?&#93;</span><br/> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) <span class="error">&#91;?:?&#93;</span><br/> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) <span class="error">&#91;?:?&#93;</span><br/> at java.lang.Thread.run(Thread.java:748) <span class="error">&#91;?:?&#93;</span><br/> Caused by: java.lang.IllegalStateException: Http context already used. Context params can be set/changed only before first usage<br/> at org.ops4j.pax.web.service.internal.HttpServiceStarted.setContextParam(HttpServiceStarted.java:707) ~<span class="error">&#91;?:?&#93;</span><br/> at org.ops4j.pax.web.service.internal.HttpServiceProxy.setContextParam(HttpServiceProxy.java:271) ~<span class="error">&#91;?:?&#93;</span><br/> at org.opendaylight.aaa.web.osgi.PaxWebServer$WebContextImpl.&lt;init&gt;(PaxWebServer.java:164) ~<span class="error">&#91;?:?&#93;</span><br/> at org.opendaylight.aaa.web.osgi.PaxWebServer$2$1.&lt;init&gt;(PaxWebServer.java:116) ~<span class="error">&#91;?:?&#93;</span><br/> at org.opendaylight.aaa.web.osgi.PaxWebServer$2.registerWebContext(PaxWebServer.java:116) ~<span class="error">&#91;?:?&#93;</span><br/> at Proxy61f92867_78e3_4dd2_a6c9_e40871b2c69a.registerWebContext(Unknown Source) ~<span class="error">&#91;?:?&#93;</span><br/> at org.opendaylight.netconf.sal.restconf.web.Bierman02WebRegistrarImpl.register(Bierman02WebRegistrarImpl.java:82) ~<span class="error">&#91;?:?&#93;</span><br/> at org.opendaylight.netconf.sal.restconf.web.Bierman02WebRegistrarImpl.registerWithAuthentication(Bierman02WebRegistrarImpl.java:55) ~<span class="error">&#91;?:?&#93;</span><br/> at Proxyb86b267b_05f2_4bbe_bd31_3106e600b567.registerWithAuthentication(Unknown Source) ~<span class="error">&#91;?:?&#93;</span><br/> at Proxyf7cb35fe_3ba9_43d0_a739_d4c847b9747b.registerWithAuthentication(Unknown Source) ~<span class="error">&#91;?:?&#93;</span><br/> at org.opendaylight.restconf.nb.bierman02.web.auth.WebInitializer.&lt;init&gt;(WebInitializer.java:19) ~<span class="error">&#91;?:?&#93;</span><br/> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~<span class="error">&#91;?:?&#93;</span><br/> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~<span class="error">&#91;?:?&#93;</span><br/> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~<span class="error">&#91;?:?&#93;</span><br/> at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~<span class="error">&#91;?:?&#93;</span><br/> at org.apache.aries.blueprint.utils.ReflectionUtils.newInstance(ReflectionUtils.java:331) ~<span class="error">&#91;?:?&#93;</span><br/> at org.apache.aries.blueprint.container.BeanRecipe.newInstance(BeanRecipe.java:984) ~<span class="error">&#91;?:?&#93;</span><br/> at org.apache.aries.blueprint.container.BeanRecipe.getInstanceFromType(BeanRecipe.java:349) ~<span class="error">&#91;?:?&#93;</span><br/> ... 22 more</p> CONTROLLER-1844

Unable to start blueprint container for bundle org.opendaylight.netconf.restconf-nb-bierman02-auth/1.8.0

Bug Medium Verified Done Tom Pantelis Victor Pickard csit:3node Tue, 26 Jun 2018 19:45:50 +0000 Thu, 28 Jun 2018 15:46:54 +0000 Thu, 28 Jun 2018 15:46:28 +0000 Fluorine clustering netconf restconf 0 5 <p>Can you please provide exact reproduction steps?</p> <p>Hi Tom,</p> <p>This is the controller csit clustering job. </p> <p>The job stops ODL on all nodes, with a kill -9 on karaf pid.</p> <p>Then, the job starts ODL on all nodes, with ../bin/start.</p> <p>When ODL starts, we see the exception, and restconf fails with error 500 from then on.</p> <p> </p> <p>Here is a link to the job with the exception. This will make it a little easier to see all the karaf logs, etc.</p> <p> </p> <p><a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/controller-csit-3node-clustering-all-fluorine/127/" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/controller-csit-3node-clustering-all-fluorine/127/</a></p> <p> </p> <p>I think there's more detail in there. You mentioned changing tell-based setting - I assume that happens after kill -9. Does it delete the data dir before restarting? Is this reproducible every time or intermittent? Is it reproducible with single node?</p> <p>I think I see the problem - I see both restconf-nb-bierman02-noauth and restconf-nb-bierman02-auth features being installed. It should be one or the other although we should put in a guard to at least avoid the ISE and emit a warning in Bierman02WebRegistrarImpl if  the web context has already been created.</p> <p>Great, if you put a link to the patch when you have it ready, I should be able to run it through csit to see how it looks. </p> <p> </p> <p>Well CSIT should not install both features <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/> so I'd suggest to fix that to unblock. Seeing that it's a mis-configuration then I think we can lower severity/priority.</p> <p>featuresBoot = odl-integration-compatible-with-all,odl-jolokia,odl-restconf-noauth,odl-clustering-test-app, 1a22a137-5311-4fe4-b894-bff5147a7819</p> <p>where is the misconfig? maybe int/dist has something wrong with the compatible-with-all? otherwise, the misconfig is in the cluster-test-app or restconf-noauth or odl-jolokia features.</p> <p>I suspect  odl-integration-compatible-with-all installs odl-restconf. It's not wrong - it's just that installing odl-restconf-noauth in addition causes the issue. Like I said, I'll push a patch to guard against it but you'll get non-deterministic behavior across runs, ie one run uses auth, the next doesn't. Maybe that's why we're still seeing time outs - we <b>think</b> it's using noauth but it's really not. It looks like we need the equivalent of odl-integration-compatible-with-all that installs noauth. Or take restconf out of odl-integration-compatible-with-all and install the desired version separately.</p> <p>Patch to guard against multiple registrations: <a href="https://git.opendaylight.org/gerrit/#/c/73488/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/73488/</a></p> <p>I assume odl-integration-compatible-with-all installs the "world" (at least the managed world). If so and this is a controller CSIT why are we installing all that? All you should need odl-clustering-test-app, odl-jolokia and odl-restconf(-noauth).</p> <p>Thanks Tom. I'm just queued a csit job that removes the odl-netconf-noauth from the features list.</p> <p> </p> <p>The odl-integration-compatible-with-all feature is installed as part of clustering config, I'll have to dig some more or get input from Jamo on how best to adjust that one.</p> <p>ok, so we got a good theory here, and I know <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=vpickard" class="user-hover" rel="vpickard">vpickard</a> is testing this in the sandbox now, but running the job without<br/> the noauth feature being installed. The protection patch c/73488 seems like a good idea as well. Need it cherry picked<br/> to oxygen too, I guess.</p> <p>We can do that but that patch really shouldn't be needed for CSIT - as I've mentioned you don't want to install both auth and no-auth - it will be non-deterministic which one will take effect on each run.  You should be able to just remove odl-integration-compatible-with-all as I've mentioned. odl-clustering-test-app installs mdsal etc - that's all I install, along with restconf, when I'm testing with the cars stuff.</p> <p>the point of compatible-with-all is to load <del>all</del> the features we have determined should be able to<br/> run together without breaking things.</p> <p>I wasn't implying the patch is needed for CSIT. If this is really what's happening, the patch to protect<br/> /restconf from being totally dead is nice to have. We will now also know that putting noauth on the<br/> same job as running compatible-with-all is wrong.</p> <p>btw, I don't understand how this explains the sporadic nature of this problem. If compatible-with-all<br/> is bringing in -auth and we are also always trying to install -noauth, then why doesn't the issue show<br/> up every time?</p> <p>It repro'ed every time I installed both - same exact error - so the theory is correct - based on how pax web works it is deterministic wrt failure. Are you sure you're really seeing it sporadically? Has there ever been a case where you saw it occur and didn't install both.</p> <p>I thought compatible-with-all was for dist-check jobs... isn't necessary for CSIT?</p> <p> </p> <p> </p> <p>ok, good that it's easy to reproduce. <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=vpickard" class="user-hover" rel="vpickard">vpickard</a>, I assumed this was one of the sporadic failures. Is this happening<br/> every time? Seems that the end result of restconf being totally dead would make every test case fail.</p> <p>compatible-with-all is what is used in every CSIT job you see with <del>all</del> in it. The jobs with <del>only</del> in it do not<br/> install that.</p> <p>Maybe it's a legacy thing. I mean if a CSIT is testing clustering with cars, why install eg lispflowmapping, sfc et al? What if one of those unused features has an orthogonal issue that affects the clustering tests?</p> <p>I took a closer look at the logs for this CSIT run. ODL is stopped and started a number of times in the CSIT.</p> <p> </p> <p>From what I see, one of the features will end up successfully registering, and the other one will fail. And it isn't the same one that wins every time.</p> <p>I think we are/should be using the bierman02-auth version, right? So, if that is the one that fails to register, then 500 from then on until ODL is restarted in next test case down the line.</p> <p>Right - that's what I mentioned before - it's non-deterministic which one gets in first.  We should be using auth. In fact no-auth really should not even exist anymore - it was kept for legacy from the days before we even had authentication. Devs wanted to keep it for convenience (including me <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/>).  You may notice the new rfc8040 restconf feature no longer has a no-auth option - I removed it recently. </p> <p>That said, the reason for using no-auth  now in CSIT was to try to isolate the restconf failures in <a href="https://jira.opendaylight.org/browse/CONTROLLER-1838" class="external-link" rel="nofollow">https://jira.opendaylight.org/browse/CONTROLLER-1838</a>. Which then led to this issue b/c CSIT was installing both which is a no-no. </p> <p>I'll lower the priority.</p> <p> </p> <p>I think that's the point, to find (and fix) the orthogonal issues that come up. If it's deemed some kind of <br/> unresolvable conflict, then we remove the feature from the compatible-with-all list.</p> <p>we used to have both <del>only</del> and <del>all</del> jobs, so we could more easily know if the problems were<br/> coming like this, but we were overloading jenkins, so where we could we removed the <del>only</del><br/> jobs. this is one such case.</p> <p>So, it's non-deterministic, but <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=tpantelis" class="user-hover" rel="tpantelis">tpantelis</a> can hit it every time which sounds <b>deterministic</b> <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>maybe the CSIT timing is random enough that it doesn't happen every time...</p> <p>Also, <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=tpantelis" class="user-hover" rel="tpantelis">tpantelis</a> this is not the same job or same feature set from <a href="https://jira.opendaylight.org/browse/CONTROLLER-1838" title="follower reports 401 (unauthorized) and 500 (Internal Error) when leader is isolated." class="issue-link" data-issue-key="CONTROLLER-1838"><del>CONTROLLER-1838</del></a>. let's<br/> not tie them together.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=vpickard" class="user-hover" rel="vpickard">vpickard</a>, once we known tihs <b>never</b> comes once we don't have -noauth, we can close this<br/> bug, I think.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=tpantelis" class="user-hover" rel="tpantelis">tpantelis</a>, I wonder if we should hide this -noauth feature somehow, since it's strictly a <br/> dev tool?</p> <p>Maybe I'm not understanding something here... but I thought it's the purpose of the distribution-check jobs which run for every gerrit patch to test feature compatibility across all the projects. Hence why compatible-with-all was created - distribution-check jobs install and verify all the features come up.</p> <p>If all that is correct (which I think it is), why do project CSITs need to duplicate what distribution-check jobs already do? </p> <p>no, that's not correct. compatible-with-all was there from the start and used in CSIT. distribution-check came<br/> later and took advantage of it. And, distribution-check just loads features. CSIT executes functional/system<br/> level tests against those features.</p> <p>example, l2switch, if loaded along side of netvirt will break things. l2switch is not compatible with other<br/> openflow-y applications. However, l2switch will load just fine with along side netvirt. Thus, we ended<br/> up creating those buckets (compatible or not) so we can make sure we can keep track of these kinds of<br/> things.</p> <p>anyway, none of this really matters as it applies to this jira.</p> <p>this jira is about us investigating and figuring out (we hope) that we should not be using -noauth<br/> (anywhere, I guess) in our CSIT jobs.</p> <p>I mean it's non-deterministic which feature will install and which will fail, ie first one in wins. The failure (w/o my patch) is deterministic.</p> <p>Not sure how you hide it. It might not just be a dev tool - it's been released for many versions so someone else may be using it out there so we technically can't/shouldn't just remove it from the release w/o going thru the deprecation/EOL cycle. That process can certainly be started if someone wants to drive it...</p> <p> </p> <p> </p> <p>Here is the patch that removed odl-netconf-noauth from the clustering csit job. Patch has been merged.</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/73535/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/73535/</a></p> Relates NETVIRT-1315 Development Rank 0|i03g2f: