OpenDaylight JIRA https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 [RELENG-85] Version bump odlparent 3.1.0 / yangtools 2.0.3 https://jira.opendaylight.org/browse/RELENG-85 releng <p>We need to bump to odlparent 3.1.0 / yangtools 2.0.3 across the board for:</p> <p> </p> <ul> <li>Fluorine <a href="https://git.opendaylight.org/gerrit/#/q/topic:odlparent-3.1.0" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/q/topic:odlparent-3.1.0</a></li> <li>Oxygen <a href="https://git.opendaylight.org/gerrit/#/q/topic:odlparent-3.1.0-oxygen" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/q/topic:odlparent-3.1.0-oxygen</a></li> </ul> RELENG-85 Version bump odlparent 3.1.0 / yangtools 2.0.3 Bug Highest Resolved Done Thanh Ha (zxiiro) Thanh Ha (zxiiro) Thu, 5 Apr 2018 20:29:49 +0000 Tue, 17 Apr 2018 02:08:44 +0000 Tue, 17 Apr 2018 02:08:44 +0000 Oxygen Autorelease 0 12 <p>Working on the Oxygen bumps now.</p> <p>The controller patch <a href="https://git.opendaylight.org/gerrit/69987" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/69987</a> is failing to build yang-jmx-generator with a test failure. Any ideas on what is causing it?</p> <p>Looks like it's failing some string comparison tests.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Failed tests: ServiceInterfaceEntryTest.testCreateFromIdentities:119 Expected: is "An extension of the simple pool of threads able to schedule work to be executed at some point in time." but: was "An extension of the simple pool of threads able to schedule\nwork to be executed at some point in time." UnknownExtensionTest&gt;ServiceInterfaceEntryTest.testCreateFromIdentities:119 Expected: is "An extension of the simple pool of threads able to schedule work to be executed at some point in time." but: was "An extension of the simple pool of threads able to schedule\nwork to be executed at some point in time." </pre> </div></div> <p>I don't believe controller changed these strings since the version bump so I think something in yangtools is possibly adding the '\n' string. I'm not sure what the right fix is for controller.</p> <p>Talked to <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=rovarga" class="user-hover" rel="rovarga">rovarga</a> on IRC and he suggested we pull in this change to fix the issue.</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/70051/1/opendaylight/config/yang-jmx-generator/src/test/java/org/opendaylight/controller/config/yangjmxgenerator/ServiceInterfaceEntryTest.java" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/70051/1/opendaylight/config/yang-jmx-generator/src/test/java/org/opendaylight/controller/config/yangjmxgenerator/ServiceInterfaceEntryTest.java</a></p> <p>Looks like we're hitting some NPE issues now:</p> <p><a href="https://jenkins.opendaylight.org/releng/job/controller-maven-verify-oxygen-mvn33-openjdk8/305/console" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/job/controller-maven-verify-oxygen-mvn33-openjdk8/305/console</a></p> <p>I'll recheck the job to see if it's intermittent.</p> <p>The NPEs are related to Akka shutdown, but they’re not the cause of the problem — the issue relates to ReplicatedLogImplEntry handling. <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=tpantelis" class="user-hover" rel="tpantelis">tpantelis</a> why would this break us now?</p> <p>Fix incoming, leftovers in sal-akka-raft-example... <a href="https://git.opendaylight.org/gerrit/70417" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/70417</a></p> <p>ClassNotFoundException: org.opendaylight.controller.cluster.raft.ReplicatedLogImplEntry - related to akka setting up serialization. Never seen that before. Was the akka and/or scala version bumped in odlparent?</p> <p>We ran into another issue with odl-yanglib in netconf</p> <p>00:45:14.030 Missing dependencies: <br/> 00:45:14.030 (objectClass=org.opendaylight.yanglib.api.YangLibRestAppService) <br/> 00:45:14.030 Declarative Services</p> <p>Considering that nothing references this feature except in int/dist we decided to comment it out and deal with it later so that we can unblock version bumping.</p> <p>Patch: <a href="https://git.opendaylight.org/gerrit/#/c/69979/6/features/yanglib/pom.xml" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/69979/6/features/yanglib/pom.xml</a></p> <p>Looks like several of the remaining projects are now failing findbugs reports.</p> <p><a href="https://git.opendaylight.org/gerrit/#/q/topic:odlparent-3.1.0-oxygen" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/q/topic:odlparent-3.1.0-oxygen</a></p> <p>For some reason the YangLibRestApp doesn't get created which causes BP to timeout waiting for the service. Of course if you keep the runtime karaf folder and re-run karaf, the YangLibRestApp loads fine. I got it to work with <a href="https://git.opendaylight.org/gerrit/#/c/70468/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/70468/</a></p> <p>4 patches remain:</p> <ul> <li>bgpcep: <a href="https://git.opendaylight.org/gerrit/70010" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/70010</a></li> <li>gbp: <a href="https://git.opendaylight.org/gerrit/70007" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/70007</a></li> <li>lispflowmapping: <a href="https://git.opendaylight.org/gerrit/69980" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/69980</a></li> <li>int/dist: <a href="https://git.opendaylight.org/gerrit/70009" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/70009</a></li> </ul> <p>We need some experts in those projects to help troubleshoot the build failures.</p> <p>Additionally we need this honeycomb/vbd patch merged to fix their merge job: <a href="https://git.opendaylight.org/gerrit/69672" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/69672</a></p> <p>The BGPCEP issue (and I suspect lispflow, too) is cause by behaviour change in Netty. Ever since <a href="https://github.com/netty/netty/issues/7290" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/netty/netty/issues/7290</a> netty threads are detaching from the TCCL in which they were created, hence when we attempt to load binding classes using TCCL we stumble on a null ClassLoader.</p> <p>Patch to harden ClassLoaderUtils against a missing TCCL. <a href="https://git.opendaylight.org/gerrit/70659" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/70659</a></p> <p>Does this mean we need a new release of yangtools and bump again?</p> <p>I have not check lispflowmapping nor gbp, but it so far a new yangtools release is not needed.</p> <p>lispflowmapping is good to go. gbp needs a fix to realign it with SFC model changes. The patch was updated, waiting for committer review.</p> <p>FYI after the upgrade we see sporadically jersey not being loaded when all features are installed by the end of the distribution-check test:</p> <p><a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/distribution-check-oxygen/239/console.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/distribution-check-oxygen/239/console.log.gz</a></p> <p>You can see in above karaf log (when REST fails) that following line (extracted from a success test) is missing:</p> <p>2018-04-10T18:43:26,196 | INFO | paxweb-extender-2-thread-1 | WebApplicationImpl | 41 - com.sun.jersey.jersey-server - 1.19.4 | Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'</p> <p>TSC decided that we should move forward with fluorine version bumps today.</p> <p>AAA and Controller merge jobs keeps failing despite repeated "remerge"s it's blocking our ability to move on with the version bumping.</p> <p>Ok got a successful Controller finally <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/> we can move on to netconf.</p> <p>remerge on netconf is failing and not able to proceed further. </p> <p>As mentioned on the patch, netconf needs an extra fix to resolve synchronisation issues which FindBugs is now flagging; see <a href="https://git.opendaylight.org/gerrit/70902" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/70902</a></p> <p>AAA also needs <a href="https://git.opendaylight.org/gerrit/70906" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/70906</a> (but that doesn’t resolve the com.sun.ws.rs.ext.RuntimeDelegateImpl error).</p> <p>Currently netconf is failing on it's odl-netconf-clustered-topology module. More blueprint related issues?</p> <p>NOK org.opendaylight.aaa.shiro:0.8.0.SNAPSHOT: OSGi state = Active, Karaf bundleState = Failure, due to: Blueprint</p> <p><a href="https://jenkins.opendaylight.org/releng/job/netconf-maven-verify-fluorine-mvn33-openjdk8/208/console" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/job/netconf-maven-verify-fluorine-mvn33-openjdk8/208/console</a></p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=tpantelis" class="user-hover" rel="tpantelis">tpantelis</a> Is the blueprint issues in netconf something you can help us get past?</p> <p>This does smell like a version mis-alignment again. Is this reproducible in a karaf shell somehow?</p> <p>Yup, we are dealing with:</p> <p> </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>javax.ws.rs.client │ 2.0.1                                  │ 104 │ javax.ws.rs-api javax.ws.rs.container │ 2.0.1                                  │ 104 │ javax.ws.rs-api javax.ws.rs.core      │ 1.1.1                                  │ 105 │ javax.ws.rs.jsr311-api javax.ws.rs.core      │ 2.0.1                                  │ 104 │ javax.ws.rs-api javax.ws.rs.ext       │ 1.1.1                                  │ 105 │ javax.ws.rs.jsr311-api javax.ws.rs.ext       │ 2.0.1                                  │ 104 │ javax.ws.rs-api javax.ws.rs           │ 1.1.1                                  │ 105 │ javax.ws.rs.jsr311-api javax.ws.rs │ 2.0.1                                  │ 104 │ javax.ws.rs-api </pre> </div></div> <p> </p> <p>This boils down to:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>nite@nitebug : ~/odl/netconf/karaf/x/netconf-karaf-1.8.0-SNAPSHOT on  $ find . -name \*-features.xml | xargs fgrep jsr311-api ./system/org/opendaylight/aaa/odl-aaa-api/0.8.0-SNAPSHOT/odl-aaa-api-0.8.0-SNAPSHOT-features.xml:        &lt;bundle&gt;mvn:javax.ws.rs/jsr311-api/1.1.1&lt;/bundle&gt; ./system/org/opendaylight/controller/odl-config-persister/0.9.0-SNAPSHOT/odl-config-persister-0.9.0-SNAPSHOT-features.xml:        &lt;bundle&gt;mvn:javax.ws.rs/jsr311-api/1.1.1&lt;/bundle&gt; nite@nitebug : ~/odl/netconf/karaf/x/netconf-karaf-1.8.0-SNAPSHOT on  $ find . -name \*-features.xml | xargs fgrep javax.ws.rs-api ./system/org/opendaylight/odlparent/odl-jackson-2.8/3.1.0/odl-jackson-2.8-3.1.0-features.xml:        &lt;bundle&gt;mvn:javax.ws.rs/javax.ws.rs-api/2.0.1&lt;/bundle&gt; ./system/org/opendaylight/aaa/odl-aaa-shiro/0.8.0-SNAPSHOT/odl-aaa-shiro-0.8.0-SNAPSHOT-features.xml:        &lt;bundle&gt;mvn:javax.ws.rs/javax.ws.rs-api/2.0.1&lt;/bundle&gt; </pre> </div></div> <p>So I suspect anyone interacting with both odl-aaa-shiro/odl-jackson and odl-aaa-api or odl-config-persister is exposed to this conflict.</p> <p>A rather blind stab at eliminating one of the uses: <a href="https://git.opendaylight.org/gerrit/70924" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/70924</a></p> <p> </p> <p>We decided to push in <a href="https://git.opendaylight.org/gerrit/70902" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/70902</a> and fight with remerges rather than rechecks in hopes that we can get one going.</p> <p>In the meantime I've also forced up some netconf artifacts with the patch included so that we can continue to progress with the version bump patches for the remaining projects.</p> <p>Disregard that patch, I was chasing ghosts. Based on evidence preserved at: <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netconf-maven-verify-fluorine-mvn33-openjdk8/210/console.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netconf-maven-verify-fluorine-mvn33-openjdk8/210/console.log.gz</a> : one aaa-related feature went through, the other one failed.</p> <p><a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netconf-maven-verify-fluorine-mvn33-openjdk8/210/features/netconf/odl-aaa-netconf-plugin/target/surefire-reports/org.opendaylight.odlparent.featuretest.SingleFeatureTest-output.txt.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netconf-maven-verify-fluorine-mvn33-openjdk8/210/features/netconf/odl-aaa-netconf-plugin/target/surefire-reports/org.opendaylight.odlparent.featuretest.SingleFeatureTest-output.txt.gz</a> is successful. Note order of installation (at the very end):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> org.opendaylight.aaa.filterchain/0.8.0.SNAPSHOT org.opendaylight.aaa.shiro/0.8.0.SNAPSHOT org.opendaylight.controller.sal-akka-raft/1.8.0.SNAPSHOT com.sun.jersey.servlet/1.19.4 com.sun.jersey.core/1.19.4 com.sun.jersey.jersey-server/1.19.4 </pre> </div></div> <p><a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netconf-maven-verify-fluorine-mvn33-openjdk8/210/features/netconf/odl-aaa-netconf-plugin-no-cluster/target/surefire-reports/org.opendaylight.odlparent.featuretest.SingleFeatureTest-output.txt.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netconf-maven-verify-fluorine-mvn33-openjdk8/210/features/netconf/odl-aaa-netconf-plugin-no-cluster/target/surefire-reports/org.opendaylight.odlparent.featuretest.SingleFeatureTest-output.txt.gz</a> is a failure. Note order of instantiation (now a bit higher):</p> <p> </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> com.sun.jersey.servlet/1.19.4 org.opendaylight.aaa.shiro/0.8.0.SNAPSHOT io.netty.common/4.1.22.Final io.netty.buffer/4.1.22.Final io.netty.transport/4.1.22.Final javax.ws.rs-api/2.0.1 com.sun.jersey.jersey-server/1.19.4 javax.mail/1.4.4 com.sun.jersey.core/1.19.4 </pre> </div></div> <p>A previous run, which tested only a single feature: <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netconf-maven-verify-fluorine-mvn33-openjdk8/205/features/netconf/odl-aaa-netconf-plugin/target/surefire-reports/org.opendaylight.odlparent.featuretest.SingleFeatureTest-output.txt.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netconf-maven-verify-fluorine-mvn33-openjdk8/205/features/netconf/odl-aaa-netconf-plugin/target/surefire-reports/org.opendaylight.odlparent.featuretest.SingleFeatureTest-output.txt.gz</a> had this order of installation:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> org.opendaylight.aaa.shiro/0.8.0.SNAPSHOT com.sun.jersey.servlet/1.19.4 org.opendaylight.controller.sal-akka-raft/1.8.0.SNAPSHOT org.opendaylight.controller.sal-distributed-datastore/1.8.0.SNAPSHOT io.netty.common/4.1.22.Final io.netty.buffer/4.1.22.Final io.netty.transport/4.1.22.Final javax.ws.rs-api/2.0.1 org.eclipse.persistence.moxy/2.6.2.v20151217-774c696 org.opendaylight.controller.sal-clustering-commons/1.8.0.SNAPSHOT com.sun.jersey.jersey-server/1.19.4 javax.mail/1.4.4 com.sun.jersey.core/1.19.4</pre> </div></div> <p>In any case we do not have logs to ascertain order of initialization, but this looks like a wiring mess in jersey, specifically servlet/server/client bundle installation and resolution.</p> <p> </p> <p>The exception is:</p> <p> </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Exception: Error when instantiating bean webInitializer of class org.opendaylight.aaa.shiro.web.env.WebInitializer org.osgi.service.blueprint.container.ComponentDefinitionException: Error when instantiating bean webInitializer of class org.opendaylight.aaa.shiro.web.env.WebInitializer at org.apache.aries.blueprint.container.BeanRecipe.wrapAsCompDefEx(BeanRecipe.java:361) at org.apache.aries.blueprint.container.BeanRecipe.getInstanceFromType(BeanRecipe.java:351) at org.apache.aries.blueprint.container.BeanRecipe.getInstance(BeanRecipe.java:282) at org.apache.aries.blueprint.container.BeanRecipe.internalCreate2(BeanRecipe.java:830) at org.apache.aries.blueprint.container.BeanRecipe.internalCreate(BeanRecipe.java:811) at org.apache.aries.blueprint.di.AbstractRecipe$1.call(AbstractRecipe.java:79) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.aries.blueprint.di.AbstractRecipe.create(AbstractRecipe.java:88) at org.apache.aries.blueprint.container.BlueprintRepository.createInstances(BlueprintRepository.java:255) at org.apache.aries.blueprint.container.BlueprintRepository.createAll(BlueprintRepository.java:186) at org.apache.aries.blueprint.container.BlueprintContainerImpl.instantiateEagerComponents(BlueprintContainerImpl.java:704) at org.apache.aries.blueprint.container.BlueprintContainerImpl.doRun(BlueprintContainerImpl.java:410) at org.apache.aries.blueprint.container.BlueprintContainerImpl.run(BlueprintContainerImpl.java:275) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.aries.blueprint.container.ExecutorServiceWrapper.run(ExecutorServiceWrapper.java:106) at org.apache.aries.blueprint.utils.threading.impl.DiscardableRunnable.run(DiscardableRunnable.java:48) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.ExceptionInInitializerError at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at com.sun.proxy.$Proxy98.&lt;clinit&gt;(Unknown Source) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at java.lang.reflect.Proxy.newProxyInstance(Proxy.java:739) at com.sun.jersey.server.impl.application.WebApplicationImpl$26.run(WebApplicationImpl.java:1626) at java.security.AccessController.doPrivileged(Native Method) at com.sun.jersey.server.impl.application.WebApplicationImpl.createProxy(WebApplicationImpl.java:1623) at com.sun.jersey.server.impl.application.WebApplicationImpl.&lt;init&gt;(WebApplicationImpl.java:335) at com.sun.jersey.server.impl.container.WebApplicationProviderImpl.createWebApplication(WebApplicationProviderImpl.java:55) at com.sun.jersey.spi.container.WebApplicationFactory.createWebApplication(WebApplicationFactory.java:66) at com.sun.jersey.spi.container.servlet.ServletContainer.create(ServletContainer.java:412) at com.sun.jersey.spi.container.servlet.ServletContainer$InternalWebComponent.create(ServletContainer.java:327) at com.sun.jersey.spi.container.servlet.WebComponent.load(WebComponent.java:603) at com.sun.jersey.spi.container.servlet.WebComponent.init(WebComponent.java:207) at com.sun.jersey.spi.container.servlet.ServletContainer.init(ServletContainer.java:394) at com.sun.jersey.spi.container.servlet.ServletContainer.init(ServletContainer.java:577) at javax.servlet.GenericServlet.init(GenericServlet.java:244) at org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:643) at org.eclipse.jetty.servlet.ServletHolder.initialize(ServletHolder.java:422) at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:892) at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1596) at org.eclipse.jetty.servlet.ServletHandler.setServletMappings(ServletHandler.java:1684) at org.eclipse.jetty.servlet.ServletHandler.addServletMapping(ServletHandler.java:1063) at org.ops4j.pax.web.service.jetty.internal.JettyServerImpl$2.call(JettyServerImpl.java:425) at org.ops4j.pax.web.service.jetty.internal.JettyServerImpl$2.call(JettyServerImpl.java:420) at org.ops4j.pax.swissbox.core.ContextClassLoaderUtils.doWithClassLoader(ContextClassLoaderUtils.java:60) at org.ops4j.pax.web.service.jetty.internal.JettyServerImpl.addServlet(JettyServerImpl.java:419) at org.ops4j.pax.web.service.jetty.internal.ServerControllerImpl$Started.addServlet(ServerControllerImpl.java:311) at org.ops4j.pax.web.service.jetty.internal.ServerControllerImpl.addServlet(ServerControllerImpl.java:120) at org.ops4j.pax.web.service.internal.HttpServiceStarted.registerServlet(HttpServiceStarted.java:245) at org.ops4j.pax.web.service.internal.HttpServiceStarted.registerServlet(HttpServiceStarted.java:401) at org.ops4j.pax.web.service.internal.HttpServiceProxy.registerServlet(HttpServiceProxy.java:463) at org.opendaylight.aaa.web.osgi.PaxWebServer$WebContextImpl.registerServlet(PaxWebServer.java:200) at org.opendaylight.aaa.web.osgi.PaxWebServer$WebContextImpl.&lt;init&gt;(PaxWebServer.java:173) at org.opendaylight.aaa.web.osgi.PaxWebServer$2$1.&lt;init&gt;(PaxWebServer.java:114) at org.opendaylight.aaa.web.osgi.PaxWebServer$2.registerWebContext(PaxWebServer.java:114) at Proxy7c938f7a_ec90_4982_a4de_c16af07c04dd.registerWebContext(Unknown Source) at org.opendaylight.aaa.shiro.web.env.WebInitializer.&lt;init&gt;(WebInitializer.java:54) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.aries.blueprint.utils.ReflectionUtils.newInstance(ReflectionUtils.java:331) at org.apache.aries.blueprint.container.BeanRecipe.newInstance(BeanRecipe.java:984) at org.apache.aries.blueprint.container.BeanRecipe.getInstanceFromType(BeanRecipe.java:349) ... 22 more Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: com.sun.ws.rs.ext.RuntimeDelegateImpl cannot be found by javax.ws.rs.jsr311-api_1.1.1 at javax.ws.rs.ext.RuntimeDelegate.findDelegate(RuntimeDelegate.java:122) at javax.ws.rs.ext.RuntimeDelegate.getInstance(RuntimeDelegate.java:91) at javax.ws.rs.core.EntityTag.&lt;clinit&gt;(EntityTag.java:35) ... 71 more Caused by: java.lang.ClassNotFoundException: com.sun.ws.rs.ext.RuntimeDelegateImpl cannot be found by javax.ws.rs.jsr311-api_1.1.1 at org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(BundleLoader.java:484) at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:395) at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:387) at org.eclipse.osgi.internal.loader.ModuleClassLoader.loadClass(ModuleClassLoader.java:150) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at javax.ws.rs.ext.FactoryFinder.newInstance(FactoryFinder.java:62) at javax.ws.rs.ext.FactoryFinder.find(FactoryFinder.java:155) at javax.ws.rs.ext.RuntimeDelegate.findDelegate(RuntimeDelegate.java:105) ... 73 more </pre> </div></div> <p> </p> <p>Unwinding the stack, we are coming in from WebInitializer, then going to jersey-servlet, then to jersey-server and are failing when jsr311-api is asked by the jersey-server to instantiate a class from jersey-client. Note that that value is the default:</p> <p><a href="http://grepcode.com/file/repo1.maven.org/maven2/javax.ws.rs/jsr311-api/1.1.1/javax/ws/rs/ext/RuntimeDelegate.java#41" class="external-link" target="_blank" rel="nofollow noopener">http://grepcode.com/file/repo1.maven.org/maven2/javax.ws.rs/jsr311-api/1.1.1/javax/ws/rs/ext/RuntimeDelegate.java#41</a></p> <p>Based on the comments in that class, and the fact we second-to-last stack is showing we are entering through:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> at javax.ws.rs.ext.RuntimeDelegate.getInstance(RuntimeDelegate.java:91)</pre> </div></div> <p>it would seem that RuntimeDelegate.setInstance() was not called by jersey-server to bind RuntimeDelegate to its internal implementation (com.sun.jersey.server.impl.provider/RuntimeDelegateImpl) and hence it is scrambling to recover (which may have worked in single-classloader environments).</p> <p> </p> <p>So delving into jersey, this gem of a package is found:</p> <p><a href="https://github.com/jersey/jersey-1.x/tree/master/jersey-core/src/main/java/com/sun/jersey/core/osgi" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/jersey/jersey-1.x/tree/master/jersey-core/src/main/java/com/sun/jersey/core/osgi</a></p> <p>Bottom line, the way I am reading this, is:</p> <p>if jersey-core cannot find jersey-server bundle (by name!) when its activator runs, it will set the delegate to the client version (if it can load it). If not, it will set it to null.</p> <p>Now ... all of this is supposedly logged via java.util.logging – do we have that bridged in karaf.log?</p> <p>As for the fix — we currently are packaging server+core in odl-aaa-api and servlet+client in odl-aaa-shiro. I would suggest bundling the entire mess into a single feature (maybe with a prerequisite) and see what gives.</p> <p>The race condition is probably that aaa-shiro is starting as soon as it has all required services (e.g. web api) and has jersey packages <b>resolved</b>. There is no guarantee of ordering between aaa-shiro and jersey-{server,core} activators. Hence aaa-shiro can get running before jersey-core is started, leading to an SFT-shattering kaboom. Jersey bundles are not publishing any services, so I fear we will need to either do a prerequisite or code something up.</p> <p>As for coding: a simple bundle tracker which publishes a dummy service (which gets injected into jersey users) when it sees jersey-core started event should work.</p> <p>Fun bit, after jersey-2 conversion we got: <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/aaa-maven-verify-fluorine-mvn33-openjdk8/163/console.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/aaa-maven-verify-fluorine-mvn33-openjdk8/163/console.log.gz</a></p> <p> </p> <p>Patches for the Jersey issue:</p> <ul> <li><a href="https://git.opendaylight.org/gerrit/70971" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/70971</a></li> <li><a href="https://git.opendaylight.org/gerrit/70979" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/70979</a></li> </ul> <p>The integration/distribution project patch is now merged. I'm going to consider this task complete as fluorine is now unblocked by version bumping.</p> <p>Only 2 remaining projects remain:</p> <ul> <li>TSDR</li> <li>Telemetry</li> </ul> <p>TSDR is blocking telemetry's merge and is currently failing due to test issues.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=scottmelton" class="user-hover" rel="scottmelton">scottmelton</a> any ideas one what the failure cause is?</p> Blocks BGPCEP-786 Development Epic Link RELENG-93 Rank 0|i03atw: