Working on the Oxygen bumps now.
The controller patch https://git.opendaylight.org/gerrit/69987 is failing to build yang-jmx-generator with a test failure. Any ideas on what is causing it?
Looks like it's failing some string comparison tests.
Failed tests:
ServiceInterfaceEntryTest.testCreateFromIdentities:119
Expected: is "An extension of the simple pool of threads able to schedule work to be executed at some point in time."
but: was "An extension of the simple pool of threads able to schedule\nwork to be executed at some point in time."
UnknownExtensionTest>ServiceInterfaceEntryTest.testCreateFromIdentities:119
Expected: is "An extension of the simple pool of threads able to schedule work to be executed at some point in time."
but: was "An extension of the simple pool of threads able to schedule\nwork to be executed at some point in time."
I don't believe controller changed these strings since the version bump so I think something in yangtools is possibly adding the '\n' string. I'm not sure what the right fix is for controller.
Talked to rovarga on IRC and he suggested we pull in this change to fix the issue.
Looks like we're hitting some NPE issues now:
I'll recheck the job to see if it's intermittent.
The NPEs are related to Akka shutdown, but they’re not the cause of the problem — the issue relates to ReplicatedLogImplEntry handling. tpantelis why would this break us now?
Fix incoming, leftovers in sal-akka-raft-example... https://git.opendaylight.org/gerrit/70417
ClassNotFoundException: org.opendaylight.controller.cluster.raft.ReplicatedLogImplEntry - related to akka setting up serialization. Never seen that before. Was the akka and/or scala version bumped in odlparent?
We ran into another issue with odl-yanglib in netconf
00:45:14.030 Missing dependencies:
00:45:14.030 (objectClass=org.opendaylight.yanglib.api.YangLibRestAppService)
00:45:14.030 Declarative Services
Considering that nothing references this feature except in int/dist we decided to comment it out and deal with it later so that we can unblock version bumping.
Patch: https://git.opendaylight.org/gerrit/#/c/69979/6/features/yanglib/pom.xml
Looks like several of the remaining projects are now failing findbugs reports.
https://git.opendaylight.org/gerrit/#/q/topic:odlparent-3.1.0-oxygen
For some reason the YangLibRestApp doesn't get created which causes BP to timeout waiting for the service. Of course if you keep the runtime karaf folder and re-run karaf, the YangLibRestApp loads fine. I got it to work with https://git.opendaylight.org/gerrit/#/c/70468/
4 patches remain:
- bgpcep: https://git.opendaylight.org/gerrit/70010
- gbp: https://git.opendaylight.org/gerrit/70007
- lispflowmapping: https://git.opendaylight.org/gerrit/69980
- int/dist: https://git.opendaylight.org/gerrit/70009
We need some experts in those projects to help troubleshoot the build failures.
Additionally we need this honeycomb/vbd patch merged to fix their merge job: https://git.opendaylight.org/gerrit/69672
The BGPCEP issue (and I suspect lispflow, too) is cause by behaviour change in Netty. Ever since https://github.com/netty/netty/issues/7290 netty threads are detaching from the TCCL in which they were created, hence when we attempt to load binding classes using TCCL we stumble on a null ClassLoader.
Patch to harden ClassLoaderUtils against a missing TCCL. https://git.opendaylight.org/gerrit/70659
Does this mean we need a new release of yangtools and bump again?
I have not check lispflowmapping nor gbp, but it so far a new yangtools release is not needed.
lispflowmapping is good to go. gbp needs a fix to realign it with SFC model changes. The patch was updated, waiting for committer review.
FYI after the upgrade we see sporadically jersey not being loaded when all features are installed by the end of the distribution-check test:
You can see in above karaf log (when REST fails) that following line (extracted from a success test) is missing:
2018-04-10T18:43:26,196 | INFO | paxweb-extender-2-thread-1 | WebApplicationImpl | 41 - com.sun.jersey.jersey-server - 1.19.4 | Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'
TSC decided that we should move forward with fluorine version bumps today.
AAA and Controller merge jobs keeps failing despite repeated "remerge"s it's blocking our ability to move on with the version bumping.
Ok got a successful Controller finally 
we can move on to netconf.
remerge on netconf is failing and not able to proceed further.
As mentioned on the patch, netconf needs an extra fix to resolve synchronisation issues which FindBugs is now flagging; see https://git.opendaylight.org/gerrit/70902
AAA also needs https://git.opendaylight.org/gerrit/70906 (but that doesn’t resolve the com.sun.ws.rs.ext.RuntimeDelegateImpl error).
Currently netconf is failing on it's odl-netconf-clustered-topology module. More blueprint related issues?
NOK org.opendaylight.aaa.shiro:0.8.0.SNAPSHOT: OSGi state = Active, Karaf bundleState = Failure, due to: Blueprint
https://jenkins.opendaylight.org/releng/job/netconf-maven-verify-fluorine-mvn33-openjdk8/208/console
tpantelis Is the blueprint issues in netconf something you can help us get past?
This does smell like a version mis-alignment again. Is this reproducible in a karaf shell somehow?
Yup, we are dealing with:
javax.ws.rs.client │ 2.0.1 │ 104 │ javax.ws.rs-api javax.ws.rs.container │ 2.0.1 │ 104 │ javax.ws.rs-api javax.ws.rs.core │ 1.1.1 │ 105 │ javax.ws.rs.jsr311-api javax.ws.rs.core │ 2.0.1 │ 104 │ javax.ws.rs-api javax.ws.rs.ext │ 1.1.1 │ 105 │ javax.ws.rs.jsr311-api javax.ws.rs.ext │ 2.0.1 │ 104 │ javax.ws.rs-api javax.ws.rs │ 1.1.1 │ 105 │ javax.ws.rs.jsr311-api javax.ws.rs │ 2.0.1 │ 104 │ javax.ws.rs-api
This boils down to:
nite@nitebug : ~/odl/netconf/karaf/x/netconf-karaf-1.8.0-SNAPSHOT on $ find . -name \*-features.xml | xargs fgrep jsr311-api ./system/org/opendaylight/aaa/odl-aaa-api/0.8.0-SNAPSHOT/odl-aaa-api-0.8.0-SNAPSHOT-features.xml: <bundle>mvn:javax.ws.rs/jsr311-api/1.1.1</bundle> ./system/org/opendaylight/controller/odl-config-persister/0.9.0-SNAPSHOT/odl-config-persister-0.9.0-SNAPSHOT-features.xml: <bundle>mvn:javax.ws.rs/jsr311-api/1.1.1</bundle> nite@nitebug : ~/odl/netconf/karaf/x/netconf-karaf-1.8.0-SNAPSHOT on $ find . -name \*-features.xml | xargs fgrep javax.ws.rs-api ./system/org/opendaylight/odlparent/odl-jackson-2.8/3.1.0/odl-jackson-2.8-3.1.0-features.xml: <bundle>mvn:javax.ws.rs/javax.ws.rs-api/2.0.1</bundle> ./system/org/opendaylight/aaa/odl-aaa-shiro/0.8.0-SNAPSHOT/odl-aaa-shiro-0.8.0-SNAPSHOT-features.xml: <bundle>mvn:javax.ws.rs/javax.ws.rs-api/2.0.1</bundle>
So I suspect anyone interacting with both odl-aaa-shiro/odl-jackson and odl-aaa-api or odl-config-persister is exposed to this conflict.
A rather blind stab at eliminating one of the uses: https://git.opendaylight.org/gerrit/70924
We decided to push in https://git.opendaylight.org/gerrit/70902 and fight with remerges rather than rechecks in hopes that we can get one going.
In the meantime I've also forced up some netconf artifacts with the patch included so that we can continue to progress with the version bump patches for the remaining projects.
Disregard that patch, I was chasing ghosts. Based on evidence preserved at: https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netconf-maven-verify-fluorine-mvn33-openjdk8/210/console.log.gz : one aaa-related feature went through, the other one failed.
https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netconf-maven-verify-fluorine-mvn33-openjdk8/210/features/netconf/odl-aaa-netconf-plugin/target/surefire-reports/org.opendaylight.odlparent.featuretest.SingleFeatureTest-output.txt.gz is successful. Note order of installation (at the very end):
org.opendaylight.aaa.filterchain/0.8.0.SNAPSHOT org.opendaylight.aaa.shiro/0.8.0.SNAPSHOT org.opendaylight.controller.sal-akka-raft/1.8.0.SNAPSHOT com.sun.jersey.servlet/1.19.4 com.sun.jersey.core/1.19.4 com.sun.jersey.jersey-server/1.19.4
https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netconf-maven-verify-fluorine-mvn33-openjdk8/210/features/netconf/odl-aaa-netconf-plugin-no-cluster/target/surefire-reports/org.opendaylight.odlparent.featuretest.SingleFeatureTest-output.txt.gz is a failure. Note order of instantiation (now a bit higher):
com.sun.jersey.servlet/1.19.4 org.opendaylight.aaa.shiro/0.8.0.SNAPSHOT io.netty.common/4.1.22.Final io.netty.buffer/4.1.22.Final io.netty.transport/4.1.22.Final javax.ws.rs-api/2.0.1 com.sun.jersey.jersey-server/1.19.4 javax.mail/1.4.4 com.sun.jersey.core/1.19.4
A previous run, which tested only a single feature: https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netconf-maven-verify-fluorine-mvn33-openjdk8/205/features/netconf/odl-aaa-netconf-plugin/target/surefire-reports/org.opendaylight.odlparent.featuretest.SingleFeatureTest-output.txt.gz had this order of installation:
org.opendaylight.aaa.shiro/0.8.0.SNAPSHOT com.sun.jersey.servlet/1.19.4 org.opendaylight.controller.sal-akka-raft/1.8.0.SNAPSHOT org.opendaylight.controller.sal-distributed-datastore/1.8.0.SNAPSHOT io.netty.common/4.1.22.Final io.netty.buffer/4.1.22.Final io.netty.transport/4.1.22.Final javax.ws.rs-api/2.0.1 org.eclipse.persistence.moxy/2.6.2.v20151217-774c696 org.opendaylight.controller.sal-clustering-commons/1.8.0.SNAPSHOT com.sun.jersey.jersey-server/1.19.4 javax.mail/1.4.4 com.sun.jersey.core/1.19.4
In any case we do not have logs to ascertain order of initialization, but this looks like a wiring mess in jersey, specifically servlet/server/client bundle installation and resolution.
The exception is:
Exception: Error when instantiating bean webInitializer of class org.opendaylight.aaa.shiro.web.env.WebInitializer org.osgi.service.blueprint.container.ComponentDefinitionException: Error when instantiating bean webInitializer of class org.opendaylight.aaa.shiro.web.env.WebInitializer at org.apache.aries.blueprint.container.BeanRecipe.wrapAsCompDefEx(BeanRecipe.java:361) at org.apache.aries.blueprint.container.BeanRecipe.getInstanceFromType(BeanRecipe.java:351) at org.apache.aries.blueprint.container.BeanRecipe.getInstance(BeanRecipe.java:282) at org.apache.aries.blueprint.container.BeanRecipe.internalCreate2(BeanRecipe.java:830) at org.apache.aries.blueprint.container.BeanRecipe.internalCreate(BeanRecipe.java:811) at org.apache.aries.blueprint.di.AbstractRecipe$1.call(AbstractRecipe.java:79) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.aries.blueprint.di.AbstractRecipe.create(AbstractRecipe.java:88) at org.apache.aries.blueprint.container.BlueprintRepository.createInstances(BlueprintRepository.java:255) at org.apache.aries.blueprint.container.BlueprintRepository.createAll(BlueprintRepository.java:186) at org.apache.aries.blueprint.container.BlueprintContainerImpl.instantiateEagerComponents(BlueprintContainerImpl.java:704) at org.apache.aries.blueprint.container.BlueprintContainerImpl.doRun(BlueprintContainerImpl.java:410) at org.apache.aries.blueprint.container.BlueprintContainerImpl.run(BlueprintContainerImpl.java:275) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.aries.blueprint.container.ExecutorServiceWrapper.run(ExecutorServiceWrapper.java:106) at org.apache.aries.blueprint.utils.threading.impl.DiscardableRunnable.run(DiscardableRunnable.java:48) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.ExceptionInInitializerError at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at com.sun.proxy.$Proxy98.<clinit>(Unknown Source) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at java.lang.reflect.Proxy.newProxyInstance(Proxy.java:739) at com.sun.jersey.server.impl.application.WebApplicationImpl$26.run(WebApplicationImpl.java:1626) at java.security.AccessController.doPrivileged(Native Method) at com.sun.jersey.server.impl.application.WebApplicationImpl.createProxy(WebApplicationImpl.java:1623) at com.sun.jersey.server.impl.application.WebApplicationImpl.<init>(WebApplicationImpl.java:335) at com.sun.jersey.server.impl.container.WebApplicationProviderImpl.createWebApplication(WebApplicationProviderImpl.java:55) at com.sun.jersey.spi.container.WebApplicationFactory.createWebApplication(WebApplicationFactory.java:66) at com.sun.jersey.spi.container.servlet.ServletContainer.create(ServletContainer.java:412) at com.sun.jersey.spi.container.servlet.ServletContainer$InternalWebComponent.create(ServletContainer.java:327) at com.sun.jersey.spi.container.servlet.WebComponent.load(WebComponent.java:603) at com.sun.jersey.spi.container.servlet.WebComponent.init(WebComponent.java:207) at com.sun.jersey.spi.container.servlet.ServletContainer.init(ServletContainer.java:394) at com.sun.jersey.spi.container.servlet.ServletContainer.init(ServletContainer.java:577) at javax.servlet.GenericServlet.init(GenericServlet.java:244) at org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:643) at org.eclipse.jetty.servlet.ServletHolder.initialize(ServletHolder.java:422) at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:892) at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1596) at org.eclipse.jetty.servlet.ServletHandler.setServletMappings(ServletHandler.java:1684) at org.eclipse.jetty.servlet.ServletHandler.addServletMapping(ServletHandler.java:1063) at org.ops4j.pax.web.service.jetty.internal.JettyServerImpl$2.call(JettyServerImpl.java:425) at org.ops4j.pax.web.service.jetty.internal.JettyServerImpl$2.call(JettyServerImpl.java:420) at org.ops4j.pax.swissbox.core.ContextClassLoaderUtils.doWithClassLoader(ContextClassLoaderUtils.java:60) at org.ops4j.pax.web.service.jetty.internal.JettyServerImpl.addServlet(JettyServerImpl.java:419) at org.ops4j.pax.web.service.jetty.internal.ServerControllerImpl$Started.addServlet(ServerControllerImpl.java:311) at org.ops4j.pax.web.service.jetty.internal.ServerControllerImpl.addServlet(ServerControllerImpl.java:120) at org.ops4j.pax.web.service.internal.HttpServiceStarted.registerServlet(HttpServiceStarted.java:245) at org.ops4j.pax.web.service.internal.HttpServiceStarted.registerServlet(HttpServiceStarted.java:401) at org.ops4j.pax.web.service.internal.HttpServiceProxy.registerServlet(HttpServiceProxy.java:463) at org.opendaylight.aaa.web.osgi.PaxWebServer$WebContextImpl.registerServlet(PaxWebServer.java:200) at org.opendaylight.aaa.web.osgi.PaxWebServer$WebContextImpl.<init>(PaxWebServer.java:173) at org.opendaylight.aaa.web.osgi.PaxWebServer$2$1.<init>(PaxWebServer.java:114) at org.opendaylight.aaa.web.osgi.PaxWebServer$2.registerWebContext(PaxWebServer.java:114) at Proxy7c938f7a_ec90_4982_a4de_c16af07c04dd.registerWebContext(Unknown Source) at org.opendaylight.aaa.shiro.web.env.WebInitializer.<init>(WebInitializer.java:54) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.aries.blueprint.utils.ReflectionUtils.newInstance(ReflectionUtils.java:331) at org.apache.aries.blueprint.container.BeanRecipe.newInstance(BeanRecipe.java:984) at org.apache.aries.blueprint.container.BeanRecipe.getInstanceFromType(BeanRecipe.java:349) ... 22 more Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: com.sun.ws.rs.ext.RuntimeDelegateImpl cannot be found by javax.ws.rs.jsr311-api_1.1.1 at javax.ws.rs.ext.RuntimeDelegate.findDelegate(RuntimeDelegate.java:122) at javax.ws.rs.ext.RuntimeDelegate.getInstance(RuntimeDelegate.java:91) at javax.ws.rs.core.EntityTag.<clinit>(EntityTag.java:35) ... 71 more Caused by: java.lang.ClassNotFoundException: com.sun.ws.rs.ext.RuntimeDelegateImpl cannot be found by javax.ws.rs.jsr311-api_1.1.1 at org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(BundleLoader.java:484) at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:395) at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:387) at org.eclipse.osgi.internal.loader.ModuleClassLoader.loadClass(ModuleClassLoader.java:150) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at javax.ws.rs.ext.FactoryFinder.newInstance(FactoryFinder.java:62) at javax.ws.rs.ext.FactoryFinder.find(FactoryFinder.java:155) at javax.ws.rs.ext.RuntimeDelegate.findDelegate(RuntimeDelegate.java:105) ... 73 more
Unwinding the stack, we are coming in from WebInitializer, then going to jersey-servlet, then to jersey-server and are failing when jsr311-api is asked by the jersey-server to instantiate a class from jersey-client. Note that that value is the default:
Based on the comments in that class, and the fact we second-to-last stack is showing we are entering through:
at javax.ws.rs.ext.RuntimeDelegate.getInstance(RuntimeDelegate.java:91)
it would seem that RuntimeDelegate.setInstance() was not called by jersey-server to bind RuntimeDelegate to its internal implementation (com.sun.jersey.server.impl.provider/RuntimeDelegateImpl) and hence it is scrambling to recover (which may have worked in single-classloader environments).
So delving into jersey, this gem of a package is found:
https://github.com/jersey/jersey-1.x/tree/master/jersey-core/src/main/java/com/sun/jersey/core/osgi
Bottom line, the way I am reading this, is:
if jersey-core cannot find jersey-server bundle (by name!) when its activator runs, it will set the delegate to the client version (if it can load it). If not, it will set it to null.
Now ... all of this is supposedly logged via java.util.logging – do we have that bridged in karaf.log?
As for the fix — we currently are packaging server+core in odl-aaa-api and servlet+client in odl-aaa-shiro. I would suggest bundling the entire mess into a single feature (maybe with a prerequisite) and see what gives.
The race condition is probably that aaa-shiro is starting as soon as it has all required services (e.g. web api) and has jersey packages resolved. There is no guarantee of ordering between aaa-shiro and jersey-{server,core} activators. Hence aaa-shiro can get running before jersey-core is started, leading to an SFT-shattering kaboom. Jersey bundles are not publishing any services, so I fear we will need to either do a prerequisite or code something up.
As for coding: a simple bundle tracker which publishes a dummy service (which gets injected into jersey users) when it sees jersey-core started event should work.
Fun bit, after jersey-2 conversion we got: https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/aaa-maven-verify-fluorine-mvn33-openjdk8/163/console.log.gz
Patches for the Jersey issue:
The integration/distribution project patch is now merged. I'm going to consider this task complete as fluorine is now unblocked by version bumping.
Only 2 remaining projects remain:
- TSDR
- Telemetry
TSDR is blocking telemetry's merge and is currently failing due to test issues.
scottmelton any ideas one what the failure cause is?