https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/ODLPARENT-82 odlparent <p>This blocks Karaf 4 work on jobs related to distribution, for example all <del>csit</del> jobs.</p> <p>Issuing "feature:install odl-restconf" succeeds, but subsequent Restconf queries do not work.<br/> More specifically, HTTP connection stays open for a few minutes, and then ODL closes the connection.</p> <p>Possible causes are described in e-mail <span class="error">&#91;2&#93;</span>.</p> <p><span class="error">&#91;2&#93;</span> <a href="https://lists.opendaylight.org/pipermail/odlparent-dev/2017-March/000895.html" class="external-link" target="_blank" rel="nofollow noopener">https://lists.opendaylight.org/pipermail/odlparent-dev/2017-March/000895.html</a></p> <p>Operating System: All<br/> Platform: All</p> ODLPARENT-82

Restconf does not start in Karaf 4

Bug Resolved Done Robert Varga Vratko Polak Thu, 30 Mar 2017 17:33:15 +0000 Wed, 24 Jan 2018 14:27:05 +0000 Mon, 10 Apr 2017 21:59:00 +0000 General 0 3 <p>opendaylight-user@root&gt;bundle:list | grep -v Active<br/> START LEVEL 100 , List Threshold: 50<br/> ID | State | Lvl | Version | Name<br/> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------<br/> 192 | Resolved | 80 | 0.6.0.SNAPSHOT | config-persister-directory-xml-adapter, Hosts: 194<br/> 193 | Resolved | 80 | 0.6.0.SNAPSHOT | config-persister-file-xml-adapter, Hosts: 194</p> <p>First warning in karaf.log:</p> <p>2017-03-30 17:37:05,772 | WARN | ender-2-thread-1 | JettyServerWrapper | 292 - org.ops4j.pax.web.pax-web-jetty - 4.3.0 | Failed to find annotated classes for ServletContainerInitializer<br/> java.lang.RuntimeException<br/> at org.objectweb.asm.MethodVisitor.visitParameter(Unknown Source)<span class="error">&#91;166:org.objectweb.asm:5.0.3&#93;</span><br/> at org.objectweb.asm.ClassReader.b(Unknown Source)<span class="error">&#91;166:org.objectweb.asm:5.0.3&#93;</span><br/> at org.objectweb.asm.ClassReader.accept(Unknown Source)<span class="error">&#91;166:org.objectweb.asm:5.0.3&#93;</span><br/> at org.objectweb.asm.ClassReader.accept(Unknown Source)<span class="error">&#91;166:org.objectweb.asm:5.0.3&#93;</span><br/> at org.apache.xbean.finder.AbstractFinder.readClassDef(AbstractFinder.java:581)<span class="error">&#91;132:org.apache.xbean.finder:4.1.0&#93;</span><br/> at org.apache.xbean.finder.BundleAnnotationFinder$AnnotationFindingCallback.foundInDirectory(BundleAnnotationFinder.java:86)<span class="error">&#91;132:org.apache.xbean.finder:4.1.0&#93;</span><br/> at org.apache.xbean.osgi.bundle.util.BundleResourceFinder.scanDirectory(BundleResourceFinder.java:151)<span class="error">&#91;131:org.apache.xbean.bundleutils:4.1.0&#93;</span><br/> at org.apache.xbean.osgi.bundle.util.BundleResourceFinder.scanBundleClassPath(BundleResourceFinder.java:120)<span class="error">&#91;131:org.apache.xbean.bundleutils:4.1.0&#93;</span><br/> at org.apache.xbean.osgi.bundle.util.BundleResourceFinder.find(BundleResourceFinder.java:88)<span class="error">&#91;131:org.apache.xbean.bundleutils:4.1.0&#93;</span><br/> at org.apache.xbean.finder.BundleAnnotationFinder.&lt;init&gt;(BundleAnnotationFinder.java:56)<span class="error">&#91;132:org.apache.xbean.finder:4.1.0&#93;</span><br/> at org.apache.xbean.finder.BundleAnnotationFinder.&lt;init&gt;(BundleAnnotationFinder.java:50)<span class="error">&#91;132:org.apache.xbean.finder:4.1.0&#93;</span><br/> at org.apache.xbean.finder.BundleAnnotationFinder.&lt;init&gt;(BundleAnnotationFinder.java:46)<span class="error">&#91;132:org.apache.xbean.finder:4.1.0&#93;</span><br/> at org.ops4j.pax.web.service.jetty.internal.JettyServerWrapper.addContext(JettyServerWrapper.java:352)<span class="error">&#91;292:org.ops4j.pax.web.pax-web-jetty:4.3.0&#93;</span><br/> at org.ops4j.pax.web.service.jetty.internal.JettyServerWrapper.getOrCreateContext(JettyServerWrapper.java:219)<span class="error">&#91;292:org.ops4j.pax.web.pax-web-jetty:4.3.0&#93;</span><br/> at org.ops4j.pax.web.service.jetty.internal.JettyServerImpl.getContext(JettyServerImpl.java:267)<span class="error">&#91;292:org.ops4j.pax.web.pax-web-jetty:4.3.0&#93;</span><br/> at org.ops4j.pax.web.service.jetty.internal.ServerControllerImpl$Started.getContext(ServerControllerImpl.java:347)<span class="error">&#91;292:org.ops4j.pax.web.pax-web-jetty:4.3.0&#93;</span><br/> at org.ops4j.pax.web.service.jetty.internal.ServerControllerImpl.getContext(ServerControllerImpl.java:176)<span class="error">&#91;292:org.ops4j.pax.web.pax-web-jetty:4.3.0&#93;</span><br/> at org.ops4j.pax.web.service.internal.HttpServiceStarted.begin(HttpServiceStarted.java:1079)<span class="error">&#91;294:org.ops4j.pax.web.pax-web-runtime:4.3.0&#93;</span><br/> at org.ops4j.pax.web.service.internal.HttpServiceProxy.begin(HttpServiceProxy.java:412)<span class="error">&#91;294:org.ops4j.pax.web.pax-web-runtime:4.3.0&#93;</span><br/> at org.ops4j.pax.web.extender.war.internal.RegisterWebAppVisitorWC.visit(RegisterWebAppVisitorWC.java:176)<span class="error">&#91;290:org.ops4j.pax.web.pax-web-extender-war:4.3.0&#93;</span><br/> at org.ops4j.pax.web.extender.war.internal.model.WebApp.accept(WebApp.java:655)<span class="error">&#91;290:org.ops4j.pax.web.pax-web-extender-war:4.3.0&#93;</span><br/> at org.ops4j.pax.web.extender.war.internal.WebAppPublisher$WebAppDependencyListener.register(WebAppPublisher.java:237)<span class="error">&#91;290:org.ops4j.pax.web.pax-web-extender-war:4.3.0&#93;</span><br/> at org.ops4j.pax.web.extender.war.internal.WebAppPublisher$WebAppDependencyListener.addingService(WebAppPublisher.java:182)<span class="error">&#91;290:org.ops4j.pax.web.pax-web-extender-war:4.3.0&#93;</span><br/> at org.ops4j.pax.web.extender.war.internal.WebAppPublisher$WebAppDependencyListener.addingService(WebAppPublisher.java:135)<span class="error">&#91;290:org.ops4j.pax.web.pax-web-extender-war:4.3.0&#93;</span><br/> at org.osgi.util.tracker.ServiceTracker$Tracked.customizerAdding(ServiceTracker.java:941)<span class="error">&#91;org.osgi.core-6.0.0.jar:&#93;</span><br/> at org.osgi.util.tracker.ServiceTracker$Tracked.customizerAdding(ServiceTracker.java:870)<span class="error">&#91;org.osgi.core-6.0.0.jar:&#93;</span><br/> at org.osgi.util.tracker.AbstractTracked.trackAdding(AbstractTracked.java:256)<span class="error">&#91;org.osgi.core-6.0.0.jar:&#93;</span><br/> at org.osgi.util.tracker.AbstractTracked.trackInitial(AbstractTracked.java:183)<span class="error">&#91;org.osgi.core-6.0.0.jar:&#93;</span><br/> at org.osgi.util.tracker.ServiceTracker.open(ServiceTracker.java:318)<span class="error">&#91;org.osgi.core-6.0.0.jar:&#93;</span><br/> at org.osgi.util.tracker.ServiceTracker.open(ServiceTracker.java:261)<span class="error">&#91;org.osgi.core-6.0.0.jar:&#93;</span><br/> at org.ops4j.pax.web.extender.war.internal.WebAppPublisher.publish(WebAppPublisher.java:101)<span class="error">&#91;290:org.ops4j.pax.web.pax-web-extender-war:4.3.0&#93;</span><br/> at org.ops4j.pax.web.extender.war.internal.WebObserver.deploy(WebObserver.java:219)<span class="error">&#91;290:org.ops4j.pax.web.pax-web-extender-war:4.3.0&#93;</span><br/> at org.ops4j.pax.web.extender.war.internal.WebObserver$1.doStart(WebObserver.java:179)<span class="error">&#91;290:org.ops4j.pax.web.pax-web-extender-war:4.3.0&#93;</span><br/> at org.ops4j.pax.web.extender.war.internal.extender.SimpleExtension.start(SimpleExtension.java:59)<span class="error">&#91;290:org.ops4j.pax.web.pax-web-extender-war:4.3.0&#93;</span><br/> at org.ops4j.pax.web.extender.war.internal.extender.AbstractExtender$1.run(AbstractExtender.java:281)<span class="error">&#91;290:org.ops4j.pax.web.pax-web-extender-war:4.3.0&#93;</span><br/> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)<span class="error">&#91;:1.8.0_121&#93;</span><br/> at java.util.concurrent.FutureTask.run(FutureTask.java:266)<span class="error">&#91;:1.8.0_121&#93;</span><br/> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)<span class="error">&#91;:1.8.0_121&#93;</span><br/> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)<span class="error">&#91;:1.8.0_121&#93;</span><br/> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)<span class="error">&#91;:1.8.0_121&#93;</span><br/> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)<span class="error">&#91;:1.8.0_121&#93;</span><br/> at java.lang.Thread.run(Thread.java:745)<span class="error">&#91;:1.8.0_121&#93;</span></p> <p>This looks worse:</p> <p>2017-03-30 17:37:07,692 | WARN | ender-2-thread-1 | AbstractLifeCycle | 148 - org.eclipse.jetty.util - 9.2.19.v20160908 | FAILED org.ops4j.pax.web.service.jetty.internal.JettyServerWrapper@192dae03: java.net.BindException: Address already in use<br/> java.net.BindException: Address already in use<br/> at sun.nio.ch.Net.bind0(Native Method)<span class="error">&#91;:1.8.0_121&#93;</span><br/> at sun.nio.ch.Net.bind(Net.java:433)<span class="error">&#91;:1.8.0_121&#93;</span><br/> at sun.nio.ch.Net.bind(Net.java:425)<span class="error">&#91;:1.8.0_121&#93;</span><br/> at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)<span class="error">&#91;:1.8.0_121&#93;</span><br/> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)<span class="error">&#91;:1.8.0_121&#93;</span><br/> at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:321)<span class="error">&#91;145:org.eclipse.jetty.server:9.2.19.v20160908&#93;</span><br/> at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)<span class="error">&#91;145:org.eclipse.jetty.server:9.2.19.v20160908&#93;</span><br/> at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:236)<span class="error">&#91;145:org.eclipse.jetty.server:9.2.19.v20160908&#93;</span><br/> at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)<span class="error">&#91;148:org.eclipse.jetty.util:9.2.19.v20160908&#93;</span><br/> at org.eclipse.jetty.server.Server.doStart(Server.java:366)<span class="error">&#91;145:org.eclipse.jetty.server:9.2.19.v20160908&#93;</span><br/> at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)<span class="error">&#91;148:org.eclipse.jetty.util:9.2.19.v20160908&#93;</span><br/> at org.ops4j.pax.web.service.jetty.internal.JettyServerImpl$1.start(JettyServerImpl.java:287)<span class="error">&#91;292:org.ops4j.pax.web.pax-web-jetty:4.3.0&#93;</span><br/> at org.ops4j.pax.web.service.internal.HttpServiceStarted.end(HttpServiceStarted.java:1096)<span class="error">&#91;294:org.ops4j.pax.web.pax-web-runtime:4.3.0&#93;</span><br/> at org.ops4j.pax.web.service.internal.HttpServiceProxy.end(HttpServiceProxy.java:417)<span class="error">&#91;294:org.ops4j.pax.web.pax-web-runtime:4.3.0&#93;</span><br/> at org.ops4j.pax.web.extender.war.internal.RegisterWebAppVisitorWC.end(RegisterWebAppVisitorWC.java:380)<span class="error">&#91;290:org.ops4j.pax.web.pax-web-extender-war:4.3.0&#93;</span><br/> at org.ops4j.pax.web.extender.war.internal.model.WebApp.accept(WebApp.java:692)<span class="error">&#91;290:org.ops4j.pax.web.pax-web-extender-war:4.3.0&#93;</span><br/> at org.ops4j.pax.web.extender.war.internal.WebAppPublisher$WebAppDependencyListener.register(WebAppPublisher.java:237)<span class="error">&#91;290:org.ops4j.pax.web.pax-web-extender-war:4.3.0&#93;</span><br/> at org.ops4j.pax.web.extender.war.internal.WebAppPublisher$WebAppDependencyListener.addingService(WebAppPublisher.java:182)<span class="error">&#91;290:org.ops4j.pax.web.pax-web-extender-war:4.3.0&#93;</span><br/> at org.ops4j.pax.web.extender.war.internal.WebAppPublisher$WebAppDependencyListener.addingService(WebAppPublisher.java:135)<span class="error">&#91;290:org.ops4j.pax.web.pax-web-extender-war:4.3.0&#93;</span><br/> at org.osgi.util.tracker.ServiceTracker$Tracked.customizerAdding(ServiceTracker.java:941)<span class="error">&#91;org.osgi.core-6.0.0.jar:&#93;</span><br/> at org.osgi.util.tracker.ServiceTracker$Tracked.customizerAdding(ServiceTracker.java:870)<span class="error">&#91;org.osgi.core-6.0.0.jar:&#93;</span><br/> at org.osgi.util.tracker.AbstractTracked.trackAdding(AbstractTracked.java:256)<span class="error">&#91;org.osgi.core-6.0.0.jar:&#93;</span><br/> at org.osgi.util.tracker.AbstractTracked.trackInitial(AbstractTracked.java:183)<span class="error">&#91;org.osgi.core-6.0.0.jar:&#93;</span><br/> at org.osgi.util.tracker.ServiceTracker.open(ServiceTracker.java:318)<span class="error">&#91;org.osgi.core-6.0.0.jar:&#93;</span><br/> at org.osgi.util.tracker.ServiceTracker.open(ServiceTracker.java:261)<span class="error">&#91;org.osgi.core-6.0.0.jar:&#93;</span><br/> at org.ops4j.pax.web.extender.war.internal.WebAppPublisher.publish(WebAppPublisher.java:101)<span class="error">&#91;290:org.ops4j.pax.web.pax-web-extender-war:4.3.0&#93;</span><br/> at org.ops4j.pax.web.extender.war.internal.WebObserver.deploy(WebObserver.java:219)<span class="error">&#91;290:org.ops4j.pax.web.pax-web-extender-war:4.3.0&#93;</span><br/> at org.ops4j.pax.web.extender.war.internal.WebObserver$1.doStart(WebObserver.java:179)<span class="error">&#91;290:org.ops4j.pax.web.pax-web-extender-war:4.3.0&#93;</span><br/> at org.ops4j.pax.web.extender.war.internal.extender.SimpleExtension.start(SimpleExtension.java:59)<span class="error">&#91;290:org.ops4j.pax.web.pax-web-extender-war:4.3.0&#93;</span><br/> at org.ops4j.pax.web.extender.war.internal.extender.AbstractExtender$1.run(AbstractExtender.java:281)<span class="error">&#91;290:org.ops4j.pax.web.pax-web-extender-war:4.3.0&#93;</span><br/> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)<span class="error">&#91;:1.8.0_121&#93;</span><br/> at java.util.concurrent.FutureTask.run(FutureTask.java:266)<span class="error">&#91;:1.8.0_121&#93;</span><br/> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)<span class="error">&#91;:1.8.0_121&#93;</span><br/> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)<span class="error">&#91;:1.8.0_121&#93;</span><br/> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)<span class="error">&#91;:1.8.0_121&#93;</span><br/> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)<span class="error">&#91;:1.8.0_121&#93;</span><br/> at java.lang.Thread.run(Thread.java:745)<span class="error">&#91;:1.8.0_121&#93;</span></p> <p>Not only restconf related, this is present in the logs even when any other feature is installed(i observed this with odl-netconf-topology) seems like jetty has some issues.</p> <p>The problem seems be coming from pax-web's integration of jetty, more specifically the code invoked around <a href="https://github.com/ops4j/org.ops4j.pax.web/blob/pax-web-4.3.x/pax-web-jetty/src/main/java/org/ops4j/pax/web/service/jetty/internal/ServerControllerImpl.java#L422" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/ops4j/org.ops4j.pax.web/blob/pax-web-4.3.x/pax-web-jetty/src/main/java/org/ops4j/pax/web/service/jetty/internal/ServerControllerImpl.java#L422</a></p> <p>I think the culprit is that our jetty.xml does not declare the port number, but only specifies the default (or something similar), leading to <a href="https://github.com/ops4j/org.ops4j.pax.web/blob/pax-web-4.3.x/pax-web-jetty/src/main/java/org/ops4j/pax/web/service/jetty/internal/ServerControllerImpl.java#L580" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/ops4j/org.ops4j.pax.web/blob/pax-web-4.3.x/pax-web-jetty/src/main/java/org/ops4j/pax/web/service/jetty/internal/ServerControllerImpl.java#L580</a> failing the compare, which triggers the code path <a href="https://github.com/ops4j/org.ops4j.pax.web/blob/pax-web-4.3.x/pax-web-jetty/src/main/java/org/ops4j/pax/web/service/jetty/internal/ServerControllerImpl.java#L461" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/ops4j/org.ops4j.pax.web/blob/pax-web-4.3.x/pax-web-jetty/src/main/java/org/ops4j/pax/web/service/jetty/internal/ServerControllerImpl.java#L461</a></p> <p>My theory is that this causes two connectors to be configured on the same port, leading the the failure.</p> <p>master: <a href="https://git.opendaylight.org/gerrit/54611" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/54611</a></p> Blocks ODLPARENT-23 Development External issue ID 8117 External issue URL Issue Type ODL SR Target Milestone Rank 0|i02373: