https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-1815 controller <p>seeing the following Exception in karaf.log for jobs like <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-pike-upstream-stateful-snat-conntrack-oxygen/184/odl_1/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">this</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2018-02-28T13:43:37,929 | ERROR | features-1-thread-1 | osgi | 186 - org.jolokia.osgi - 1.5.0 | Namespace Exception: org.osgi.service.http.NamespaceException: alias: '/jolokia' is already in use in this or another context org.osgi.service.http.NamespaceException: alias: '/jolokia' is already in use in this or another context at org.ops4j.pax.web.service.spi.model.ServerModel.addServletModel(ServerModel.java:124) ~[?:?] at org.ops4j.pax.web.service.internal.HttpServiceStarted.registerServlet(HttpServiceStarted.java:241) ~[?:?] at org.ops4j.pax.web.service.internal.HttpServiceStarted.registerServlet(HttpServiceStarted.java:221) ~[?:?] at org.ops4j.pax.web.service.internal.HttpServiceStarted.registerServlet(HttpServiceStarted.java:205) ~[?:?] at org.ops4j.pax.web.service.internal.HttpServiceProxy.registerServlet(HttpServiceProxy.java:65) ~[?:?] at org.jolokia.osgi.JolokiaActivator$HttpServiceCustomizer.addingService(JolokiaActivator.java:315) ~[?:?] at org.osgi.util.tracker.ServiceTracker$Tracked.customizerAdding(ServiceTracker.java:941) ~[?:?] at org.osgi.util.tracker.ServiceTracker$Tracked.customizerAdding(ServiceTracker.java:870) ~[?:?] at org.osgi.util.tracker.AbstractTracked.trackAdding(AbstractTracked.java:256) ~[?:?] at org.osgi.util.tracker.AbstractTracked.trackInitial(AbstractTracked.java:183) ~[?:?] at org.osgi.util.tracker.ServiceTracker.open(ServiceTracker.java:318) ~[?:?] at org.osgi.util.tracker.ServiceTracker.open(ServiceTracker.java:261) ~[?:?] at org.jolokia.osgi.JolokiaActivator.start(JolokiaActivator.java:102) ~[?:?] at org.eclipse.osgi.internal.framework.BundleContextImpl$3.run(BundleContextImpl.java:774) ~[?:?] at org.eclipse.osgi.internal.framework.BundleContextImpl$3.run(BundleContextImpl.java:1) ~[?:?] at java.security.AccessController.doPrivileged(Native Method) ~[?:?] at org.eclipse.osgi.internal.framework.BundleContextImpl.startActivator(BundleContextImpl.java:767) ~[?:?] at org.eclipse.osgi.internal.framework.BundleContextImpl.start(BundleContextImpl.java:724) ~[?:?] at org.eclipse.osgi.internal.framework.EquinoxBundle.startWorker0(EquinoxBundle.java:932) ~[?:?] at org.eclipse.osgi.internal.framework.EquinoxBundle$EquinoxModule.startWorker(EquinoxBundle.java:309) ~[?:?] at org.eclipse.osgi.container.Module.doStart(Module.java:581) ~[?:?] at org.eclipse.osgi.container.Module.start(Module.java:449) ~[?:?] at org.eclipse.osgi.internal.framework.EquinoxBundle.start(EquinoxBundle.java:383) ~[?:?] at org.eclipse.osgi.internal.framework.EquinoxBundle.start(EquinoxBundle.java:402) ~[?:?] at org.apache.karaf.features.internal.service.FeaturesServiceImpl.startBundle(FeaturesServiceImpl.java:1361) ~[?:?] at org.apache.karaf.features.internal.service.Deployer.deploy(Deployer.java:888) ~[?:?] at org.apache.karaf.features.internal.service.FeaturesServiceImpl.doProvision(FeaturesServiceImpl.java:1248) ~[?:?] at org.apache.karaf.features.internal.service.FeaturesServiceImpl.lambda$doProvisionInThread$1(FeaturesServiceImpl.java:1147) ~[?:?] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:?] at java.lang.Thread.run(Thread.java:748) [?:?]</pre> </div></div> CONTROLLER-1815

NamespaceException: alias: '/jolokia' is already in use in this or another context

Bug Medium Resolved Cannot Reproduce Michael Vorburger Jamo Luhrsen Thu, 1 Mar 2018 01:05:33 +0000 Tue, 24 Apr 2018 07:29:10 +0000 Mon, 23 Apr 2018 17:44:30 +0000 Oxygen Fluorine Oxygen SR1 Fluorine 0 4 <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=rgoulding" class="user-hover" rel="rgoulding">rgoulding</a> I don't remember seeing this one a little while ago, so I suspect it could have been caused by <a href="https://jira.opendaylight.org/browse/AAA-147" title="odl-jolokia credentials should be centrally backed by AAA" class="issue-link" data-issue-key="AAA-147"><del>AAA-147</del></a> ? That <a href="https://git.opendaylight.org/gerrit/#/c/68070/2/aaa-shiro/impl/pom.xml" class="external-link" target="_blank" rel="nofollow noopener">bumps Jolokia to 1.5.0</a>, which perhaps has something to do with this.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jluhrsen" class="user-hover" rel="jluhrsen">jluhrsen</a> that NamespaceException basically means that we are trying to re-register a Servlet on the /jolokia URI. It should be harmless, but is not nice and an indication of a lifecycle error, somewhere...  minor severity, in the overall scheme of things.</p> <p>From a quick grep in controller, I don't see us registering to /jolokia in our own code (unless some ODL application does; but that's unlikely I guess), so perhaps the <tt><a href="https://github.com/rhuss/jolokia/blob/master/agent/osgi/src/main/java/org/jolokia/osgi/JolokiaActivator.java" class="external-link" target="_blank" rel="nofollow noopener">org.jolokia.osgi.JolokiaActivator</a></tt> started twice, or re-started (feature/bundle re-refresh?), and did not unregister? It does do a httpService.unregister(getServletAlias()) though..</p> <p>Hi,</p> <p>I just saw this using Opendaylight Oxygen (from Ubuntu PPA) in Armband Fuel@OPNFV.</p> <p>Unfortunately, the karaf.log reports a lot of other errors, so I'm not sure this is really useful for debugging, but for the sake of completeness, here's the log <span class="error">&#91;1&#93;</span>.</p> <p>It is highly likely this is related to another issue we had with Fuel@OPNFV deploying ODL, where we used to restart the ODL service while it was still booting, leaving behind corrupted cache data (the fix for this was just merged today, so it's not included in the above log - we mask the ODL service and only enable&amp;start it after all configuration is done).</p> <p><span class="error">&#91;1&#93;</span> <a href="http://paste.ubuntu.com/p/4Bwg4Z6k5X/" class="external-link" target="_blank" rel="nofollow noopener">http://paste.ubuntu.com/p/4Bwg4Z6k5X/</a></p> <p>So I've looked more into this one today, and tried to locally reproduce it... here are some conclusions, and next step:</p> <p>I cannot reproduce this in <tt>netvirt/karaf</tt> from today's master, after <tt>feature:install odl-netvirt-openstack</tt>. But re-reading <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=rgoulding" class="user-hover" rel="rgoulding">rgoulding</a> comment in <a href="https://lists.opendaylight.org/pipermail/aaa-dev/2018-April/001650.html" class="external-link" target="_blank" rel="nofollow noopener">https://lists.opendaylight.org/pipermail/aaa-dev/2018-April/001650.html</a>, I got what's really going on here:</p> <p>This happens if, for some reason, we have the Karaf core <tt>jolokia</tt> feature installed before our own <tt>odl-jolokia</tt> feature. Indeed in a <tt>netvirt/karaf</tt> from today's master where this is not happening, you would see, which is correct:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>karaf@root()&gt; feature:list | grep jolokia odl-jolokia │ 1.11.0.SNAPSHOT │ │ Started │ odl-extras-1.11.0-SNAPSHOT │ Jolokia JMX/HTTP bridge jolokia │ 1.3.5 │ │ Uninstalled │ standard-4.1.5 │ Jolokia monitoring support</pre> </div></div> <p>But indeed (only) if we <tt>feature:install jolokia odl-netvirt-openstack</tt>, THEN we can reproduce this issue!</p> <p>&gt; It should be harmless, but is not nice and an indication of a lifecycle error,</p> <p>just to "qualify" my earlier "harmless" comment with a bit more background now: While this will not impact core SDN functionality, it does mean that the "wrong" Jolokia is installed, and e.g. /jolokia will not be protected by AAA. If one never uses /jolokia for JMX-over-HTTP monitoring, and is not overly security paranoid, then that is not a huge issue.</p> <p>So the real question now is just to find out why on earth in some distributions it's the <tt>jolokia</tt> instead of the <tt>odl-jolokia</tt> feature that gets installed?! Is this wrongly hard-coded in some ... place, upstream or downstream installation orchestration, somewhere? It should not be so anywhere, because e.g. <tt>odl-netvirt-openstack</tt> (indirectly, same applies for other end-user application features) seems to correctly depend on <tt>odl-jolokia</tt> already.</p> <p>Two next stepss: 1. I'll have a closer look at <tt>integration/distribution/karaf</tt> if that is wrong, 2. I'll email some people asking if there is a <tt>jolokia</tt> instead of the <tt>odl-jolokia</tt> hard-coded somewhere.</p> <p>&gt; closer look at <tt>integration/distribution/karaf</tt> </p> <p>It's the same as with <tt>netvirt/karaf</tt> here - it works out of the box.</p> <p>So either this is different in releases based off older code than today's Fluorine master (but I doubt that), or something else is wrong to explicitly install the <tt>jolokia</tt> Karaf feature... we now just need to find out what! I am suspecting some CSIT script? But <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1569665" class="external-link" target="_blank" rel="nofollow noopener">we're also hearing about this downstream now</a>, not just in upstream CSIT, so... perhaps there is also a hard-coded wrong explicit <tt>jolokia</tt> Karaf feature install in... TripleO?</p> <p>PS, just for the record: I've also, briefly, considered some hack to "remove" the <tt>jolokia</tt> Karaf feature all together (<a href="https://issues.apache.org/jira/browse/KARAF-5376?focusedCommentId=16431939&amp;page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16431939" class="external-link" target="_blank" rel="nofollow noopener">based on this</a>, or documentation which may be written in KARAF-5540 in the future), but concluded that actually won't really solve our immediate issue here (it would just cause something to fail, not fix anything), so I would rather focus efforts on finding the root cause instead.</p> <p>&gt; either this is different in releases based off older code than today's Fluorine master (but I doubt that)</p> <p>Verified this by testing above against <a href="https://www.opendaylight.org/technical-community/getting-started-for-developers/downloads-and-documentation" class="external-link" target="_blank" rel="nofollow noopener">the Oxygen release from March 22</a>, and noticed an interesting disprecancy:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>opendaylight-user@root&gt;feature:list | grep jolokia odl-jolokia │ 1.10.0 │ │ Uninstalled │ odl-extras-1.10.0 │ Jolokia JMX/HTTP bridge jolokia │ 1.3.5 │ │ Uninstalled │ standard-4.1.3 │ Jolokia monitoring support opendaylight-user@root&gt;feature:install odl-netvirt-openstack opendaylight-user@root&gt;feature:list | grep jolokia odl-jolokia │ 1.10.0 │ │ Uninstalled │ odl-extras-1.10.0 │ Jolokia JMX/HTTP bridge jolokia │ 1.3.5 │ │ Uninstalled │ standard-4.1.3 │ Jolokia monitoring support</pre> </div></div> <p>so it would seem that in the past Jolokia (neither the wrong nor the right one) got automatically installed as a dependency... which reinforces my suspicion that we probably hard-coded this in the past - but we (always, most likely) had chosen the wrong one. The right thing to do now is to just remove any mention of jolokia and let it get installed automatically as a dependency.</p> <p>Same in a local <tt>netvirt/karaf</tt> build of today's <tt>stable/oxygen</tt> branch, which is about to be released as Oxygen SP1:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>karaf@root()&gt; feature:list | grep jolokia odl-jolokia │ 1.10.1.SNAPSHOT │ │ Uninstalled │ odl-extras-1.10.1-SNAPSHOT │ Jolokia JMX/HTTP bridge jolokia │ 1.3.5 │ │ Uninstalled │ standard-4.1.5 │ Jolokia monitoring support karaf@root()&gt; feature:install odl-netvirt-openstack karaf@root()&gt; feature:list | grep jolokia odl-jolokia │ 1.10.1.SNAPSHOT │ │ Uninstalled │ odl-extras-1.10.1-SNAPSHOT │ Jolokia JMX/HTTP bridge jolokia │ 1.3.5 │ │ Uninstalled │ standard-4.1.5 │ Jolokia monitoring support karaf@root()&gt; log:display | grep /jolokia</pre> </div></div> <p>So while for master (future Fluorine) we seem to have a dependy to odl-jolokia somewhere, for Oxygen you can just explicitly install the <tt>odl-jolokia</tt> feature (but NOT the one similarly named just <tt>jolokia</tt>!), e.g. like so, and it's all good:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>karaf@root()&gt; feature:install odl-jolokia odl-netvirt-openstack karaf@root()&gt; feature:list | grep jolokia odl-jolokia │ 1.10.1.SNAPSHOT │ x │ Started │ odl-extras-1.10.1-SNAPSHOT │ Jolokia JMX/HTTP bridge jolokia │ 1.3.5 │ │ Uninstalled │ standard-4.1.5 │ Jolokia monitoring support karaf@root()&gt; log:display | grep /jolokia karaf@root()&gt;</pre> </div></div> <p><img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/sad.png" height="16" width="16" align="absmiddle" alt="" border="0"/>  I'm lost because I'm probably just stupid. can you TL;DR it for me?</p> <p>You're not stupid, I was beeing detailed and lengthy above <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/> so here's an attempt at a TL;DR:</p> <p>1. The problem shown above is not reproducible after feature:install odl-netvirt-openstack in integration/distribution/karaf and netvirt/karaf on current master and stable/oxygen.</p> <p>2. The problem IS easily reproduced if one installs the Karaf feature named "jolokia" (not "odl-jolokia") before installing odl-netvirt-openstack on master (where odl-netvirt-openstack also installs odl-jolokia), or both "jolokia" and "odl-jolokia" on Oxygen.</p> <p>3. Therefore, my conclusion is that "something" in CSIT <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/help_16.png" height="16" width="16" align="absmiddle" alt="" border="0"/> is manually installing the Karaf "jolokia" (not "odl-jolokia") feature? Perhaps you could add a "feature:list | grep jolokia" at the start of CSIT so that we could have more certainty about this, if there is doubt?</p> <p>Does this help? For more details, read above... <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/wink.png" height="16" width="16" align="absmiddle" alt="" border="0"/> fair? <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/wink.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>PS, FYI: On master, odl-netvirt-openstack also installs odl-jolokia; on stable/oxygen it does not. On neither I can see the Karaf "jolokia" feature (not "odl-jolokia") automatically installed by odl-netvirt-openstack.</p> <p>&gt; Perhaps you could add a "feature:list | grep jolokia"</p> <p>even better, based on something I just learnt from <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=skitt" class="user-hover" rel="skitt">skitt</a>, please do this for me so that we can get to the bottom of this:</p> <p>Please attach what this outputs for in an environment where this problem is seen:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>karaf@root()&gt; feature:list -r</pre> </div></div> <p>and then please attach output of "<tt>feature:info -t $FN</tt>" for N times for each line (feature shown), where FN is the name of 1 feature, say:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>feature:info -t odl-mdsal-trace feature:info -t odl-netvirt-openstack ...</pre> </div></div> <p><a href="https://bugzilla.redhat.com/show_bug.cgi?id=1569665#c13" class="external-link" target="_blank" rel="nofollow noopener">Wait a second...</a> the theory (above) re. the Karaf features named "jolokia" and "odl-jolokia" may be the completely wrong explanation... Or rather, that would also cause this same problem, but is perhaps not actually the cause in this instance!</p> <p>Why? Because what I've missed before is that in <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-pike-upstream-stateful-snat-conntrack-oxygen/184/odl_1/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">this</a> log originally linked above, there is this beauty worth noting:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2018-02-28T13:43:16,314 | INFO | features-1-thread-1 | FeaturesServiceImpl | 7 - org.apache.karaf.features.core - 4.1.3 | mvn:org.jolokia/jolokia-osgi/1.3.7 2018-02-28T13:43:16,314 | INFO | features-1-thread-1 | FeaturesServiceImpl | 7 - org.apache.karaf.features.core - 4.1.3 | mvn:org.jolokia/jolokia-osgi/1.5.0 (...) 2018-02-28T13:43:16,339 | INFO | features-1-thread-1 | FeaturesServiceImpl | 7 - org.apache.karaf.features.core - 4.1.3 | wrap:mvn:org.jolokia/jolokia-core/1.3.7 2018-02-28T13:43:16,339 | INFO | features-1-thread-1 | FeaturesServiceImpl | 7 - org.apache.karaf.features.core - 4.1.3 | wrap:mvn:org.jolokia/jolokia-core/1.5.0 2018-02-28T13:43:16,339 | INFO | features-1-thread-1 | FeaturesServiceImpl | 7 - org.apache.karaf.features.core - 4.1.3 | wrap:mvn:org.jolokia/jolokia-jsr160/1.3.7 2018-02-28T13:43:16,339 | INFO | features-1-thread-1 | FeaturesServiceImpl | 7 - org.apache.karaf.features.core - 4.1.3 | wrap:mvn:org.jolokia/jolokia-jsr160/1.5.0 (...) 2018-02-28T13:43:37,363 | INFO | features-1-thread-1 | FeaturesServiceImpl | 7 - org.apache.karaf.features.core - 4.1.3 | wrap_file__tmp_karaf-0.8.0-SNAPSHOT_system_org_jolokia_jolokia-core_1.3.7_jolokia-core-1.3.7.jar/0.0.0 2018-02-28T13:43:37,364 | INFO | features-1-thread-1 | FeaturesServiceImpl | 7 - org.apache.karaf.features.core - 4.1.3 | com.google.gson/2.8.2 2018-02-28T13:43:37,365 | INFO | features-1-thread-1 | FeaturesServiceImpl | 7 - org.apache.karaf.features.core - 4.1.3 | org.apache.oltu.oauth2.authzserver/1.0.2 2018-02-28T13:43:37,366 | INFO | features-1-thread-1 | FeaturesServiceImpl | 7 - org.apache.karaf.features.core - 4.1.3 | org.h2/1.4.193 2018-02-28T13:43:37,379 | INFO | features-1-thread-1 | FeaturesServiceImpl | 7 - org.apache.karaf.features.core - 4.1.3 | wrap_file__tmp_karaf-0.8.0-SNAPSHOT_system_org_jolokia_jolokia-core_1.5.0_jolokia-core-1.5.0.jar/0.0.0 (...) 2018-02-28T13:43:37,391 | INFO | features-1-thread-1 | FeaturesServiceImpl | 7 - org.apache.karaf.features.core - 4.1.3 | org.jolokia.osgi/1.5.0 2018-02-28T13:43:37,398 | INFO | features-1-thread-1 | HttpServiceFactoryImpl | 450 - org.ops4j.pax.web.pax-web-runtime - 6.0.7 | Binding bundle: [org.jolokia.osgi_1.5.0 [186]] to http service (...) 2018-02-28T13:43:37,609 | INFO | features-1-thread-1 | HttpServiceContext | 448 - org.ops4j.pax.web.pax-web-jetty - 6.0.7 | registering context DefaultHttpContext [bundle=org.jolokia.osgi_1.5.0 [186], contextID=custom], with context-name: 2018-02-28T13:43:37,811 | INFO | features-1-thread-1 | osgi | 186 - org.jolokia.osgi - 1.5.0 | No access restrictor found, access to any MBean is allowed 2018-02-28T13:43:37,856 | INFO | features-1-thread-1 | ContextHandler | 164 - org.eclipse.jetty.util - 9.3.14.v20161028 | Started HttpServiceContext{httpContext=DefaultHttpContext [bundle=org.jolokia.osgi_1.5.0 [186], contextID=custom]} (...) 2018-02-28T13:43:37,922 | INFO | features-1-thread-1 | FeaturesServiceImpl | 7 - org.apache.karaf.features.core - 4.1.3 | org.jolokia.osgi/1.3.7 2018-02-28T13:43:37,927 | INFO | features-1-thread-1 | HttpServiceFactoryImpl | 450 - org.ops4j.pax.web.pax-web-runtime - 6.0.7 | Binding bundle: [org.jolokia.osgi_1.3.7 [185]] to http service 2018-02-28T13:43:37,929 | ERROR | features-1-thread-1 | osgi | 186 - org.jolokia.osgi - 1.5.0 | Namespace Exception: org.osgi.service.http.NamespaceException: alias: '/jolokia' is already in use in this or another context org.osgi.service.http.NamespaceException: alias: '/jolokia' is already in use in this or another context </pre> </div></div> <p>Looking carefully above, there is something curious - <b>what's this 1.3.7 VS 1.5.0 business? We shouldn't have 2 versions of Jolokia bundles not features!</b></p> <p>To be 100% sure that it's related to that, could I ask for, in addition to above, also getting the output of:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>bundle:list -s -l</pre> </div></div> <p>I was looking to update a CSIT job to collect this info, but realized we aren't seeing this Exception in a recent CSIT<br/> job. Like this <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-pike-upstream-stateful-snat-conntrack-oxygen/260/odl_1/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">one</a></p> <p>you want this info on a recent distro, right? in other words, on a distro that is not showing this problem?</p> <p>&gt; I was looking to update a CSIT job to collect this info, but realized we aren't seeing this Exception in a recent CSIT job. Like this <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-pike-upstream-stateful-snat-conntrack-oxygen/260/odl_1/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">one</a></p> <p>Very interesting. And in that new log - guess what? There is only jolokia-core|jsr160|osgi 1.5.0, only 1.3.7 - so that's it! So looks like something during "the big bump" which happened some time after April 10th (<a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jluhrsen" class="user-hover" rel="jluhrsen">jluhrsen</a> IRC: "I think Apr 10th is the last time this showed up in that job."), most likely the Karaf upgrade that was part of that bump, did the trick here.</p> <p>For the record, I was seeing this 100% of the time with Oxygen SR0, not anymore with Oxygen SR1 (well, the daily build from 22 Apr to be accurate).</p> <p>Feature list for working (22 Apr) - text to long to paste here (pasted one more, previous version was trimmed too):</p> <p><a href="https://paste.ubuntu.com/p/jWGFS5BhrR/" class="external-link" target="_blank" rel="nofollow noopener">https://paste.ubuntu.com/p/jWGFS5BhrR/</a></p> <p>No forget about that, that from (just minutes) before I figured it out; this is no longer needed.</p> <p>&gt; For the record, I was seeing this 100% of the time with Oxygen SR0, not anymore with Oxygen SR1 (well, the daily build from 22 Apr to be accurate).</p> <p>Great, that confirms above - because "the big bump" with a Karaf upgrade also happened on Oxygen SR1.</p> <p>I'm therefore now closing this issue as CANNOT REPRODUCE, in the sense that it is no longer seen on the latest Fluorine, and confirmed fixed on Oxygen SR1.</p> <p>FTR: <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1569665" class="external-link" target="_blank" rel="nofollow noopener">https://bugzilla.redhat.com/show_bug.cgi?id=1569665</a> continues the discussion for a certain red vendor's downstream Oxygen (not yet SR1) based package, but that is no longer relevant to bleeding edge master nor stable/oxygen maintenance in this upstream JIRA.</p> <p>Yes, this is one of the bugs we fixed with the odlparent 3.1 bump, along with <a href="https://jira.opendaylight.org/browse/AAA-147" title="odl-jolokia credentials should be centrally backed by AAA" class="issue-link" data-issue-key="AAA-147"><del>AAA-147</del></a>.</p> Development Rank 0|i03azj: