rgoulding I don't remember seeing this one a little while ago, so I suspect it could have been caused by AAA-147 ? That bumps Jolokia to 1.5.0, which perhaps has something to do with this.
jluhrsen that NamespaceException basically means that we are trying to re-register a Servlet on the /jolokia URI. It should be harmless, but is not nice and an indication of a lifecycle error, somewhere... minor severity, in the overall scheme of things.
From a quick grep in controller, I don't see us registering to /jolokia in our own code (unless some ODL application does; but that's unlikely I guess), so perhaps the org.jolokia.osgi.JolokiaActivator started twice, or re-started (feature/bundle re-refresh?), and did not unregister? It does do a httpService.unregister(getServletAlias()) though..
Hi,
I just saw this using Opendaylight Oxygen (from Ubuntu PPA) in Armband Fuel@OPNFV.
Unfortunately, the karaf.log reports a lot of other errors, so I'm not sure this is really useful for debugging, but for the sake of completeness, here's the log [1].
It is highly likely this is related to another issue we had with Fuel@OPNFV deploying ODL, where we used to restart the ODL service while it was still booting, leaving behind corrupted cache data (the fix for this was just merged today, so it's not included in the above log - we mask the ODL service and only enable&start it after all configuration is done).
So I've looked more into this one today, and tried to locally reproduce it... here are some conclusions, and next step:
I cannot reproduce this in netvirt/karaf from today's master, after feature:install odl-netvirt-openstack. But re-reading rgoulding comment in https://lists.opendaylight.org/pipermail/aaa-dev/2018-April/001650.html, I got what's really going on here:
This happens if, for some reason, we have the Karaf core jolokia feature installed before our own odl-jolokia feature. Indeed in a netvirt/karaf from today's master where this is not happening, you would see, which is correct:
karaf@root()> feature:list | grep jolokia odl-jolokia │ 1.11.0.SNAPSHOT │ │ Started │ odl-extras-1.11.0-SNAPSHOT │ Jolokia JMX/HTTP bridge jolokia │ 1.3.5 │ │ Uninstalled │ standard-4.1.5 │ Jolokia monitoring support
But indeed (only) if we feature:install jolokia odl-netvirt-openstack, THEN we can reproduce this issue!
> It should be harmless, but is not nice and an indication of a lifecycle error,
just to "qualify" my earlier "harmless" comment with a bit more background now: While this will not impact core SDN functionality, it does mean that the "wrong" Jolokia is installed, and e.g. /jolokia will not be protected by AAA. If one never uses /jolokia for JMX-over-HTTP monitoring, and is not overly security paranoid, then that is not a huge issue.
So the real question now is just to find out why on earth in some distributions it's the jolokia instead of the odl-jolokia feature that gets installed?! Is this wrongly hard-coded in some ... place, upstream or downstream installation orchestration, somewhere? It should not be so anywhere, because e.g. odl-netvirt-openstack (indirectly, same applies for other end-user application features) seems to correctly depend on odl-jolokia already.
Two next stepss: 1. I'll have a closer look at integration/distribution/karaf if that is wrong, 2. I'll email some people asking if there is a jolokia instead of the odl-jolokia hard-coded somewhere.
> closer look at integration/distribution/karaf
It's the same as with netvirt/karaf here - it works out of the box.
So either this is different in releases based off older code than today's Fluorine master (but I doubt that), or something else is wrong to explicitly install the jolokia Karaf feature... we now just need to find out what! I am suspecting some CSIT script? But we're also hearing about this downstream now, not just in upstream CSIT, so... perhaps there is also a hard-coded wrong explicit jolokia Karaf feature install in... TripleO?
PS, just for the record: I've also, briefly, considered some hack to "remove" the jolokia Karaf feature all together (based on this, or documentation which may be written in KARAF-5540 in the future), but concluded that actually won't really solve our immediate issue here (it would just cause something to fail, not fix anything), so I would rather focus efforts on finding the root cause instead.
> either this is different in releases based off older code than today's Fluorine master (but I doubt that)
Verified this by testing above against the Oxygen release from March 22, and noticed an interesting disprecancy:
opendaylight-user@root>feature:list | grep jolokia odl-jolokia │ 1.10.0 │ │ Uninstalled │ odl-extras-1.10.0 │ Jolokia JMX/HTTP bridge jolokia │ 1.3.5 │ │ Uninstalled │ standard-4.1.3 │ Jolokia monitoring support opendaylight-user@root>feature:install odl-netvirt-openstack opendaylight-user@root>feature:list | grep jolokia odl-jolokia │ 1.10.0 │ │ Uninstalled │ odl-extras-1.10.0 │ Jolokia JMX/HTTP bridge jolokia │ 1.3.5 │ │ Uninstalled │ standard-4.1.3 │ Jolokia monitoring support
so it would seem that in the past Jolokia (neither the wrong nor the right one) got automatically installed as a dependency... which reinforces my suspicion that we probably hard-coded this in the past - but we (always, most likely) had chosen the wrong one. The right thing to do now is to just remove any mention of jolokia and let it get installed automatically as a dependency.
Same in a local netvirt/karaf build of today's stable/oxygen branch, which is about to be released as Oxygen SP1:
karaf@root()> feature:list | grep jolokia odl-jolokia │ 1.10.1.SNAPSHOT │ │ Uninstalled │ odl-extras-1.10.1-SNAPSHOT │ Jolokia JMX/HTTP bridge jolokia │ 1.3.5 │ │ Uninstalled │ standard-4.1.5 │ Jolokia monitoring support karaf@root()> feature:install odl-netvirt-openstack karaf@root()> feature:list | grep jolokia odl-jolokia │ 1.10.1.SNAPSHOT │ │ Uninstalled │ odl-extras-1.10.1-SNAPSHOT │ Jolokia JMX/HTTP bridge jolokia │ 1.3.5 │ │ Uninstalled │ standard-4.1.5 │ Jolokia monitoring support karaf@root()> log:display | grep /jolokia
So while for master (future Fluorine) we seem to have a dependy to odl-jolokia somewhere, for Oxygen you can just explicitly install the odl-jolokia feature (but NOT the one similarly named just jolokia!), e.g. like so, and it's all good:
karaf@root()> feature:install odl-jolokia odl-netvirt-openstack karaf@root()> feature:list | grep jolokia odl-jolokia │ 1.10.1.SNAPSHOT │ x │ Started │ odl-extras-1.10.1-SNAPSHOT │ Jolokia JMX/HTTP bridge jolokia │ 1.3.5 │ │ Uninstalled │ standard-4.1.5 │ Jolokia monitoring support karaf@root()> log:display | grep /jolokia karaf@root()>
I'm lost because I'm probably just stupid. can you TL;DR it for me?
You're not stupid, I was beeing detailed and lengthy above 
so here's an attempt at a TL;DR:
1. The problem shown above is not reproducible after feature:install odl-netvirt-openstack in integration/distribution/karaf and netvirt/karaf on current master and stable/oxygen.
2. The problem IS easily reproduced if one installs the Karaf feature named "jolokia" (not "odl-jolokia") before installing odl-netvirt-openstack on master (where odl-netvirt-openstack also installs odl-jolokia), or both "jolokia" and "odl-jolokia" on Oxygen.
3. Therefore, my conclusion is that "something" in CSIT 
is manually installing the Karaf "jolokia" (not "odl-jolokia") feature? Perhaps you could add a "feature:list | grep jolokia" at the start of CSIT so that we could have more certainty about this, if there is doubt?
Does this help? For more details, read above... 
fair?
PS, FYI: On master, odl-netvirt-openstack also installs odl-jolokia; on stable/oxygen it does not. On neither I can see the Karaf "jolokia" feature (not "odl-jolokia") automatically installed by odl-netvirt-openstack.
> Perhaps you could add a "feature:list | grep jolokia"
even better, based on something I just learnt from skitt, please do this for me so that we can get to the bottom of this:
Please attach what this outputs for in an environment where this problem is seen:
karaf@root()> feature:list -r
and then please attach output of "feature:info -t $FN" for N times for each line (feature shown), where FN is the name of 1 feature, say:
feature:info -t odl-mdsal-trace feature:info -t odl-netvirt-openstack ...
Wait a second... the theory (above) re. the Karaf features named "jolokia" and "odl-jolokia" may be the completely wrong explanation... Or rather, that would also cause this same problem, but is perhaps not actually the cause in this instance!
Why? Because what I've missed before is that in this log originally linked above, there is this beauty worth noting:
2018-02-28T13:43:16,314 | INFO | features-1-thread-1 | FeaturesServiceImpl | 7 - org.apache.karaf.features.core - 4.1.3 | mvn:org.jolokia/jolokia-osgi/1.3.7
2018-02-28T13:43:16,314 | INFO | features-1-thread-1 | FeaturesServiceImpl | 7 - org.apache.karaf.features.core - 4.1.3 | mvn:org.jolokia/jolokia-osgi/1.5.0
(...)
2018-02-28T13:43:16,339 | INFO | features-1-thread-1 | FeaturesServiceImpl | 7 - org.apache.karaf.features.core - 4.1.3 | wrap:mvn:org.jolokia/jolokia-core/1.3.7
2018-02-28T13:43:16,339 | INFO | features-1-thread-1 | FeaturesServiceImpl | 7 - org.apache.karaf.features.core - 4.1.3 | wrap:mvn:org.jolokia/jolokia-core/1.5.0
2018-02-28T13:43:16,339 | INFO | features-1-thread-1 | FeaturesServiceImpl | 7 - org.apache.karaf.features.core - 4.1.3 | wrap:mvn:org.jolokia/jolokia-jsr160/1.3.7
2018-02-28T13:43:16,339 | INFO | features-1-thread-1 | FeaturesServiceImpl | 7 - org.apache.karaf.features.core - 4.1.3 | wrap:mvn:org.jolokia/jolokia-jsr160/1.5.0
(...)
2018-02-28T13:43:37,363 | INFO | features-1-thread-1 | FeaturesServiceImpl | 7 - org.apache.karaf.features.core - 4.1.3 | wrap_file__tmp_karaf-0.8.0-SNAPSHOT_system_org_jolokia_jolokia-core_1.3.7_jolokia-core-1.3.7.jar/0.0.0
2018-02-28T13:43:37,364 | INFO | features-1-thread-1 | FeaturesServiceImpl | 7 - org.apache.karaf.features.core - 4.1.3 | com.google.gson/2.8.2
2018-02-28T13:43:37,365 | INFO | features-1-thread-1 | FeaturesServiceImpl | 7 - org.apache.karaf.features.core - 4.1.3 | org.apache.oltu.oauth2.authzserver/1.0.2
2018-02-28T13:43:37,366 | INFO | features-1-thread-1 | FeaturesServiceImpl | 7 - org.apache.karaf.features.core - 4.1.3 | org.h2/1.4.193
2018-02-28T13:43:37,379 | INFO | features-1-thread-1 | FeaturesServiceImpl | 7 - org.apache.karaf.features.core - 4.1.3 | wrap_file__tmp_karaf-0.8.0-SNAPSHOT_system_org_jolokia_jolokia-core_1.5.0_jolokia-core-1.5.0.jar/0.0.0
(...)
2018-02-28T13:43:37,391 | INFO | features-1-thread-1 | FeaturesServiceImpl | 7 - org.apache.karaf.features.core - 4.1.3 | org.jolokia.osgi/1.5.0
2018-02-28T13:43:37,398 | INFO | features-1-thread-1 | HttpServiceFactoryImpl | 450 - org.ops4j.pax.web.pax-web-runtime - 6.0.7 | Binding bundle: [org.jolokia.osgi_1.5.0 [186]] to http service
(...)
2018-02-28T13:43:37,609 | INFO | features-1-thread-1 | HttpServiceContext | 448 - org.ops4j.pax.web.pax-web-jetty - 6.0.7 | registering context DefaultHttpContext [bundle=org.jolokia.osgi_1.5.0 [186], contextID=custom], with context-name:
2018-02-28T13:43:37,811 | INFO | features-1-thread-1 | osgi | 186 - org.jolokia.osgi - 1.5.0 | No access restrictor found, access to any MBean is allowed
2018-02-28T13:43:37,856 | INFO | features-1-thread-1 | ContextHandler | 164 - org.eclipse.jetty.util - 9.3.14.v20161028 | Started HttpServiceContext{httpContext=DefaultHttpContext [bundle=org.jolokia.osgi_1.5.0 [186], contextID=custom]}
(...)
2018-02-28T13:43:37,922 | INFO | features-1-thread-1 | FeaturesServiceImpl | 7 - org.apache.karaf.features.core - 4.1.3 | org.jolokia.osgi/1.3.7
2018-02-28T13:43:37,927 | INFO | features-1-thread-1 | HttpServiceFactoryImpl | 450 - org.ops4j.pax.web.pax-web-runtime - 6.0.7 | Binding bundle: [org.jolokia.osgi_1.3.7 [185]] to http service
2018-02-28T13:43:37,929 | ERROR | features-1-thread-1 | osgi | 186 - org.jolokia.osgi - 1.5.0 | Namespace Exception: org.osgi.service.http.NamespaceException: alias: '/jolokia' is already in use in this or another context
org.osgi.service.http.NamespaceException: alias: '/jolokia' is already in use in this or another context
Looking carefully above, there is something curious - what's this 1.3.7 VS 1.5.0 business? We shouldn't have 2 versions of Jolokia bundles not features!
To be 100% sure that it's related to that, could I ask for, in addition to above, also getting the output of:
bundle:list -s -l
I was looking to update a CSIT job to collect this info, but realized we aren't seeing this Exception in a recent CSIT
job. Like this one
you want this info on a recent distro, right? in other words, on a distro that is not showing this problem?
> I was looking to update a CSIT job to collect this info, but realized we aren't seeing this Exception in a recent CSIT job. Like this one
Very interesting. And in that new log - guess what? There is only jolokia-core|jsr160|osgi 1.5.0, only 1.3.7 - so that's it! So looks like something during "the big bump" which happened some time after April 10th (jluhrsen IRC: "I think Apr 10th is the last time this showed up in that job."), most likely the Karaf upgrade that was part of that bump, did the trick here.
For the record, I was seeing this 100% of the time with Oxygen SR0, not anymore with Oxygen SR1 (well, the daily build from 22 Apr to be accurate).
Feature list for working (22 Apr) - text to long to paste here (pasted one more, previous version was trimmed too):
No forget about that, that from (just minutes) before I figured it out; this is no longer needed.
> For the record, I was seeing this 100% of the time with Oxygen SR0, not anymore with Oxygen SR1 (well, the daily build from 22 Apr to be accurate).
Great, that confirms above - because "the big bump" with a Karaf upgrade also happened on Oxygen SR1.
I'm therefore now closing this issue as CANNOT REPRODUCE, in the sense that it is no longer seen on the latest Fluorine, and confirmed fixed on Oxygen SR1.
FTR: https://bugzilla.redhat.com/show_bug.cgi?id=1569665 continues the discussion for a certain red vendor's downstream Oxygen (not yet SR1) based package, but that is no longer relevant to bleeding edge master nor stable/oxygen maintenance in this upstream JIRA.
Yes, this is one of the bugs we fixed with the odlparent 3.1 bump, along with AAA-147.