[CONTROLLER-1264] Config subsystem does not load services when features are installed in karaf instances Created: 20/Apr/15  Updated: 28/May/15  Resolved: 28/May/15

Status: Resolved
Project: controller
Component/s: config
Affects Version/s: Helium
Fix Version/s: None

Type: Bug
Reporter: Noam Mor Assignee: Maros Marsalek
Resolution: Cannot Reproduce Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: All
Platform: All

External issue ID: 3027

 Description   

First reported here: https://lists.opendaylight.org/pipermail/controller-dev/2015-April/008945.html

There is a difference in the behavior of the config subsystem in normal karaf and in a karaf instance (a separate process created by karaf). It does not seem to register services as it should.

Problem verified in Helium-SR1 and Helium-SR3.

Minimal steps to reproduce in Helium-SR1:

./bin/karaf
instance:create test

run:
echo 'org.ops4j.pax.url.mvn.localRepository=${karaf.home}/${karaf.default.repository}' >> ./instances/test/etc/org.ops4j.pax.url.mvn.cfg

instance:start test
instance:connect test
feature:repo-add mvn:org.opendaylight.integration/features-integration/0.2.1-Helium-SR1/xml/features

feature:install odl-config-netty

services | grep io.netty.util.Timer

Expected output:
[io.netty.util.Timer]

Actual output: None

The same feature can be installed in root context, and it would expose the service from netty-timer-config. In instance context, the service is not there, even though all the bundles are active.

Here are the bundle details on my machine:

– Instance context: –

opendaylight-user@test>bundle:capabilities 147
org.opendaylight.controller.netty-timer-config [147] provides:
--------------------------------------------------------------
// ... snipped ...
service; [org.opendaylight.controller.config.spi.ModuleFactory] with properties:
service.id = 634
Used by:
org.opendaylight.controller.config-manager [120]

– Root context: –

opendaylight-user@root>bundle:capabilities 147
org.opendaylight.controller.netty-timer-config_0.2.6.Helium-SR1 [147] provides:
-------------------------------------------------------------------------------
// ... snipped ...
service; [org.opendaylight.controller.config.spi.ModuleFactory] with properties:
service.id = 657
Used by:
org.opendaylight.controller.config-manager_0.2.6.Helium-SR1 [120]
service; [io.netty.util.Timer] with properties:
name = global-timer
service.id = 664

 Comments   
Comment by Maros Marsalek [ 30/Apr/15 ]

Tested this with stable/lithium and HeliumSR3. But I tried to install odl-restconf-all feature instead of odl-config-netty and everything worked fine. Restconf was available and the capabilities command for netty timer returned:

org.opendaylight.controller.netty-timer-config_0.2.8.Helium-SR3 [212] provides:
-------------------------------------------------------------------------------
...
service; [org.opendaylight.controller.config.spi.ModuleFactory] with properties:
service.id = 726
Used by:
org.opendaylight.controller.config-manager_0.2.8.Helium-SR3 [183]
service; [io.netty.util.Timer] with properties:
name = global-timer
service.id = 750

So it looks like it works, just the feature you installed might not be referencing the config files to instantiate the timer, just the config bindings.

Comment by Maros Marsalek [ 30/Apr/15 ]

Was able to reproduce on Helium-SR3 with following exceptions:

java.lang.NullPointerException: FeatureWrapper requires non-null Feature in constructor

Applied workaround for this issue from master branch:

https://git.opendaylight.org/gerrit/#/c/16695/

But the problem was still present with following exception:

Exception in thread "config-pusher" java.lang.IllegalStateException: Max wait for capabilities reached.Netconf service not stable for 00-netty.xml(odl-mdsal-all,odl-config-netty)
at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.getOperationServiceWithRetries(ConfigPusherImpl.java:159)
at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.pushConfigWithConflictingVersionRetries(ConfigPusherImpl.java:132)
at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.internalPushConfigs(ConfigPusherImpl.java:109)
at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.process(ConfigPusherImpl.java:76)
at org.opendaylight.controller.netconf.persist.impl.osgi.ConfigPersisterActivator$InnerCustomizer$1.run(ConfigPersisterActivator.java:178)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl$NotEnoughCapabilitiesException: Netconf service not stable for 00-netty.xml(odl-mdsal-all,odl-config-netty)
at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.getOperationService(ConfigPusherImpl.java:186)
at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.getOperationServiceWithRetries(ConfigPusherImpl.java:152)
... 5 more
Caused by: java.lang.SecurityException: Insufficient roles/credentials for operation
at org.apache.karaf.management.KarafMBeanServerGuard.handleInvoke(KarafMBeanServerGuard.java:228)
at org.apache.karaf.management.KarafMBeanServerGuard.handleGetAttribute(KarafMBeanServerGuard.java:192)
at org.apache.karaf.management.KarafMBeanServerGuard.invoke(KarafMBeanServerGuard.java:63)
at org.apache.karaf.management.boot.KarafMBeanServerBuilder$MBeanInvocationHandler.invoke(KarafMBeanServerBuilder.java:63)
at com.sun.proxy.$Proxy0.getAttribute(Unknown Source)
at com.sun.jmx.mbeanserver.MXBeanProxy$GetHandler.invoke(MXBeanProxy.java:122)
at com.sun.jmx.mbeanserver.MXBeanProxy.invoke(MXBeanProxy.java:167)
at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:252)
at com.sun.proxy.$Proxy20.getAvailableModuleFactoryQNames(Unknown Source)
at org.opendaylight.controller.config.util.ConfigRegistryJMXClient.getAvailableModuleFactoryQNames(ConfigRegistryJMXClient.java:220)
at org.opendaylight.controller.netconf.confignetconfconnector.osgi.NetconfOperationServiceImpl.checkConsistencyBetweenYangStoreAndConfig(NetconfOperationServiceImpl.java:58)
at org.opendaylight.controller.netconf.confignetconfconnector.osgi.NetconfOperationServiceImpl.<init>(NetconfOperationServiceImpl.java:45)
at org.opendaylight.controller.netconf.confignetconfconnector.osgi.NetconfOperationServiceFactoryImpl.createService(NetconfOperationServiceFactoryImpl.java:71)
at org.opendaylight.controller.netconf.confignetconfconnector.osgi.NetconfOperationServiceFactoryImpl.createService(NetconfOperationServiceFactoryImpl.java:19)
at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.getOperationService(ConfigPusherImpl.java:184)
... 6 more

Karaf prevented config subsystem to push configs due to security exception.

Comment by Maros Marsalek [ 07/May/15 ]

Maybe linked to this issue in karaf
https://issues.apache.org/jira/browse/KARAF-3147

Noticed that when starting child karaf instances, the jvm is started with following option:
-Djavax.management.builder.initial=org.apache.karaf.management.boot.KarafMBeanServerBuilder

which causes the jvm to provide KarafMBeanServerGuard as the default platform MBeanServer. This guard rejects method invocation over JMX even if the privileges are correctly set in jmx.acl.cfg file. Overriding/Removing this option might fix the issue (when starting karaf instance, user can provide jvm options using -o switch so this might be used to override mbean server builder option: "-Djavax.management.builder.initial= ").

Comment by Maros Marsalek [ 12/May/15 ]

This seem to work with lithium (karaf 3.0.3) but not with Helium (karaf 3.0.1). Workaround needs to be tested.

Comment by Noam Mor [ 13/May/15 ]

We checked out the Helium-SR3 code, modified its pom.xmls to depend on karaf 3.0.2, and built it. We did not observe the bug in this version. In my opinion, it's almost certainly a bug in karaf 3.0.1.

We haven't yet found a workaround for the 3.0.1 build.

Comment by Maros Marsalek [ 28/May/15 ]

Closing as works for me. This issue is not present with Lithium and karaf version 3.0.3. For Helium there is a workaround.

Generated at Wed Feb 07 19:55:05 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.