Operating System: All
Platform: All

Attachment karaf.log has been added with description: logs

This isn't an issue with clustering. It appears to be an issue with bundles refreshing due to the order of the features installed which has been a problem before, eg

2017-06-28 16:40:36,181 | INFO | pool-2-thread-1 | FeaturesServiceImpl | 9 - org.apache.karaf.features.core - 4.0.9 | Refreshing bundles:
2017-06-28 16:40:36,181 | INFO | pool-2-thread-1 | FeaturesServiceImpl | 9 - org.apache.karaf.features.core - 4.0.9 | config-persister-feature-adapter/0.7.0.SNAPSHOT (Wired to org.opendaylight.controller.config-persister-impl/0.7.0.SNAPSHOT which is being refreshed)

The error noted above is caused by:

Caused by: java.lang.ClassCastException: org.opendaylight.controller.config.yang.config.distributed_datastore_provider.OperationalProperties cannot be cast to org.opendaylight.controller.config.yang.config.distributed_datastore_provider.OperationalProperties

Since the class names to and from the cast are the same that means they were loaded by different class loaders which means the bundle was restarted.

2017-08-24 14:56:51,884 | INFO | pool-2-thread-1 | FeaturesServiceImpl | 6 - org.apache.karaf.features.core - 4.0.9 | Refreshing bundles:
2017-08-24 14:56:51,884 | INFO | pool-2-thread-1 | FeaturesServiceImpl | 6 - org.apache.karaf.features.core - 4.0.9 | com.typesafe.akka.actor/2.4.18 (Wired to org.opendaylight.controller.sal-distributed-datastore/1.6.0 which is being refreshed)
2017-08-24 14:56:51,885 | INFO | pool-2-thread-1 | FeaturesServiceImpl | 6 - org.apache.karaf.features.core - 4.0.9 | com.typesafe.akka.cluster/2.4.18 (Wired to com.typesafe.akka.actor/2.4.18 which is being refreshed)
2017-08-24 14:56:51,885 | INFO | pool-2-thread-1 | FeaturesServiceImpl | 6 - org.apache.karaf.features.core - 4.0.9 | com.typesafe.akka.osgi/2.4.18 (Wired to com.typesafe.akka.actor/2.4.18 which is being refreshed)
2017-08-24 14:56:51,885 | INFO | pool-2-thread-1 | FeaturesServiceImpl | 6 - org.apache.karaf.features.core - 4.0.9 | com.typesafe.akka.persistence/2.4.18 (Wired to com.typesafe.akka.actor/2.4.18 which is being refreshed)
2017-08-24 14:56:51,885 | INFO | pool-2-thread-1 | FeaturesServiceImpl | 6 - org.apache.karaf.features.core - 4.0.9 | com.typesafe.akka.remote/2.4.18 (Wired to org.jboss.netty/3.10.6.Final which is being refreshed)
2017-08-24 14:56:51,885 | INFO | pool-2-thread-1 | FeaturesServiceImpl | 6 - org.apache.karaf.features.core - 4.0.9 | com.typesafe.akka.slf4j/2.4.18 (Wired to com.typesafe.akka.actor/2.4.18 which is being refreshed)
2017-08-24 14:56:51,885 | INFO | pool-2-thread-1 | FeaturesServiceImpl | 6 - org.apache.karaf.features.core - 4.0.9 | com.typesafe.akka.stream/2.4.18 (Wired to com.typesafe.akka.actor/2.4.18 which is being refreshed)
2017-08-24 14:56:51,885 | INFO | pool-2-thread-1 | FeaturesServiceImpl | 6 - org.apache.karaf.features.core - 4.0.9 | config-persister-feature-adapter/0.7.0 (Wired to org.opendaylight.controller.config-persister-impl/0.7.0 which is being refreshed)
2017-08-24 14:56:51,885 | INFO | pool-2-thread-1 | FeaturesServiceImpl | 6 - org.apache.karaf.features.core - 4.0.9 | org.apache.karaf.config.core/4.0.9 (Should be wired to: org.apache.felix.metatype/1.1.2 (through [org.apache.karaf.config.core/4.0.9] osgi.wiring.package; filter:="(&(osgi.wiring.package=org.osgi.service.metatype)(version>=1.2.0)(!(version>=2.0.0)))"; resolution:=optional))
2017-08-24 14:56:51,885 | INFO | pool-2-thread-1 | FeaturesServiceImpl | 6 - org.apache.karaf.features.core - 4.0.9 | org.apache.servicemix.bundles.jasypt/1.9.2.1 (Should be wired to: javax.servlet-api/3.1.0 (through [org.apache.servicemix.bundles.jasypt/1.9.2.1] osgi.wiring.package; filter:="(osgi.wiring.package=javax.servlet)"; resolution:=optional))
2017-08-24 14:56:51,885 | INFO | pool-2-thread-1 | FeaturesServiceImpl | 6 - org.apache.karaf.features.core - 4.0.9 | org.eclipse.persistence.core/2.6.2.v20151217-774c696 (Should be wired to: javax.mail/1.4.4 (through [org.eclipse.persistence.core/2.6.2.v20151217-774c696] osgi.wiring.package; filter:="(osgi.wiring.package=javax.mail)"; resolution:=optional))
2017-08-24 14:56:51,885 | INFO | pool-2-thread-1 | FeaturesServiceImpl | 6 - org.apache.karaf.features.core - 4.0.9 | org.eclipse.persistence.moxy/2.6.2.v20151217-774c696 (Wired to org.eclipse.persistence.core/2.6.2.v20151217-774c696 which is being refreshed)
2017-08-24 14:56:51,885 | INFO | pool-2-thread-1 | FeaturesServiceImpl | 6 - org.apache.karaf.features.core - 4.0.9 | org.jboss.netty/3.10.6.Final (Should be wired to: javax.servlet-api/3.1.0 (through [org.jboss.netty/3.10.6.Final] osgi.wiring.package; filter:="(osgi.wiring.package=javax.servlet)"; resolution:=optional))
2017-08-24 14:56:51,885 | INFO | pool-2-thread-1 | FeaturesServiceImpl | 6 - org.apache.karaf.features.core - 4.0.9 | org.opendaylight.controller.cds-access-api/1.2.0 (Wired to com.typesafe.akka.actor/2.4.18 which is being refreshed)
2017-08-24 14:56:51,885 | INFO | pool-2-thread-1 | FeaturesServiceImpl | 6 - org.apache.karaf.features.core - 4.0.9 | org.opendaylight.controller.cds-access-client/1.2.0 (Wired to com.typesafe.akka.actor/2.4.18 which is being refreshed)
2017-08-24 14:56:51,885 | INFO | pool-2-thread-1 | FeaturesServiceImpl | 6 - org.apache.karaf.features.core - 4.0.9 | org.opendaylight.controller.config-persister-directory-xml-adapter/0.7.0 (Wired to org.opendaylight.controller.config-persister-impl/0.7.0 which is being refreshed)
2017-08-24 14:56:51,885 | INFO | pool-2-thread-1 | FeaturesServiceImpl | 6 - org.apache.karaf.features.core - 4.0.9 | org.opendaylight.controller.config-persister-file-xml-adapter/0.7.0 (Wired to org.opendaylight.controller.config-persister-impl/0.7.0 which is being refreshed)
2017-08-24 14:56:51,885 | INFO | pool-2-thread-1 | FeaturesServiceImpl | 6 - org.apache.karaf.features.core - 4.0.9 | org.opendaylight.controller.config-persister-impl/0.7.0 (Wired to org.eclipse.persistence.moxy/2.6.2.v20151217-774c696 which is being refreshed)
2017-08-24 14:56:51,885 | INFO | pool-2-thread-1 | FeaturesServiceImpl | 6 - org.apache.karaf.features.core - 4.0.9 | org.opendaylight.controller.sal-akka-raft/1.6.0 (Wired to com.typesafe.akka.actor/2.4.18 which is being refreshed)
2017-08-24 14:56:51,885 | INFO | pool-2-thread-1 | FeaturesServiceImpl | 6 - org.apache.karaf.features.core - 4.0.9 | org.opendaylight.controller.sal-cluster-admin-impl/1.6.0 (Wired to com.typesafe.akka.actor/2.4.18 which is being refreshed)
2017-08-24 14:56:51,885 | INFO | pool-2-thread-1 | FeaturesServiceImpl | 6 - org.apache.karaf.features.core - 4.0.9 | org.opendaylight.controller.sal-clustering-commons/1.6.0 (Wired to com.typesafe.akka.remote/2.4.18 which is being refreshed)
2017-08-24 14:56:51,885 | INFO | pool-2-thread-1 | FeaturesServiceImpl | 6 - org.apache.karaf.features.core - 4.0.9 | org.opendaylight.controller.sal-distributed-datastore/1.6.0 (Wired to org.opendaylight.controller.sal-clustering-commons/1.6.0 which is being refreshed)
2017-08-24 14:56:51,886 | INFO | pool-2-thread-1 | FeaturesServiceImpl | 6 - org.apache.karaf.features.core - 4.0.9 | org.opendaylight.controller.sal-remoterpc-connector/1.6.0 (Wired to org.opendaylight.controller.sal-clustering-commons/1.6.0 which is being refreshed)
2017-08-24 14:56:51,886 | INFO | pool-2-thread-1 | FeaturesServiceImpl | 6 - org.apache.karaf.features.core - 4.0.9 | org.ops4j.pax.jdbc.config/1.0.1 (Wired to org.apache.servicemix.bundles.jasypt/1.9.2.1 which is being refreshed)

Which boils down to akka being refreshed because of servlet-3.1.0 being optionally wired into netty-3.10.6

mcauffiez@inocybe.com is this still happening?

I try this order:

feature:install odl-bgpcep-bgp
feature:install odl-bgpcep-pcep
feature:install odl-restconf-all

and his working

And if I do this there is an issue

feature:install odl-bgpcep-pcep
feature:install odl-bgpcep-bgp
feature:install odl-restconf-all

it is failling when I do the feature:install odl-bgpcep-bgp

with this error

Exception in thread "Thread-18" java.lang.IllegalStateException: Schema for node with name topology-pcep and namespace urn:opendaylight:params:xml:ns:yang:topology:pcep does not exist at AbsoluteSchemaPath

at com.google.common.base.Preconditions.checkState(Preconditions.java:847)
at org.opendaylight.yangtools.yang.data.codec.xml.XmlParserStream.read(XmlParserStream.java:361)
at org.opendaylight.yangtools.yang.data.codec.xml.XmlParserStream.read(XmlParserStream.java:367)
at org.opendaylight.yangtools.yang.data.codec.xml.XmlParserStream.read(XmlParserStream.java:291)
at org.opendaylight.yangtools.yang.data.codec.xml.XmlParserStream.read(XmlParserStream.java:367)
at org.opendaylight.yangtools.yang.data.codec.xml.XmlParserStream.parse(XmlParserStream.java:184)
at org.opendaylight.bgpcep.config.loader.impl.ConfigLoaderImpl.parseDefaultConfigFile(ConfigLoaderImpl.java:128)
at org.opendaylight.bgpcep.config.loader.impl.ConfigLoaderImpl.handleConfigFile(ConfigLoaderImpl.java:87)
at org.opendaylight.bgpcep.config.loader.impl.ConfigLoaderImpl.access$300(ConfigLoaderImpl.java:53)
at org.opendaylight.bgpcep.config.loader.impl.ConfigLoaderImpl$ConfigLoaderImplRunnable.lambda$handleEvent$2(ConfigLoaderImpl.java:227)
at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
at java.util.HashMap$EntrySpliterator.forEachRemaining(HashMap.java:1696)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
at org.opendaylight.bgpcep.config.loader.impl.ConfigLoaderImpl$ConfigLoaderImplRunnable.handleEvent(ConfigLoaderImpl.java:227)
at java.util.ArrayList.forEach(ArrayList.java:1257)
at org.opendaylight.bgpcep.config.loader.impl.ConfigLoaderImpl$ConfigLoaderImplRunnable.handleChanges(ConfigLoaderImpl.java:215)
at org.opendaylight.bgpcep.config.loader.impl.ConfigLoaderImpl$ConfigLoaderImplRunnable.run(ConfigLoaderImpl.java:195)
at java.lang.Thread.run(Thread.java:748)

The ordering you listed above for both is the same. 

Indeed, the order that is not working is

feature:install odl-bgpcep-pcep
feature:install odl-bgpcep-bgp
feature:install odl-restconf-all

I have updated the previous post

cdgasparini can you check what is going on here? it looks like reloads, but there is also config-loader involved and that did have safety issues...

I opened BGPCEP-852 for track mcauffiez@inocybe.com issue. But I'll say that he is hitting a different issue that the one for was this bug was opened at that time.

Regards,

The ISE is here: https://git.opendaylight.org/gerrit/#/q/Iecee17e19b3ac380bcb2b7fc1c46f27e789c20bd and it will be correctly reported as XMLStreamException – the second part will need to be addressed in BGPCEP-852.