Operating System: All
Platform: All

Besides Rx/Tx transaction, I noticed similar LinkageError when receiving data change notification for a container that uses a field with the inet:ip-address type (which is also a union).

2017-04-04 18:23:29,362 | ERROR | on-dispatcher-57 | DataChangeListener | 193 - org.opendaylight.controller.sal-distributed-datastore - 1.4.2.Boron-SR2 | Error notifying listener org.acme.provisioning.impl.listeners.ProvisioningManager
java.lang.IllegalStateException: Unable to load codec for class org.opendaylight.yang.gen.v1.urn.ietf.params.xml.ns.yang.ietf.inet.types.rev130715.IpAddress
at org.opendaylight.yangtools.binding.data.codec.impl.BindingCodecContext.getCodecForBindingClass(BindingCodecContext.java:306)[91:org.opendaylight.mdsal.binding-dom-codec:0.9.2.Boron-SR2]
at org.opendaylight.yangtools.binding.data.codec.impl.BindingCodecContext.getCodec(BindingCodecContext.java:291)[91:org.opendaylight.mdsal.binding-dom-codec:0.9.2.Boron-SR2]
at org.opendaylight.yangtools.binding.data.codec.impl.BindingCodecContext.getCodec(BindingCodecContext.java:274)[91:org.opendaylight.mdsal.binding-dom-codec:0.9.2.Boron-SR2]
at org.opendaylight.yangtools.binding.data.codec.impl.BindingCodecContext.getLeafNodesUsingReflection(BindingCodecContext.java:262)[91:org.opendaylight.mdsal.binding-dom-codec:0.9.2.Boron-SR2]
at org.opendaylight.yangtools.binding.data.codec.impl.BindingCodecContext.getLeafNodes(BindingCodecContext.java:229)[91:org.opendaylight.mdsal.binding-dom-codec:0.9.2.Boron-SR2]
at org.opendaylight.yangtools.binding.data.codec.impl.DataObjectCodecContext.<init>(DataObjectCodecContext.java:84)[91:org.opendaylight.mdsal.binding-dom-codec:0.9.2.Boron-SR2]
at org.opendaylight.yangtools.binding.data.codec.impl.ListNodeCodecContext.<init>(ListNodeCodecContext.java:22)[91:org.opendaylight.mdsal.binding-dom-codec:0.9.2.Boron-SR2]
at org.opendaylight.yangtools.binding.data.codec.impl.KeyedListNodeCodecContext.<init>(KeyedListNodeCodecContext.java:29)[91:org.opendaylight.mdsal.binding-dom-codec:0.9.2.Boron-SR2]
at org.opendaylight.yangtools.binding.data.codec.impl.DataContainerCodecPrototype.createInstance(DataContainerCodecPrototype.java:129)[91:org.opendaylight.mdsal.binding-dom-codec:0.9.2.Boron-SR2]
at org.opendaylight.yangtools.binding.data.codec.impl.DataContainerCodecPrototype.get(DataContainerCodecPrototype.java:112)[91:org.opendaylight.mdsal.binding-dom-codec:0.9.2.Boron-SR2]
at org.opendaylight.yangtools.binding.data.codec.impl.DataContainerCodecPrototype.get(DataContainerCodecPrototype.java:30)[91:org.opendaylight.mdsal.binding-dom-codec:0.9.2.Boron-SR2]
at org.opendaylight.yangtools.binding.data.codec.impl.DataObjectCodecContext.yangPathArgumentChild(DataObjectCodecContext.java:209)[91:org.opendaylight.mdsal.binding-dom-codec:0.9.2.Boron-SR2]
at org.opendaylight.yangtools.binding.data.codec.impl.BindingCodecContext.getCodecContextNode(BindingCodecContext.java:153)[91:org.opendaylight.mdsal.binding-dom-codec:0.9.2.Boron-SR2]
at org.opendaylight.yangtools.binding.data.codec.impl.BindingNormalizedNodeCodecRegistry.fromNormalizedNode(BindingNormalizedNodeCodecRegistry.java:182)[91:org.opendaylight.mdsal.binding-dom-codec:0.9.2.Boron-SR2]
at org.opendaylight.controller.md.sal.binding.impl.BindingToNormalizedNodeCodec.toBinding(BindingToNormalizedNodeCodec.java:221)[163:org.opendaylight.controller.sal-binding-broker-impl:1.4.2.Boron-SR2]
at org.opendaylight.controller.md.sal.binding.impl.AbstractForwardedDataBroker.toBinding(AbstractForwardedDataBroker.java:89)[163:org.opendaylight.controller.sal-binding-broker-impl:1.4.2.Boron-SR2]
at org.opendaylight.controller.md.sal.binding.impl.AbstractForwardedDataBroker$TranslatedDataChangeEvent.getCreatedData(AbstractForwardedDataBroker.java:187)[163:org.opendaylight.controller.sal-binding-broker-impl:1.4.2.Boron-SR2]
at org.acme.provisioning.impl.listeners.TunnelProvisioningManager.onDataChanged(TunnelProvisioningManager.java:104)[322:org.acme.network.provisioning-mgmt-svc-impl:0.1.0.SNAPSHOT]
at org.opendaylight.controller.md.sal.binding.impl.AbstractForwardedDataBroker$TranslatingDataChangeInvoker.onDataChanged(AbstractForwardedDataBroker.java:143)[163:org.opendaylight.controller.sal-binding-broker-impl:1.4.2.Boron-SR2]
at org.opendaylight.controller.cluster.datastore.DataChangeListener.dataChanged(DataChangeListener.java:73)[193:org.opendaylight.controller.sal-distributed-datastore:1.4.2.Boron-SR2]
at org.opendaylight.controller.cluster.datastore.DataChangeListener.handleReceive(DataChangeListener.java:45)[193:org.opendaylight.controller.sal-distributed-datastore:1.4.2.Boron-SR2]
at org.opendaylight.controller.cluster.common.actor.AbstractUntypedActor.onReceive(AbstractUntypedActor.java:26)[187:org.opendaylight.controller.sal-clustering-commons:1.4.2.Boron-SR2]
at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:165)[175:com.typesafe.akka.actor:2.4.7]
at akka.actor.Actor$class.aroundReceive(Actor.scala:484)[175:com.typesafe.akka.actor:2.4.7]
at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:95)[175:com.typesafe.akka.actor:2.4.7]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)[175:com.typesafe.akka.actor:2.4.7]
at akka.actor.ActorCell.invoke(ActorCell.scala:495)[175:com.typesafe.akka.actor:2.4.7]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)[175:com.typesafe.akka.actor:2.4.7]
at akka.dispatch.Mailbox.run(Mailbox.scala:224)[175:com.typesafe.akka.actor:2.4.7]
at akka.dispatch.Mailbox.exec(Mailbox.scala:234)[175:com.typesafe.akka.actor:2.4.7]
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)[171:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)[171:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)[171:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)[171:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8]
Caused by: java.lang.IllegalStateException: Failed to access constructor for class org.opendaylight.yang.gen.v1.urn.ietf.params.xml.ns.yang.ietf.inet.types.rev130715.Ipv4Address in type class org.opendaylight.yang.gen.v1.urn.ietf.params.xml.ns.yang.ietf.inet.types.rev130715.IpAddress
at org.opendaylight.yangtools.binding.data.codec.impl.UnionValueOptionContext.<init>(UnionValueOptionContext.java:43)[91:org.opendaylight.mdsal.binding-dom-codec:0.9.2.Boron-SR2]
at org.opendaylight.yangtools.binding.data.codec.impl.UnionTypeCodec.lambda$loader$0(UnionTypeCodec.java:36)[91:org.opendaylight.mdsal.binding-dom-codec:0.9.2.Boron-SR2]
at org.opendaylight.yangtools.binding.data.codec.impl.BindingCodecContext.getCodecForBindingClass(BindingCodecContext.java:304)[91:org.opendaylight.mdsal.binding-dom-codec:0.9.2.Boron-SR2]
... 33 more
Caused by: java.lang.IllegalAccessException: no such constructor: org.opendaylight.yang.gen.v1.urn.ietf.params.xml.ns.yang.ietf.inet.types.rev130715.IpAddress.<init>(Ipv4Address)void/newInvokeSpecial
at java.lang.invoke.MemberName.makeAccessException(MemberName.java:869)[:1.8.0_77]
at java.lang.invoke.MemberName$Factory.resolveOrFail(MemberName.java:1005)[:1.8.0_77]
at java.lang.invoke.MethodHandles$Lookup.resolveOrFail(MethodHandles.java:1382)[:1.8.0_77]
at java.lang.invoke.MethodHandles$Lookup.findConstructor(MethodHandles.java:920)[:1.8.0_77]
at org.opendaylight.yangtools.binding.data.codec.impl.UnionValueOptionContext.<init>(UnionValueOptionContext.java:40)[91:org.opendaylight.mdsal.binding-dom-codec:0.9.2.Boron-SR2]
... 35 more
Caused by: java.lang.LinkageError: loader constraint violation: when resolving method "org.opendaylight.yang.gen.v1.urn.ietf.params.xml.ns.yang.ietf.inet.types.rev130715.IpAddress.<init>(Lorg/opendaylight/yang/gen/v1/urn/ietf/params/xml/ns/yang/ietf/inet/types/rev130715/Ipv4Address;)V" the class loader (instance of <bootloader>) of the current class, java/lang/Object, and the class loader (instance of org/eclipse/osgi/internal/baseadaptor/DefaultClassLoader) for the method's defining class, org/opendaylight/yang/gen/v1/urn/ietf/params/xml/ns/yang/ietf/inet/types/rev130715/IpAddress, have different Class objects for the type org/opendaylight/yang/gen/v1/urn/ietf/params/xml/ns/yang/ietf/inet/types/rev130715/Ipv4Address used in the signature
at java.lang.invoke.MethodHandleNatives.resolve(Native Method)[:1.8.0_77]
at java.lang.invoke.MemberName$Factory.resolve(MemberName.java:977)[:1.8.0_77]
at java.lang.invoke.MemberName$Factory.resolveOrFail(MemberName.java:1002)[:1.8.0_77]
... 38 more

This is not an issue with clustering. Looks like a yangtools issue so moving there.

(In reply to Tom Pantelis from comment #2)
> This is not an issue with clustering. Looks like a yangtools issue so moving
> there.

I filed the bug for clustering because this happens only in clustered mode and when one of the cluster instance is restarted.

The exception is thrown from MDSAL and it seems like a binding generator or binding codecs issue, so moved to MDSAL.

at first sight, it doesn't seem to be issue with binding-generator and I can't confirm binding codec issue so far. It contains message

Caused by: java.lang.LinkageError: loader constraint violation: when resolving method

as we can see here too: https://lists.opendaylight.org/pipermail/netconf-dev/2017-April/001029.html

and by description of Srini, he sees it connected to clustering. For start, can we confirm root cause of clustered environment?

Like I mention in the email thread, I'm able to reproduce this by just doing the following two steps while having data populated in my journal (similar to what is triggered by controller/opendaylight/md-sal/sal-distributed-datastore/src/main/java/org/opendaylight/controller/config/yang/config/actor_system_provider/impl/ActorSystemProviderImpl.java).

opendaylight-user@root>property karaf.restart true
opendaylight-user@root>stop -f 0

Somehow the data/cache causes this LinkageError whenever a restart is caused through Quarantine. It doesn't happen for regular warm restarts. Not sure where the wiring in Karaf/OSGi is screwing up.

Note that I was unable to reproduce this with Beryllium-SR3.

Are you able reproduce it with 1 node on stable/master branch?

Yes, this was reproduce in a 1-node setup too on Boron-sr3. I will try with stable master branch and let you know.

OK, will wait for logs. Do you mind if I decrease importance to major? You're saying that it works most of the time. Thanks.

Yes, it is fine to move to Major. I realized that this bug doesn't actually impact the standalone mode much because users rarely do a karaf.restart. The programmatic restart happens more often in clustered mode.

OK, changed to major, will wait for logs from you. Thank you.

Are any of the installed bundles being restarted?

Any updates on this issue? I think it should have critical priority because this bug renders the cluster unusable after network partitioning. Once a node is auto-downed/quarantined, the QuarantinedMonitorActor in ActorSystemProviderImpl will try to restart karaf, which will end with the exception mentioned above. Note that this problem does not occur in Beryllium. Steps to reproduce:
1.feature:install odl-restconf odl-netconf-topology odl-netconf-connector-ssh
2.check that controller-config mount point works using restconf
3.system:property karaf.restart true ; bundle:stop -f 0
4.see that the java.lang.LinkageError is in logs, mount point does not work anymore

Restarting/reloading bundles is not something that is supported, hence this certainly does not warrant being called a critical bug.

What sal-distributed-datastore is doing is incorrect, as after failure the entire JVM state needs to be cleaned, e.g. karaf needs to be re-executed. Otherwise state present on the boot classpath is not cleaned.

Furthermore it sal-distributed-datastore implementation should not be tied to OSGi, as that ties implementation with runtime details.

I agree that restarting bundles is wrong, and it should be implemented as you mentioned. However I still believe this is critical, because it affects every deployment of clustered ODL.

https://git.opendaylight.org/gerrit/60485

rovarga Can we cherry pick this fix to Carbon and Nitrogen?