[MDSAL-122] Deadlock on loading Binding Serializer Created: 11/Dec/15  Updated: 09/Mar/18  Resolved: 08/Feb/16

Status: Resolved
Project: mdsal
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug
Reporter: Moiz Raja Assignee: Robert Varga
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: All
Platform: All

Issue Links:
Duplicate
is duplicated by AAA-48 MD-SAL Cluster Token Store get stuck ... Resolved
External issue ID: 4764

 Description   

We are observing an issue on controller restart where modification operations (put/merge) on transactions get deadlocked on looking for a serializer.

Here is the link to the threads that are locked up.

https://gist.github.com/moizr/079888bd1547559820f8

pool-55-thread-1 is BLOCKED on getting the Serializer.
pendaylight-cluster-data-akka.actor.default-dispatcher-3 is WAITING for the generation of a Serializer to complete.

It appears that the generating of the serializer is not completing and thus the deadlock.

 Comments   
Comment by Moiz Raja [ 11/Dec/15 ]

Note that this issue is observed on Li-SR2

Comment by Tony Tkacik [ 14/Dec/15 ]

Please provide full thread dump, thread dump you provided does not show any deadlock
(pool thread is waiting for different monitor than akka thread) - which may get unblocked.
In order to determine this is deadlock or livelock and how to fix it, we will need full thread dump and ideally steps to reproduce.

Comment by Tony Tkacik [ 14/Dec/15 ]

"opendaylight-cluster-data-akka.actor.default-dispatcher-3" prio=10 tid=0x00007f95884d0000 nid=0x6794 waiting on condition [0x00007f95870ec000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)

  • parking to wait for <0x00000007a874ef40> (a com.google.common.util.concurrent.AbstractFuture$Sync)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
    at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:285)
    at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
    at com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:137)
    at com.google.common.cache.LocalCache$LoadingValueReference.waitForValue(LocalCache.java:3557)
    at com.google.common.cache.LocalCache$Segment.waitForLoadingValue(LocalCache.java:2302)
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2191)
    at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
    at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
    at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
    at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4830)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator.loadSerializerFor(AbstractStreamWriterGenerator.java:99)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.DataObjectSerializerSource.staticInvokeEmitter(DataObjectSerializerSource.java:139)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.DataNodeContainerSerializerSource.emitList(DataNodeContainerSerializerSource.java:211)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.DataNodeContainerSerializerSource.emitChildInner(DataNodeContainerSerializerSource.java:176)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.DataNodeContainerSerializerSource.emitChild(DataNodeContainerSerializerSource.java:158)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.DataNodeContainerSerializerSource.emitBody(DataNodeContainerSerializerSource.java:148)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.DataNodeContainerSerializerSource.getSerializerBody(DataNodeContainerSerializerSource.java:78)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator$1$1.get(AbstractStreamWriterGenerator.java:202)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator$1$1.get(AbstractStreamWriterGenerator.java:199)
    at org.opendaylight.yangtools.util.ClassLoaderUtils.withClassLoader(ClassLoaderUtils.java:53)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator$1.customizeClass(AbstractStreamWriterGenerator.java:199)
    at org.opendaylight.yangtools.sal.binding.generator.util.JavassistUtils.instantiatePrototype(JavassistUtils.java:131)
  • locked <0x000000067b140020> (a org.opendaylight.yangtools.sal.binding.generator.util.JavassistUtils)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator.generateEmitter0(AbstractStreamWriterGenerator.java:192)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator.access$300(AbstractStreamWriterGenerator.java:46)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator$SerializerImplementationLoader.generateSerializer(AbstractStreamWriterGenerator.java:136)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator$SerializerImplementationLoader.load(AbstractStreamWriterGenerator.java:124)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator$SerializerImplementationLoader.load(AbstractStreamWriterGenerator.java:102)
    at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
    at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
    at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
  • locked <0x00000007a8d10a38> (a com.google.common.cache.LocalCache$WeakEntry)
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
    at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
    at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
    at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
    at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4830)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator.loadSerializerFor(AbstractStreamWriterGenerator.java:99)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.DataObjectSerializerSource.staticInvokeEmitter(DataObjectSerializerSource.java:139)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.DataNodeContainerSerializerSource.emitChildInner(DataNodeContainerSerializerSource.java:178)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.DataNodeContainerSerializerSource.emitChild(DataNodeContainerSerializerSource.java:158)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.DataNodeContainerSerializerSource.emitBody(DataNodeContainerSerializerSource.java:148)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.DataNodeContainerSerializerSource.getSerializerBody(DataNodeContainerSerializerSource.java:78)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator$1$1.get(AbstractStreamWriterGenerator.java:202)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator$1$1.get(AbstractStreamWriterGenerator.java:199)
    at org.opendaylight.yangtools.util.ClassLoaderUtils.withClassLoader(ClassLoaderUtils.java:53)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator$1.customizeClass(AbstractStreamWriterGenerator.java:199)
    at org.opendaylight.yangtools.sal.binding.generator.util.JavassistUtils.instantiatePrototype(JavassistUtils.java:131)
  • locked <0x000000067b140020> (a org.opendaylight.yangtools.sal.binding.generator.util.JavassistUtils)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator.generateEmitter0(AbstractStreamWriterGenerator.java:192)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator.access$300(AbstractStreamWriterGenerator.java:46)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator$SerializerImplementationLoader.generateSerializer(AbstractStreamWriterGenerator.java:136)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator$SerializerImplementationLoader.load(AbstractStreamWriterGenerator.java:124)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator$SerializerImplementationLoader.load(AbstractStreamWriterGenerator.java:102)
    at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
    at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
    at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
  • locked <0x00000007a7b1ca68> (a com.google.common.cache.LocalCache$WeakEntry)
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
    at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
    at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
    at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
    at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4830)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator.getSerializer(AbstractStreamWriterGenerator.java:89)
    at org.opendaylight.yangtools.binding.data.codec.impl.BindingNormalizedNodeCodecRegistry$GeneratorLoader.load(BindingNormalizedNodeCodecRegistry.java:276)
    at org.opendaylight.yangtools.binding.data.codec.impl.BindingNormalizedNodeCodecRegistry$GeneratorLoader.load(BindingNormalizedNodeCodecRegistry.java:273)
    at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
    at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
    at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
  • locked <0x00000007a7b1c9a8> (a com.google.common.cache.LocalCache$WeakEntry)
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
    at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
    at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
    at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
    at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4830)
    at org.opendaylight.yangtools.binding.data.codec.impl.BindingNormalizedNodeCodecRegistry.getSerializer(BindingNormalizedNodeCodecRegistry.java:72)
    at org.opendaylight.yangtools.binding.data.codec.impl.BindingNormalizedNodeCodecRegistry.toNormalizedNode(BindingNormalizedNodeCodecRegistry.java:106)
    at org.opendaylight.controller.md.sal.binding.impl.BindingToNormalizedNodeCodec.toNormalizedNode(BindingToNormalizedNodeCodec.java:133)
    at org.opendaylight.controller.md.sal.binding.impl.AbstractWriteTransaction.merge(AbstractWriteTransaction.java:53)
    at org.opendaylight.controller.md.sal.binding.impl.BindingDOMWriteTransactionAdapter.merge(BindingDOMWriteTransactionAdapter.java:46)
    at com.cisco.osc.dc.overlay.listeners.InventoryDataChangeListener.updateDevice(InventoryDataChangeListener.java:162)
    at com.cisco.osc.dc.overlay.listeners.InventoryDataChangeListener.onDataChanged(InventoryDataChangeListener.java:116)
    at org.opendaylight.controller.md.sal.binding.impl.AbstractForwardedDataBroker$TranslatingDataChangeInvoker.onDataChanged(AbstractForwardedDataBroker.java:143)
    at org.opendaylight.controller.cluster.datastore.DataChangeListener.dataChanged(DataChangeListener.java:66)
    at org.opendaylight.controller.cluster.datastore.DataChangeListener.handleReceive(DataChangeListener.java:39)
    at org.opendaylight.controller.cluster.common.actor.AbstractUntypedActor.onReceive(AbstractUntypedActor.java:34)
    at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:167)
    at akka.actor.Actor$class.aroundReceive(Actor.scala:467)
    at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:97)
    at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
    at akka.actor.ActorCell.invoke(ActorCell.scala:487)
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)
    at akka.dispatch.Mailbox.run(Mailbox.scala:221)
    at akka.dispatch.Mailbox.exec(Mailbox.scala:231)
    at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:1253)
    at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1346)
    at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

"pool-55-thread-1" prio=10 tid=0x0000000000bed800 nid=0x6933 waiting for monitor entry [0x00007f9554802000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.opendaylight.yangtools.sal.binding.generator.util.JavassistUtils.appendClassLoaderIfMissing(JavassistUtils.java:197)

  • waiting to lock <0x000000067b140020> (a org.opendaylight.yangtools.sal.binding.generator.util.JavassistUtils)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator.generateEmitterSource(AbstractStreamWriterGenerator.java:162)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator.access$200(AbstractStreamWriterGenerator.java:46)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator$SerializerImplementationLoader.generateSerializer(AbstractStreamWriterGenerator.java:135)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator$SerializerImplementationLoader.load(AbstractStreamWriterGenerator.java:124)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator$SerializerImplementationLoader.load(AbstractStreamWriterGenerator.java:102)
    at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
    at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
    at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
  • locked <0x00000007a874efa8> (a com.google.common.cache.LocalCache$WeakEntry)
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
    at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
    at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
    at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
    at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4830)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator.getSerializer(AbstractStreamWriterGenerator.java:89)
    at org.opendaylight.yangtools.binding.data.codec.impl.BindingNormalizedNodeCodecRegistry$GeneratorLoader.load(BindingNormalizedNodeCodecRegistry.java:276)
    at org.opendaylight.yangtools.binding.data.codec.impl.BindingNormalizedNodeCodecRegistry$GeneratorLoader.load(BindingNormalizedNodeCodecRegistry.java:273)
    at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
    at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
    at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
  • locked <0x00000007a874eee8> (a com.google.common.cache.LocalCache$WeakEntry)
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
    at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
    at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
    at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
    at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4830)
    at org.opendaylight.yangtools.binding.data.codec.impl.BindingNormalizedNodeCodecRegistry.getSerializer(BindingNormalizedNodeCodecRegistry.java:72)
    at org.opendaylight.yangtools.binding.data.codec.impl.BindingNormalizedNodeCodecRegistry.toNormalizedNode(BindingNormalizedNodeCodecRegistry.java:106)
    at org.opendaylight.controller.md.sal.binding.impl.BindingToNormalizedNodeCodec.toNormalizedNode(BindingToNormalizedNodeCodec.java:133)
    at org.opendaylight.controller.md.sal.binding.impl.AbstractWriteTransaction.merge(AbstractWriteTransaction.java:53)
    at org.opendaylight.controller.md.sal.binding.impl.BindingDOMWriteTransactionAdapter.merge(BindingDOMWriteTransactionAdapter.java:46)
    ...
Comment by Robert Varga [ 15/Dec/15 ]

Moving to MD-SAL, as that's where the code lives these days.

Comment by Moiz Raja [ 15/Dec/15 ]

Whatever the fix it will also be needed on stable/lithium

Comment by Robert Varga [ 16/Dec/15 ]

Be: https://git.opendaylight.org/gerrit/31439

Comment by Robert Varga [ 16/Dec/15 ]

Li: https://git.opendaylight.org/gerrit/31441

Generated at Wed Feb 07 20:08:42 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.