https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/MDSAL-122 mdsal <p>We are observing an issue on controller restart where modification operations (put/merge) on transactions get deadlocked on looking for a serializer. </p> <p>Here is the link to the threads that are locked up.</p> <p><a href="https://gist.github.com/moizr/079888bd1547559820f8" class="external-link" target="_blank" rel="nofollow noopener">https://gist.github.com/moizr/079888bd1547559820f8</a></p> <p>pool-55-thread-1 is BLOCKED on getting the Serializer.<br/> pendaylight-cluster-data-akka.actor.default-dispatcher-3 is WAITING for the generation of a Serializer to complete.</p> <p>It appears that the generating of the serializer is not completing and thus the deadlock.</p> <p>Operating System: All<br/> Platform: All</p> MDSAL-122

Deadlock on loading Binding Serializer

Bug Resolved Done Robert Varga Moiz Raja Fri, 11 Dec 2015 19:33:29 +0000 Fri, 9 Mar 2018 18:00:12 +0000 Mon, 8 Feb 2016 15:29:25 +0000 0 3 <p>Note that this issue is observed on Li-SR2</p> <p>Please provide full thread dump, thread dump you provided does not show any deadlock<br/> (pool thread is waiting for different monitor than akka thread) - which may get unblocked.<br/> In order to determine this is deadlock or livelock and how to fix it, we will need full thread dump and ideally steps to reproduce.</p> <p>"opendaylight-cluster-data-akka.actor.default-dispatcher-3" prio=10 tid=0x00007f95884d0000 nid=0x6794 waiting on condition <span class="error">&#91;0x00007f95870ec000&#93;</span><br/> java.lang.Thread.State: WAITING (parking)<br/> at sun.misc.Unsafe.park(Native Method)</p> <ul class="alternate" type="square"> <li>parking to wait for &lt;0x00000007a874ef40&gt; (a com.google.common.util.concurrent.AbstractFuture$Sync)<br/> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)<br/> at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)<br/> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)<br/> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)<br/> at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:285)<br/> at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)<br/> at com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:137)<br/> at com.google.common.cache.LocalCache$LoadingValueReference.waitForValue(LocalCache.java:3557)<br/> at com.google.common.cache.LocalCache$Segment.waitForLoadingValue(LocalCache.java:2302)<br/> at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2191)<br/> at com.google.common.cache.LocalCache.get(LocalCache.java:3937)<br/> at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)<br/> at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)<br/> at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4830)<br/> at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator.loadSerializerFor(AbstractStreamWriterGenerator.java:99)<br/> at org.opendaylight.yangtools.binding.data.codec.gen.impl.DataObjectSerializerSource.staticInvokeEmitter(DataObjectSerializerSource.java:139)<br/> at org.opendaylight.yangtools.binding.data.codec.gen.impl.DataNodeContainerSerializerSource.emitList(DataNodeContainerSerializerSource.java:211)<br/> at org.opendaylight.yangtools.binding.data.codec.gen.impl.DataNodeContainerSerializerSource.emitChildInner(DataNodeContainerSerializerSource.java:176)<br/> at org.opendaylight.yangtools.binding.data.codec.gen.impl.DataNodeContainerSerializerSource.emitChild(DataNodeContainerSerializerSource.java:158)<br/> at org.opendaylight.yangtools.binding.data.codec.gen.impl.DataNodeContainerSerializerSource.emitBody(DataNodeContainerSerializerSource.java:148)<br/> at org.opendaylight.yangtools.binding.data.codec.gen.impl.DataNodeContainerSerializerSource.getSerializerBody(DataNodeContainerSerializerSource.java:78)<br/> at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator$1$1.get(AbstractStreamWriterGenerator.java:202)<br/> at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator$1$1.get(AbstractStreamWriterGenerator.java:199)<br/> at org.opendaylight.yangtools.util.ClassLoaderUtils.withClassLoader(ClassLoaderUtils.java:53)<br/> at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator$1.customizeClass(AbstractStreamWriterGenerator.java:199)<br/> at org.opendaylight.yangtools.sal.binding.generator.util.JavassistUtils.instantiatePrototype(JavassistUtils.java:131)</li> <li>locked &lt;0x000000067b140020&gt; (a org.opendaylight.yangtools.sal.binding.generator.util.JavassistUtils)<br/> at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator.generateEmitter0(AbstractStreamWriterGenerator.java:192)<br/> at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator.access$300(AbstractStreamWriterGenerator.java:46)<br/> at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator$SerializerImplementationLoader.generateSerializer(AbstractStreamWriterGenerator.java:136)<br/> at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator$SerializerImplementationLoader.load(AbstractStreamWriterGenerator.java:124)<br/> at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator$SerializerImplementationLoader.load(AbstractStreamWriterGenerator.java:102)<br/> at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)<br/> at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)<br/> at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)</li> <li>locked &lt;0x00000007a8d10a38&gt; (a com.google.common.cache.LocalCache$WeakEntry)<br/> at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)<br/> at com.google.common.cache.LocalCache.get(LocalCache.java:3937)<br/> at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)<br/> at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)<br/> at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4830)<br/> at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator.loadSerializerFor(AbstractStreamWriterGenerator.java:99)<br/> at org.opendaylight.yangtools.binding.data.codec.gen.impl.DataObjectSerializerSource.staticInvokeEmitter(DataObjectSerializerSource.java:139)<br/> at org.opendaylight.yangtools.binding.data.codec.gen.impl.DataNodeContainerSerializerSource.emitChildInner(DataNodeContainerSerializerSource.java:178)<br/> at org.opendaylight.yangtools.binding.data.codec.gen.impl.DataNodeContainerSerializerSource.emitChild(DataNodeContainerSerializerSource.java:158)<br/> at org.opendaylight.yangtools.binding.data.codec.gen.impl.DataNodeContainerSerializerSource.emitBody(DataNodeContainerSerializerSource.java:148)<br/> at org.opendaylight.yangtools.binding.data.codec.gen.impl.DataNodeContainerSerializerSource.getSerializerBody(DataNodeContainerSerializerSource.java:78)<br/> at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator$1$1.get(AbstractStreamWriterGenerator.java:202)<br/> at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator$1$1.get(AbstractStreamWriterGenerator.java:199)<br/> at org.opendaylight.yangtools.util.ClassLoaderUtils.withClassLoader(ClassLoaderUtils.java:53)<br/> at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator$1.customizeClass(AbstractStreamWriterGenerator.java:199)<br/> at org.opendaylight.yangtools.sal.binding.generator.util.JavassistUtils.instantiatePrototype(JavassistUtils.java:131)</li> <li>locked &lt;0x000000067b140020&gt; (a org.opendaylight.yangtools.sal.binding.generator.util.JavassistUtils)<br/> at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator.generateEmitter0(AbstractStreamWriterGenerator.java:192)<br/> at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator.access$300(AbstractStreamWriterGenerator.java:46)<br/> at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator$SerializerImplementationLoader.generateSerializer(AbstractStreamWriterGenerator.java:136)<br/> at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator$SerializerImplementationLoader.load(AbstractStreamWriterGenerator.java:124)<br/> at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator$SerializerImplementationLoader.load(AbstractStreamWriterGenerator.java:102)<br/> at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)<br/> at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)<br/> at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)</li> <li>locked &lt;0x00000007a7b1ca68&gt; (a com.google.common.cache.LocalCache$WeakEntry)<br/> at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)<br/> at com.google.common.cache.LocalCache.get(LocalCache.java:3937)<br/> at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)<br/> at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)<br/> at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4830)<br/> at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator.getSerializer(AbstractStreamWriterGenerator.java:89)<br/> at org.opendaylight.yangtools.binding.data.codec.impl.BindingNormalizedNodeCodecRegistry$GeneratorLoader.load(BindingNormalizedNodeCodecRegistry.java:276)<br/> at org.opendaylight.yangtools.binding.data.codec.impl.BindingNormalizedNodeCodecRegistry$GeneratorLoader.load(BindingNormalizedNodeCodecRegistry.java:273)<br/> at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)<br/> at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)<br/> at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)</li> <li>locked &lt;0x00000007a7b1c9a8&gt; (a com.google.common.cache.LocalCache$WeakEntry)<br/> at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)<br/> at com.google.common.cache.LocalCache.get(LocalCache.java:3937)<br/> at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)<br/> at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)<br/> at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4830)<br/> at org.opendaylight.yangtools.binding.data.codec.impl.BindingNormalizedNodeCodecRegistry.getSerializer(BindingNormalizedNodeCodecRegistry.java:72)<br/> at org.opendaylight.yangtools.binding.data.codec.impl.BindingNormalizedNodeCodecRegistry.toNormalizedNode(BindingNormalizedNodeCodecRegistry.java:106)<br/> at org.opendaylight.controller.md.sal.binding.impl.BindingToNormalizedNodeCodec.toNormalizedNode(BindingToNormalizedNodeCodec.java:133)<br/> at org.opendaylight.controller.md.sal.binding.impl.AbstractWriteTransaction.merge(AbstractWriteTransaction.java:53)<br/> at org.opendaylight.controller.md.sal.binding.impl.BindingDOMWriteTransactionAdapter.merge(BindingDOMWriteTransactionAdapter.java:46)<br/> at com.cisco.osc.dc.overlay.listeners.InventoryDataChangeListener.updateDevice(InventoryDataChangeListener.java:162)<br/> at com.cisco.osc.dc.overlay.listeners.InventoryDataChangeListener.onDataChanged(InventoryDataChangeListener.java:116)<br/> at org.opendaylight.controller.md.sal.binding.impl.AbstractForwardedDataBroker$TranslatingDataChangeInvoker.onDataChanged(AbstractForwardedDataBroker.java:143)<br/> at org.opendaylight.controller.cluster.datastore.DataChangeListener.dataChanged(DataChangeListener.java:66)<br/> at org.opendaylight.controller.cluster.datastore.DataChangeListener.handleReceive(DataChangeListener.java:39)<br/> at org.opendaylight.controller.cluster.common.actor.AbstractUntypedActor.onReceive(AbstractUntypedActor.java:34)<br/> at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:167)<br/> at akka.actor.Actor$class.aroundReceive(Actor.scala:467)<br/> at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:97)<br/> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)<br/> at akka.actor.ActorCell.invoke(ActorCell.scala:487)<br/> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)<br/> at akka.dispatch.Mailbox.run(Mailbox.scala:221)<br/> at akka.dispatch.Mailbox.exec(Mailbox.scala:231)<br/> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)<br/> at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:1253)<br/> at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1346)<br/> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)<br/> at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)</li> </ul> <p>"pool-55-thread-1" prio=10 tid=0x0000000000bed800 nid=0x6933 waiting for monitor entry <span class="error">&#91;0x00007f9554802000&#93;</span><br/> java.lang.Thread.State: BLOCKED (on object monitor)<br/> at org.opendaylight.yangtools.sal.binding.generator.util.JavassistUtils.appendClassLoaderIfMissing(JavassistUtils.java:197)</p> <ul class="alternate" type="square"> <li>waiting to lock &lt;0x000000067b140020&gt; (a org.opendaylight.yangtools.sal.binding.generator.util.JavassistUtils)<br/> at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator.generateEmitterSource(AbstractStreamWriterGenerator.java:162)<br/> at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator.access$200(AbstractStreamWriterGenerator.java:46)<br/> at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator$SerializerImplementationLoader.generateSerializer(AbstractStreamWriterGenerator.java:135)<br/> at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator$SerializerImplementationLoader.load(AbstractStreamWriterGenerator.java:124)<br/> at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator$SerializerImplementationLoader.load(AbstractStreamWriterGenerator.java:102)<br/> at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)<br/> at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)<br/> at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)</li> <li>locked &lt;0x00000007a874efa8&gt; (a com.google.common.cache.LocalCache$WeakEntry)<br/> at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)<br/> at com.google.common.cache.LocalCache.get(LocalCache.java:3937)<br/> at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)<br/> at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)<br/> at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4830)<br/> at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator.getSerializer(AbstractStreamWriterGenerator.java:89)<br/> at org.opendaylight.yangtools.binding.data.codec.impl.BindingNormalizedNodeCodecRegistry$GeneratorLoader.load(BindingNormalizedNodeCodecRegistry.java:276)<br/> at org.opendaylight.yangtools.binding.data.codec.impl.BindingNormalizedNodeCodecRegistry$GeneratorLoader.load(BindingNormalizedNodeCodecRegistry.java:273)<br/> at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)<br/> at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)<br/> at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)</li> <li>locked &lt;0x00000007a874eee8&gt; (a com.google.common.cache.LocalCache$WeakEntry)<br/> at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)<br/> at com.google.common.cache.LocalCache.get(LocalCache.java:3937)<br/> at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)<br/> at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)<br/> at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4830)<br/> at org.opendaylight.yangtools.binding.data.codec.impl.BindingNormalizedNodeCodecRegistry.getSerializer(BindingNormalizedNodeCodecRegistry.java:72)<br/> at org.opendaylight.yangtools.binding.data.codec.impl.BindingNormalizedNodeCodecRegistry.toNormalizedNode(BindingNormalizedNodeCodecRegistry.java:106)<br/> at org.opendaylight.controller.md.sal.binding.impl.BindingToNormalizedNodeCodec.toNormalizedNode(BindingToNormalizedNodeCodec.java:133)<br/> at org.opendaylight.controller.md.sal.binding.impl.AbstractWriteTransaction.merge(AbstractWriteTransaction.java:53)<br/> at org.opendaylight.controller.md.sal.binding.impl.BindingDOMWriteTransactionAdapter.merge(BindingDOMWriteTransactionAdapter.java:46)<br/> ...</li> </ul> <p>Moving to MD-SAL, as that's where the code lives these days.</p> <p>Whatever the fix it will also be needed on stable/lithium</p> <p>Be: <a href="https://git.opendaylight.org/gerrit/31439" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/31439</a></p> <p>Li: <a href="https://git.opendaylight.org/gerrit/31441" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/31441</a></p> Duplicate AAA-48 Development External issue ID 4764 External issue URL Issue Type ODL SR Target Milestone Rank 0|i02wg7: