[OPNFLWPLUG-548] Stable Lithium OFPlugin : Sporadic seizure of Flow Inventory when controller starts/restarts Created: 16/Sep/15  Updated: 27/Sep/21  Resolved: 21/Jul/16

Status: Resolved
Project: OpenFlowPlugin
Component/s: General
Affects Version/s: None
Fix Version/s: None

Type: Bug
Reporter: Muthukumaran Kothandaraman Assignee: Unassigned
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: All
Platform: All

Attachments: File thread.dump    
External issue ID: 4320

 Description   

Observed in : Lithium Stable release

Symptoms :
1. When controller starts, default flows do not get pushed
2. Querying Inventory Oper DS from RESTCONF would hang

Looking at FlowCapableInventoryProvider.enque method along with thread-dump, this MAY occur in few other scenarios too (not tested though)

Upon taking thread-dump following is noticed (see 'locked' cases)

"FlowCapableInventoryProvider" daemon prio=10 tid=0x00007f4a7d4ae000 nid=0x1e11 waiting on condition [0x00007f4a69cd9000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)

  • parking to wait for <0x00000007a3927390> (a com.google.common.util.concurrent.AbstractFuture$Sync)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
    at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:285)
    at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
    at com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:137)
    at com.google.common.cache.LocalCache$LoadingValueReference.waitForValue(LocalCache.java:3557)
    at com.google.common.cache.LocalCache$Segment.waitForLoadingValue(LocalCache.java:2302)
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2191)
    at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
    at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
    at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
    at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4830)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator.loadSerializerFor(AbstractStreamWriterGenerator.java:99)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.DataObjectSerializerSource.staticInvokeEmitter(DataObjectSerializerSource.java:139)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.DataNodeContainerSerializerSource.emitList(DataNodeContainerSerializerSource.java:211)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.DataNodeContainerSerializerSource.emitChildInner(DataNodeContainerSerializerSource.java:176)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.DataNodeContainerSerializerSource.emitChild(DataNodeContainerSerializerSource.java:158)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.DataNodeContainerSerializerSource.emitBody(DataNodeContainerSerializerSource.java:148)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.DataNodeContainerSerializerSource.getSerializerBody(DataNodeContainerSerializerSource.java:78)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator$1$1.get(AbstractStreamWriterGenerator.java:202)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator$1$1.get(AbstractStreamWriterGenerator.java:199)
    at org.opendaylight.yangtools.util.ClassLoaderUtils.withClassLoader(ClassLoaderUtils.java:53)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator$1.customizeClass(AbstractStreamWriterGenerator.java:199)
    at org.opendaylight.yangtools.sal.binding.generator.util.JavassistUtils.instantiatePrototype(JavassistUtils.java:131)
  • locked <0x00000007a20af098> (a org.opendaylight.yangtools.sal.binding.generator.util.JavassistUtils)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator.generateEmitter0(AbstractStreamWriterGenerator.java:192)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator.access$300(AbstractStreamWriterGenerator.java:46)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator$SerializerImplementationLoader.generateSerializer(AbstractStreamWriterGenerator.java:136)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator$SerializerImplementationLoader.load(AbstractStreamWriterGenerator.java:124)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator$SerializerImplementationLoader.load(AbstractStreamWriterGenerator.java:102)
    at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
    at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
    at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
  • locked <0x00000007a3923880> (a com.google.common.cache.LocalCache$WeakEntry)
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
    at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
    at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
    at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
    at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4830)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator.loadSerializerFor(AbstractStreamWriterGenerator.java:99)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.DataObjectSerializerSource.staticInvokeEmitter(DataObjectSerializerSource.java:139)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.DataNodeContainerSerializerSource.emitList(DataNodeContainerSerializerSource.java:211)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.DataNodeContainerSerializerSource.emitChildInner(DataNodeContainerSerializerSource.java:176)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.DataNodeContainerSerializerSource.emitChild(DataNodeContainerSerializerSource.java:158)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.DataNodeContainerSerializerSource.emitBody(DataNodeContainerSerializerSource.java:148)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.DataNodeContainerSerializerSource.getSerializerBody(DataNodeContainerSerializerSource.java:78)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator$1$1.get(AbstractStreamWriterGenerator.java:202)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator$1$1.get(AbstractStreamWriterGenerator.java:199)
    at org.opendaylight.yangtools.util.ClassLoaderUtils.withClassLoader(ClassLoaderUtils.java:53)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator$1.customizeClass(AbstractStreamWriterGenerator.java:199)
    at org.opendaylight.yangtools.sal.binding.generator.util.JavassistUtils.instantiatePrototype(JavassistUtils.java:131)
  • locked <0x00000007a20af098> (a org.opendaylight.yangtools.sal.binding.generator.util.JavassistUtils)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator.generateEmitter0(AbstractStreamWriterGenerator.java:192)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator.access$300(AbstractStreamWriterGenerator.java:46)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator$SerializerImplementationLoader.generateSerializer(AbstractStreamWriterGenerator.java:136)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator$SerializerImplementationLoader.load(AbstractStreamWriterGenerator.java:124)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator$SerializerImplementationLoader.load(AbstractStreamWriterGenerator.java:102)
    at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
    at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
    at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
  • locked <0x00000007a3a59818> (a com.google.common.cache.LocalCache$WeakEntry)
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
    at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
    at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
    at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
    at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4830)
    at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator.getSerializer(AbstractStreamWriterGenerator.java:89)
    at org.opendaylight.yangtools.binding.data.codec.impl.BindingNormalizedNodeCodecRegistry$GeneratorLoader.load(BindingNormalizedNodeCodecRegistry.java:276)
    at org.opendaylight.yangtools.binding.data.codec.impl.BindingNormalizedNodeCodecRegistry$GeneratorLoader.load(BindingNormalizedNodeCodecRegistry.java:273)
    at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
    at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
    at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
  • locked <0x00000007a39275e8> (a com.google.common.cache.LocalCache$WeakEntry)
    at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
    at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
    at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
    at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
    at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4830)
    at org.opendaylight.yangtools.binding.data.codec.impl.BindingNormalizedNodeCodecRegistry.getSerializer(BindingNormalizedNodeCodecRegistry.java:72)
    at org.opendaylight.yangtools.binding.data.codec.impl.BindingNormalizedNodeCodecRegistry.toNormalizedNode(BindingNormalizedNodeCodecRegistry.java:106)
    at org.opendaylight.controller.md.sal.binding.impl.BindingToNormalizedNodeCodec.toNormalizedNode(BindingToNormalizedNodeCodec.java:133)
    at org.opendaylight.controller.md.sal.binding.impl.AbstractWriteTransaction.merge(AbstractWriteTransaction.java:53)
    at org.opendaylight.openflowplugin.applications.inventory.manager.NodeChangeCommiter$5.applyOperation(NodeChangeCommiter.java:181)
    at org.opendaylight.openflowplugin.applications.inventory.manager.FlowCapableInventoryProvider.createCustomizedTransaction(FlowCapableInventoryProvider.java:167)
    at org.opendaylight.openflowplugin.applications.inventory.manager.FlowCapableInventoryProvider.submitOperations(FlowCapableInventoryProvider.java:108)
    at org.opendaylight.openflowplugin.applications.inventory.manager.FlowCapableInventoryProvider.run(FlowCapableInventoryProvider.java:82)
    at java.lang.Thread.run(Thread.java:745)

Root lock points to
locked <0x00000007a20af098> (a org.opendaylight.yangtools.sal.binding.generator.util.JavassistUtils)

This occurs 4 or 5 out of 10 times for nodal reboot / start scenarios.

 Comments   
Comment by Muthukumaran Kothandaraman [ 16/Sep/15 ]

Attachment thread.dump has been added with description: Thread dump for sporadic seizure in FlowCapableInventoryProvider

Comment by Muthukumaran Kothandaraman [ 21/Jul/16 ]

No longer seen in later builds.

Generated at Wed Feb 07 20:32:46 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.