Stable Lithium OFPlugin : Sporadic seizure of Flow Inventory when controller starts/restarts

Observed in : Lithium Stable release

Symptoms :
1. When controller starts, default flows do not get pushed
2. Querying Inventory Oper DS from RESTCONF would hang

Looking at FlowCapableInventoryProvider.enque method along with thread-dump, this MAY occur in few other scenarios too (not tested though)

Upon taking thread-dump following is noticed (see 'locked' cases)

"FlowCapableInventoryProvider" daemon prio=10 tid=0x00007f4a7d4ae000 nid=0x1e11 waiting on condition [0x00007f4a69cd9000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)

• parking to wait for <0x00000007a3927390> (a com.google.common.util.concurrent.AbstractFuture$Sync)
  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
  at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:285)
  at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
  at com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:137)
  at com.google.common.cache.LocalCache$LoadingValueReference.waitForValue(LocalCache.java:3557)
  at com.google.common.cache.LocalCache$Segment.waitForLoadingValue(LocalCache.java:2302)
  at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2191)
  at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
  at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
  at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
  at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4830)
  at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator.loadSerializerFor(AbstractStreamWriterGenerator.java:99)
  at org.opendaylight.yangtools.binding.data.codec.gen.impl.DataObjectSerializerSource.staticInvokeEmitter(DataObjectSerializerSource.java:139)
  at org.opendaylight.yangtools.binding.data.codec.gen.impl.DataNodeContainerSerializerSource.emitList(DataNodeContainerSerializerSource.java:211)
  at org.opendaylight.yangtools.binding.data.codec.gen.impl.DataNodeContainerSerializerSource.emitChildInner(DataNodeContainerSerializerSource.java:176)
  at org.opendaylight.yangtools.binding.data.codec.gen.impl.DataNodeContainerSerializerSource.emitChild(DataNodeContainerSerializerSource.java:158)
  at org.opendaylight.yangtools.binding.data.codec.gen.impl.DataNodeContainerSerializerSource.emitBody(DataNodeContainerSerializerSource.java:148)
  at org.opendaylight.yangtools.binding.data.codec.gen.impl.DataNodeContainerSerializerSource.getSerializerBody(DataNodeContainerSerializerSource.java:78)
  at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator$1$1.get(AbstractStreamWriterGenerator.java:202)
  at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator$1$1.get(AbstractStreamWriterGenerator.java:199)
  at org.opendaylight.yangtools.util.ClassLoaderUtils.withClassLoader(ClassLoaderUtils.java:53)
  at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator$1.customizeClass(AbstractStreamWriterGenerator.java:199)
  at org.opendaylight.yangtools.sal.binding.generator.util.JavassistUtils.instantiatePrototype(JavassistUtils.java:131)
• locked <0x00000007a20af098> (a org.opendaylight.yangtools.sal.binding.generator.util.JavassistUtils)
  at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator.generateEmitter0(AbstractStreamWriterGenerator.java:192)
  at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator.access$300(AbstractStreamWriterGenerator.java:46)
  at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator$SerializerImplementationLoader.generateSerializer(AbstractStreamWriterGenerator.java:136)
  at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator$SerializerImplementationLoader.load(AbstractStreamWriterGenerator.java:124)
  at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator$SerializerImplementationLoader.load(AbstractStreamWriterGenerator.java:102)
  at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
  at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
  at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
• locked <0x00000007a3923880> (a com.google.common.cache.LocalCache$WeakEntry)
  at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
  at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
  at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
  at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
  at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4830)
  at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator.loadSerializerFor(AbstractStreamWriterGenerator.java:99)
  at org.opendaylight.yangtools.binding.data.codec.gen.impl.DataObjectSerializerSource.staticInvokeEmitter(DataObjectSerializerSource.java:139)
  at org.opendaylight.yangtools.binding.data.codec.gen.impl.DataNodeContainerSerializerSource.emitList(DataNodeContainerSerializerSource.java:211)
  at org.opendaylight.yangtools.binding.data.codec.gen.impl.DataNodeContainerSerializerSource.emitChildInner(DataNodeContainerSerializerSource.java:176)
  at org.opendaylight.yangtools.binding.data.codec.gen.impl.DataNodeContainerSerializerSource.emitChild(DataNodeContainerSerializerSource.java:158)
  at org.opendaylight.yangtools.binding.data.codec.gen.impl.DataNodeContainerSerializerSource.emitBody(DataNodeContainerSerializerSource.java:148)
  at org.opendaylight.yangtools.binding.data.codec.gen.impl.DataNodeContainerSerializerSource.getSerializerBody(DataNodeContainerSerializerSource.java:78)
  at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator$1$1.get(AbstractStreamWriterGenerator.java:202)
  at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator$1$1.get(AbstractStreamWriterGenerator.java:199)
  at org.opendaylight.yangtools.util.ClassLoaderUtils.withClassLoader(ClassLoaderUtils.java:53)
  at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator$1.customizeClass(AbstractStreamWriterGenerator.java:199)
  at org.opendaylight.yangtools.sal.binding.generator.util.JavassistUtils.instantiatePrototype(JavassistUtils.java:131)
• locked <0x00000007a20af098> (a org.opendaylight.yangtools.sal.binding.generator.util.JavassistUtils)
  at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator.generateEmitter0(AbstractStreamWriterGenerator.java:192)
  at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator.access$300(AbstractStreamWriterGenerator.java:46)
  at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator$SerializerImplementationLoader.generateSerializer(AbstractStreamWriterGenerator.java:136)
  at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator$SerializerImplementationLoader.load(AbstractStreamWriterGenerator.java:124)
  at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator$SerializerImplementationLoader.load(AbstractStreamWriterGenerator.java:102)
  at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
  at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
  at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
• locked <0x00000007a3a59818> (a com.google.common.cache.LocalCache$WeakEntry)
  at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
  at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
  at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
  at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
  at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4830)
  at org.opendaylight.yangtools.binding.data.codec.gen.impl.AbstractStreamWriterGenerator.getSerializer(AbstractStreamWriterGenerator.java:89)
  at org.opendaylight.yangtools.binding.data.codec.impl.BindingNormalizedNodeCodecRegistry$GeneratorLoader.load(BindingNormalizedNodeCodecRegistry.java:276)
  at org.opendaylight.yangtools.binding.data.codec.impl.BindingNormalizedNodeCodecRegistry$GeneratorLoader.load(BindingNormalizedNodeCodecRegistry.java:273)
  at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
  at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
  at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
• locked <0x00000007a39275e8> (a com.google.common.cache.LocalCache$WeakEntry)
  at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
  at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
  at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
  at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
  at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4830)
  at org.opendaylight.yangtools.binding.data.codec.impl.BindingNormalizedNodeCodecRegistry.getSerializer(BindingNormalizedNodeCodecRegistry.java:72)
  at org.opendaylight.yangtools.binding.data.codec.impl.BindingNormalizedNodeCodecRegistry.toNormalizedNode(BindingNormalizedNodeCodecRegistry.java:106)
  at org.opendaylight.controller.md.sal.binding.impl.BindingToNormalizedNodeCodec.toNormalizedNode(BindingToNormalizedNodeCodec.java:133)
  at org.opendaylight.controller.md.sal.binding.impl.AbstractWriteTransaction.merge(AbstractWriteTransaction.java:53)
  at org.opendaylight.openflowplugin.applications.inventory.manager.NodeChangeCommiter$5.applyOperation(NodeChangeCommiter.java:181)
  at org.opendaylight.openflowplugin.applications.inventory.manager.FlowCapableInventoryProvider.createCustomizedTransaction(FlowCapableInventoryProvider.java:167)
  at org.opendaylight.openflowplugin.applications.inventory.manager.FlowCapableInventoryProvider.submitOperations(FlowCapableInventoryProvider.java:108)
  at org.opendaylight.openflowplugin.applications.inventory.manager.FlowCapableInventoryProvider.run(FlowCapableInventoryProvider.java:82)
  at java.lang.Thread.run(Thread.java:745)

Root lock points to
locked <0x00000007a20af098> (a org.opendaylight.yangtools.sal.binding.generator.util.JavassistUtils)

This occurs 4 or 5 out of 10 times for nodal reboot / start scenarios.