[OPNFLWPLUG-599] He: FlowCapableInventoryProvider thread dies

Mon, 27 Sep 2021 09:01:42 +0000

The thread that polls from the queue (Named “FlowCapableInventoryProvider”) died at our system, because of an exception in one of the handlers.
The run() loop should be protected from exceptions.

The Exception that made the thread die:
Exception in thread "FlowCapableInventoryProvider" java.lang.IllegalArgumentException: Instance identifier references (urn:opendaylight:inventory?revision=2013-08-19)node-connector[

{(urn:opendaylight:inventory?revision=2013-08-19)id=openflow:4:3}

] but data identifier is ImmutableMapEntryNode{nodeIdentifier=(urn:opendaylight:inventory?revision=2013-08-19)node-connector[

{(urn:opendaylight:inventory?revision=2013-08-19)id=3}

], value=[ImmutableLeafNode{nodeIdentifier=(urn:opendaylight:inventory?revision=2013-08-19)id, value=3, attributes={}}], attributes={}}

This caused deadlocks in the system, because inventory change operations are now stuck.

This is the thread that polls from the Notifications Queue, it is stuck because the FlowCapableInventoryProvider is dead:
"pool-31-thread-1" prio=10 tid=0x00007fd9d4f02000 nid=0x17e7 waiting on condition [0x00007fda80f70000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)

parking to wait for <0x000000072b8d69c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(Unknown Source)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source)
at java.util.concurrent.LinkedBlockingDeque.putLast(Unknown Source)
at java.util.concurrent.LinkedBlockingDeque.put(Unknown Source)
at org.opendaylight.openflowplugin.applications.inventory.manager.FlowCapableInventoryProvider.enqueue(FlowCapableInventoryProvider.java:60)
at org.opendaylight.openflowplugin.applications.inventory.manager.NodeChangeCommiter.onNodeRemoved(NodeChangeCommiter.java:125)
locked <0x000000072b8d6a40> (a org.opendaylight.openflowplugin.applications.inventory.manager.NodeChangeCommiter)
at org.opendaylight.yangtools.yang.binding.util.NotificationListenerInvoker.invokeNotification(NotificationListenerInvoker.java:91)
at org.opendaylight.controller.md.sal.binding.impl.BindingDOMNotificationListenerAdapter.onNotification(BindingDOMNotificationListenerAdapter.java:44)
at org.opendaylight.controller.md.sal.dom.broker.impl.DOMNotificationRouterEvent.deliverNotification(DOMNotificationRouterEvent.java:50)
at org.opendaylight.controller.md.sal.dom.broker.impl.DOMNotificationRouter$1.onEvent(DOMNotificationRouter.java:68)
at org.opendaylight.controller.md.sal.dom.broker.impl.DOMNotificationRouter$1.onEvent(DOMNotificationRouter.java:65)
at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

So the Notifications Queue is full, and then we can’t put new Notifications:
"nioEventLoopGroup-11-83" prio=10 tid=0x00007fd698061800 nid=0xb7bf runnable [0x00007fd5a6997000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
at com.lmax.disruptor.MultiProducerSequencer.next(MultiProducerSequencer.java:136)
at com.lmax.disruptor.MultiProducerSequencer.next(MultiProducerSequencer.java:105)
at com.lmax.disruptor.RingBuffer.next(RingBuffer.java:246)
at org.opendaylight.controller.md.sal.dom.broker.impl.DOMNotificationRouter.putNotification(DOMNotificationRouter.java:200)
at org.opendaylight.controller.md.sal.binding.impl.BindingDOMNotificationPublishServiceAdapter.putNotification(BindingDOMNotificationPublishServiceAdapter.java:51)
at org.opendaylight.controller.md.sal.binding.compat.HeliumNotificationProviderServiceAdapter.publish(HeliumNotificationProviderServiceAdapter.java:33)
at org.opendaylight.openflowplugin.openflow.md.core.sal.OFRpcTaskUtil$1FutureCallbackImpl.onSuccess(OFRpcTaskUtil.java:115)
at org.opendaylight.openflowplugin.openflow.md.core.sal.OFRpcTaskUtil$1FutureCallbackImpl.onSuccess(OFRpcTaskUtil.java:101)
….

OpenDaylight JIRA

[OPNFLWPLUG-599] He: FlowCapableInventoryProvider thread dies