[GENIUS-17] Packet IN handler thread in deadlock after high ARP rate

Duplicate — Wed, 2 Nov 2016 09:10:30 +0000

Due to high ARP rate the notification queue fills up quickly.
ArpUtilService gets stuck on the blocking method putNotification() forever

Thread dump:
"pool-15-thread-1" #185 prio=5 os_prio=0 tid=0x00000000013ef800 nid=0x3057 runnable [0x00007f92a857d000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:338)
at com.lmax.disruptor.MultiProducerSequencer.next(MultiProducerSequencer.java:136)
at com.lmax.disruptor.MultiProducerSequencer.next(MultiProducerSequencer.java:105)
at com.lmax.disruptor.RingBuffer.next(RingBuffer.java:253)
at org.opendaylight.controller.md.sal.dom.broker.impl.DOMNotificationRouter.putNotification(DOMNotificationRouter.java:202)
at Proxya20eb9ef_033e_4602_ba27_a76d4290fa59.putNotification(Unknown Source)
at org.opendaylight.controller.md.sal.binding.impl.BindingDOMNotificationPublishServiceAdapter.putNotification(BindingDOMNotificationPublishServiceAdapter.java:51)
at org.opendaylight.genius.arputil.internal.ArpUtilImpl.fireArpReqRecvdNotification(ArpUtilImpl.java:620)
at org.opendaylight.genius.arputil.internal.ArpUtilImpl.onPacketReceived(ArpUtilImpl.java:501)
at java.lang.invoke.LambdaForm$DMH/266922253.invokeInterface_LL_V(LambdaForm$DMH)
at java.lang.invoke.LambdaForm$BMH/1678808217.reinvoke(LambdaForm$BMH)
at java.lang.invoke.LambdaForm$MH/732849773.invokeExact_MT(LambdaForm$MH)
at org.opendaylight.yangtools.yang.binding.util.NotificationListenerInvoker.invokeNotification(NotificationListenerInvoker.java:91)
at org.opendaylight.controller.md.sal.binding.impl.BindingDOMNotificationListenerAdapter.onNotification(BindingDOMNotificationListenerAdapter.java:44)
at org.opendaylight.controller.md.sal.dom.broker.impl.DOMNotificationRouterEvent.deliverNotification(DOMNotificationRouterEvent.java:56)
at org.opendaylight.controller.md.sal.dom.broker.impl.DOMNotificationRouter$1.onEvent(DOMNotificationRouter.java:68)
at org.opendaylight.controller.md.sal.dom.broker.impl.DOMNotificationRouter$1.onEvent(DOMNotificationRouter.java:65)
at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:129)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Need to consider using offerNotification rather than putNotification which is time-bounded or separate to other thread

OpenDaylight JIRA

[GENIUS-17] Packet IN handler thread in deadlock after high ARP rate