[GENIUS-17] Packet IN handler thread in deadlock after high ARP rate Created: 04/Sep/16  Updated: 02/Nov/16  Resolved: 02/Nov/16

Status: Resolved
Project: genius
Component/s: General
Affects Version/s: (unspecified)
Fix Version/s: None

Type: Bug
Reporter: Tali Ben-Meir Assignee: Shlomi Alfasi
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: All
Platform: All

Issue Links:
Duplicate
is duplicated by NETVIRT-226 VM can’t access neutron services Resolved
External issue ID: 6626

 Description   

Due to high ARP rate the notification queue fills up quickly.
ArpUtilService gets stuck on the blocking method putNotification() forever

Thread dump:
"pool-15-thread-1" #185 prio=5 os_prio=0 tid=0x00000000013ef800 nid=0x3057 runnable [0x00007f92a857d000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:338)
at com.lmax.disruptor.MultiProducerSequencer.next(MultiProducerSequencer.java:136)
at com.lmax.disruptor.MultiProducerSequencer.next(MultiProducerSequencer.java:105)
at com.lmax.disruptor.RingBuffer.next(RingBuffer.java:253)
at org.opendaylight.controller.md.sal.dom.broker.impl.DOMNotificationRouter.putNotification(DOMNotificationRouter.java:202)
at Proxya20eb9ef_033e_4602_ba27_a76d4290fa59.putNotification(Unknown Source)
at org.opendaylight.controller.md.sal.binding.impl.BindingDOMNotificationPublishServiceAdapter.putNotification(BindingDOMNotificationPublishServiceAdapter.java:51)
at org.opendaylight.genius.arputil.internal.ArpUtilImpl.fireArpReqRecvdNotification(ArpUtilImpl.java:620)
at org.opendaylight.genius.arputil.internal.ArpUtilImpl.onPacketReceived(ArpUtilImpl.java:501)
at java.lang.invoke.LambdaForm$DMH/266922253.invokeInterface_LL_V(LambdaForm$DMH)
at java.lang.invoke.LambdaForm$BMH/1678808217.reinvoke(LambdaForm$BMH)
at java.lang.invoke.LambdaForm$MH/732849773.invokeExact_MT(LambdaForm$MH)
at org.opendaylight.yangtools.yang.binding.util.NotificationListenerInvoker.invokeNotification(NotificationListenerInvoker.java:91)
at org.opendaylight.controller.md.sal.binding.impl.BindingDOMNotificationListenerAdapter.onNotification(BindingDOMNotificationListenerAdapter.java:44)
at org.opendaylight.controller.md.sal.dom.broker.impl.DOMNotificationRouterEvent.deliverNotification(DOMNotificationRouterEvent.java:56)
at org.opendaylight.controller.md.sal.dom.broker.impl.DOMNotificationRouter$1.onEvent(DOMNotificationRouter.java:68)
at org.opendaylight.controller.md.sal.dom.broker.impl.DOMNotificationRouter$1.onEvent(DOMNotificationRouter.java:65)
at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:129)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Need to consider using offerNotification rather than putNotification which is time-bounded or separate to other thread

 Comments   
Comment by Koby Aizer [ 02/Nov/16 ]

Fixed in https://git.opendaylight.org/gerrit/47623

Generated at Wed Feb 07 19:59:42 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.