https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/NETCONF-284 netconf <p>Recently we met a deadlock between filterNotification of NetconfDevice and onSessionDown of NetconfDeviceCommunicator. The filterNotification of NetconfDevice was executed in a thread in the threadpool of remote-connector-processing-executor to filter a NetconfCapabilityChange notification which would cause disconnecting the netconf connector to odl internal netconf server. The onSessionDown of NetconfDeviceCommunicator was executed in a netty thread when the netty channel of netconf client received the close operation issued by the disconnecting. The onSessionDown would first hold the sessionLock and then would call NetconfDevice's onRemoteSessionDown which would call NotificationHandler's onRemoteSchemaDown which is a synchoronized method. But the monitor of NotificationHandler was held by the filterNotification of NetconfDevice from the beginning of its execution. Its following excecution was to call NetconfDeviceCommunicator's onSessionTerminated which would also try to hold sessionLock. Then deadlocked! The deadlock information from jstack is as following:</p> <p>Found one Java-level deadlock:<br/> =============================<br/> "nettyThreadgroupModule$NioEventLoopGroupCloseable-2-4":<br/> waiting to lock monitor 0x00007fda6408f4f8 (object 0x00000005c544ed48, a org.opendaylight.netconf.sal.connect.netconf.NotificationHandler),<br/> which is held by "remote-connector-processing-executor-9"<br/> "remote-connector-processing-executor-9":<br/> waiting for ownable synchronizer 0x00000005c5450158, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),<br/> which is held by "nettyThreadgroupModule$NioEventLoopGroupCloseable-2-4"</p> <p>Java stack information for the threads listed above:<br/> ===================================================<br/> "nettyThreadgroupModule$NioEventLoopGroupCloseable-2-4":<br/> at org.opendaylight.netconf.sal.connect.netconf.NotificationHandler.onRemoteSchemaDown(NotificationHandler.java:97)</p> <ul class="alternate" type="square"> <li>waiting to lock &lt;0x00000005c544ed48&gt; (a org.opendaylight.netconf.sal.connect.netconf.NotificationHandler)<br/> at org.opendaylight.netconf.sal.connect.netconf.NetconfDevice.onRemoteSessionDown(NetconfDevice.java:249)<br/> at org.opendaylight.netconf.sal.connect.netconf.listener.NetconfDeviceCommunicator.tearDown(NetconfDeviceCommunicator.java:175)<br/> at org.opendaylight.netconf.sal.connect.netconf.listener.NetconfDeviceCommunicator.onSessionDown(NetconfDeviceCommunicator.java:206)<br/> at org.opendaylight.netconf.sal.connect.netconf.listener.NetconfDeviceCommunicator.onSessionDown(NetconfDeviceCommunicator.java:47)<br/> at org.opendaylight.netconf.nettyutil.AbstractNetconfSession.endOfInput(AbstractNetconfSession.java:107)<br/> at org.opendaylight.protocol.framework.AbstractProtocolSession.channelInactive(AbstractProtocolSession.java:40)<br/> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:233)<br/> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:219)<br/> at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:212)<br/> at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:360)<br/> at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:325)<br/> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:233)<br/> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:219)<br/> at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:212)<br/> at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:360)<br/> at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:325)<br/> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:233)<br/> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:219)<br/> at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:212)<br/> at org.opendaylight.netconf.nettyutil.handler.ssh.client.AsyncSshHandler.disconnect(AsyncSshHandler.java:243)</li> <li>locked &lt;0x00000005c9eb3300&gt; (a org.opendaylight.netconf.nettyutil.handler.ssh.client.AsyncSshHandler)<br/> at org.opendaylight.netconf.nettyutil.handler.ssh.client.AsyncSshHandler.close(AsyncSshHandler.java:234)<br/> at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:604)<br/> at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:588)<br/> at io.netty.channel.ChannelOutboundHandlerAdapter.close(ChannelOutboundHandlerAdapter.java:71)<br/> at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:604)<br/> at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:588)<br/> at io.netty.channel.ChannelOutboundHandlerAdapter.close(ChannelOutboundHandlerAdapter.java:71)<br/> at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:604)<br/> at io.netty.channel.AbstractChannelHandlerContext.access$1100(AbstractChannelHandlerContext.java:33)<br/> at io.netty.channel.AbstractChannelHandlerContext$13.run(AbstractChannelHandlerContext.java:593)<br/> at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:358)<br/> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:394)<br/> at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)<br/> at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:145)<br/> at java.lang.Thread.run(Thread.java:745)<br/> "remote-connector-processing-executor-9":<br/> at sun.misc.Unsafe.park(Native Method)</li> <li>parking to wait for &lt;0x00000005c5450158&gt; (a java.util.concurrent.locks.ReentrantLock$NonfairSync)<br/> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)<br/> at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)<br/> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)<br/> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)<br/> at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)<br/> at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)<br/> at org.opendaylight.netconf.sal.connect.netconf.listener.NetconfDeviceCommunicator.tearDown(NetconfDeviceCommunicator.java:154)<br/> at org.opendaylight.netconf.sal.connect.netconf.listener.NetconfDeviceCommunicator.onSessionTerminated(NetconfDeviceCommunicator.java:212)<br/> at org.opendaylight.netconf.sal.connect.netconf.listener.NetconfDeviceCommunicator.onSessionTerminated(NetconfDeviceCommunicator.java:47)<br/> at org.opendaylight.netconf.nettyutil.AbstractNetconfSession.close(AbstractNetconfSession.java:58)<br/> at org.opendaylight.netconf.sal.connect.netconf.listener.NetconfDeviceCommunicator.disconnect(NetconfDeviceCommunicator.java:147)<br/> at org.opendaylight.netconf.sal.connect.netconf.NetconfDevice$3.filterNotification(NetconfDevice.java:178)<br/> at org.opendaylight.netconf.sal.connect.netconf.NotificationHandler.passNotification(NotificationHandler.java:87)</li> <li>locked &lt;0x00000005c544ed48&gt; (a org.opendaylight.netconf.sal.connect.netconf.NotificationHandler)<br/> at org.opendaylight.netconf.sal.connect.netconf.NotificationHandler.onRemoteSchemaUp(NotificationHandler.java:61)</li> <li>locked &lt;0x00000005c544ed48&gt; (a org.opendaylight.netconf.sal.connect.netconf.NotificationHandler)<br/> at org.opendaylight.netconf.sal.connect.netconf.NetconfDevice.handleSalInitializationSuccess(NetconfDevice.java:216)<br/> at org.opendaylight.netconf.sal.connect.netconf.NetconfDevice$RecursiveSchemaSetup$2.onSuccess(NetconfDevice.java:457)<br/> at org.opendaylight.netconf.sal.connect.netconf.NetconfDevice$RecursiveSchemaSetup$2.onSuccess(NetconfDevice.java:449)<br/> at com.google.common.util.concurrent.Futures$6.run(Futures.java:1319)<br/> at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)<br/> at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)<br/> at com.google.common.util.concurrent.ExecutionList.add(ExecutionList.java:101)<br/> at com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:170)<br/> at com.google.common.util.concurrent.ForwardingListenableFuture.addListener(ForwardingListenableFuture.java:47)<br/> at com.google.common.util.concurrent.Futures.addCallback(Futures.java:1322)<br/> at com.google.common.util.concurrent.Futures.addCallback(Futures.java:1258)<br/> at org.opendaylight.netconf.sal.connect.netconf.NetconfDevice$RecursiveSchemaSetup.setUpSchema(NetconfDevice.java:489)<br/> at org.opendaylight.netconf.sal.connect.netconf.NetconfDevice$RecursiveSchemaSetup.run(NetconfDevice.java:411)<br/> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)<br/> at java.util.concurrent.FutureTask.run(FutureTask.java:266)<br/> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)<br/> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)<br/> at java.lang.Thread.run(Thread.java:745)</li> </ul> <p>Operating System: Linux<br/> Platform: PC</p> NETCONF-284

Deadlock between filterNotification of NetconfDevice and onSessionDown of NetconfDeviceCommunicator

Bug Resolved Done Kangqian Yin Kangqian Yin Sat, 24 Sep 2016 03:52:46 +0000 Fri, 15 Mar 2019 22:22:30 +0000 Fri, 30 Sep 2016 11:27:14 +0000 netconf 0 2 <p>Hi Kanggian,</p> <p>Thank you for your report. How many times have you ran into this deadlock?</p> <p>100% deadlocking in installing one of our own features.</p> <p>I've found the deadlocking condition and the way to break the conditon. I'll push a patch to fix it later.</p> <p>I can confirm this deadlock. The thing is that onCapabilityChange notification should be cached and after successful connection is processed by thread of remote-connector-processing-executor. Not cached notifications are processed in netty thread so they shouldn't deadlock.</p> <p>I simulated this behavior by always caching onCapabilityChange notification, but I wasn't able to reproduce this deadlock on every reconnect. </p> <p>So If I am missing something, please let me know what other condition has to be met. Also patches are always welcomed so feel free to push your fix for this deadlock. Since you are working on this, you can also take this bug here in bugzilla.</p> <p>I met this deadlock in Beryllium-SR3 distribution. In this version, YangStoreService's notifiyListeners has the function to issue NetconfCapabilityChange notification to registered remote netconf clients. </p> <p>Just as you say, if the NetconfCapabilityChange notification is executed in netty thread, deadlock won't happen. </p> <p>In all the deadlocks I met, the NetconfCapabilityChange notification is executed in a thread of remote-connector-processing-executor in the calling tree of NetconfDevice$RecursiveSchemaSetup.setUpSchema which is submitted by NetconfDevice.onRemoteSessionUp, i.e, triggered when the netconf session is reconnected. The filter of NetconfCapabilityChange will cause disconnect this netconf session again. </p> <p>The deadlock condition requires there must be a netty thread executing the close of this netconf session at the same time. If so, the two threads will race for NetconfDeviceCommunicator's tearDown operation. If the netty thread enters tearDown first, deadlock will happen for such tearDown cannot take the monitor of NotificationHandler. The monitor of Notification is held by the remote-connector-processing-executor thread at the beginning of filtering of NetconfCapabilityChange.</p> <p>Nevertheless, reproducing this deadlock is hard. There're several features I've tested, but only one feature to install will cause this deadlock.</p> <p>However, the racing of NetconfDeviceCommunicator's tearDown actually exists and can be drawn from just the code of NetconfDeviceCommunicator. Both onSessionDown and onSessionTerminated will call tearDown, and they can be executed in two independent threads seperately.</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/46270/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/46270/</a></p> Development External issue ID 6797 External issue URL Rank 0|i01xlb: