Deadlock between filterNotification of NetconfDevice and onSessionDown of NetconfDeviceCommunicator

Recently we met a deadlock between filterNotification of NetconfDevice and onSessionDown of NetconfDeviceCommunicator. The filterNotification of NetconfDevice was executed in a thread in the threadpool of remote-connector-processing-executor to filter a NetconfCapabilityChange notification which would cause disconnecting the netconf connector to odl internal netconf server. The onSessionDown of NetconfDeviceCommunicator was executed in a netty thread when the netty channel of netconf client received the close operation issued by the disconnecting. The onSessionDown would first hold the sessionLock and then would call NetconfDevice's onRemoteSessionDown which would call NotificationHandler's onRemoteSchemaDown which is a synchoronized method. But the monitor of NotificationHandler was held by the filterNotification of NetconfDevice from the beginning of its execution. Its following excecution was to call NetconfDeviceCommunicator's onSessionTerminated which would also try to hold sessionLock. Then deadlocked! The deadlock information from jstack is as following:

Found one Java-level deadlock:
=============================
"nettyThreadgroupModule$NioEventLoopGroupCloseable-2-4":
waiting to lock monitor 0x00007fda6408f4f8 (object 0x00000005c544ed48, a org.opendaylight.netconf.sal.connect.netconf.NotificationHandler),
which is held by "remote-connector-processing-executor-9"
"remote-connector-processing-executor-9":
waiting for ownable synchronizer 0x00000005c5450158, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
which is held by "nettyThreadgroupModule$NioEventLoopGroupCloseable-2-4"

Java stack information for the threads listed above:
===================================================
"nettyThreadgroupModule$NioEventLoopGroupCloseable-2-4":
at org.opendaylight.netconf.sal.connect.netconf.NotificationHandler.onRemoteSchemaDown(NotificationHandler.java:97)

• waiting to lock <0x00000005c544ed48> (a org.opendaylight.netconf.sal.connect.netconf.NotificationHandler)
  at org.opendaylight.netconf.sal.connect.netconf.NetconfDevice.onRemoteSessionDown(NetconfDevice.java:249)
  at org.opendaylight.netconf.sal.connect.netconf.listener.NetconfDeviceCommunicator.tearDown(NetconfDeviceCommunicator.java:175)
  at org.opendaylight.netconf.sal.connect.netconf.listener.NetconfDeviceCommunicator.onSessionDown(NetconfDeviceCommunicator.java:206)
  at org.opendaylight.netconf.sal.connect.netconf.listener.NetconfDeviceCommunicator.onSessionDown(NetconfDeviceCommunicator.java:47)
  at org.opendaylight.netconf.nettyutil.AbstractNetconfSession.endOfInput(AbstractNetconfSession.java:107)
  at org.opendaylight.protocol.framework.AbstractProtocolSession.channelInactive(AbstractProtocolSession.java:40)
  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:233)
  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:219)
  at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:212)
  at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:360)
  at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:325)
  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:233)
  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:219)
  at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:212)
  at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:360)
  at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:325)
  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:233)
  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:219)
  at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:212)
  at org.opendaylight.netconf.nettyutil.handler.ssh.client.AsyncSshHandler.disconnect(AsyncSshHandler.java:243)
• locked <0x00000005c9eb3300> (a org.opendaylight.netconf.nettyutil.handler.ssh.client.AsyncSshHandler)
  at org.opendaylight.netconf.nettyutil.handler.ssh.client.AsyncSshHandler.close(AsyncSshHandler.java:234)
  at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:604)
  at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:588)
  at io.netty.channel.ChannelOutboundHandlerAdapter.close(ChannelOutboundHandlerAdapter.java:71)
  at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:604)
  at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:588)
  at io.netty.channel.ChannelOutboundHandlerAdapter.close(ChannelOutboundHandlerAdapter.java:71)
  at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:604)
  at io.netty.channel.AbstractChannelHandlerContext.access$1100(AbstractChannelHandlerContext.java:33)
  at io.netty.channel.AbstractChannelHandlerContext$13.run(AbstractChannelHandlerContext.java:593)
  at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:358)
  at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:394)
  at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)
  at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:145)
  at java.lang.Thread.run(Thread.java:745)
  "remote-connector-processing-executor-9":
  at sun.misc.Unsafe.park(Native Method)
• parking to wait for <0x00000005c5450158> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
  at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
  at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
  at org.opendaylight.netconf.sal.connect.netconf.listener.NetconfDeviceCommunicator.tearDown(NetconfDeviceCommunicator.java:154)
  at org.opendaylight.netconf.sal.connect.netconf.listener.NetconfDeviceCommunicator.onSessionTerminated(NetconfDeviceCommunicator.java:212)
  at org.opendaylight.netconf.sal.connect.netconf.listener.NetconfDeviceCommunicator.onSessionTerminated(NetconfDeviceCommunicator.java:47)
  at org.opendaylight.netconf.nettyutil.AbstractNetconfSession.close(AbstractNetconfSession.java:58)
  at org.opendaylight.netconf.sal.connect.netconf.listener.NetconfDeviceCommunicator.disconnect(NetconfDeviceCommunicator.java:147)
  at org.opendaylight.netconf.sal.connect.netconf.NetconfDevice$3.filterNotification(NetconfDevice.java:178)
  at org.opendaylight.netconf.sal.connect.netconf.NotificationHandler.passNotification(NotificationHandler.java:87)
• locked <0x00000005c544ed48> (a org.opendaylight.netconf.sal.connect.netconf.NotificationHandler)
  at org.opendaylight.netconf.sal.connect.netconf.NotificationHandler.onRemoteSchemaUp(NotificationHandler.java:61)
• locked <0x00000005c544ed48> (a org.opendaylight.netconf.sal.connect.netconf.NotificationHandler)
  at org.opendaylight.netconf.sal.connect.netconf.NetconfDevice.handleSalInitializationSuccess(NetconfDevice.java:216)
  at org.opendaylight.netconf.sal.connect.netconf.NetconfDevice$RecursiveSchemaSetup$2.onSuccess(NetconfDevice.java:457)
  at org.opendaylight.netconf.sal.connect.netconf.NetconfDevice$RecursiveSchemaSetup$2.onSuccess(NetconfDevice.java:449)
  at com.google.common.util.concurrent.Futures$6.run(Futures.java:1319)
  at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
  at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
  at com.google.common.util.concurrent.ExecutionList.add(ExecutionList.java:101)
  at com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:170)
  at com.google.common.util.concurrent.ForwardingListenableFuture.addListener(ForwardingListenableFuture.java:47)
  at com.google.common.util.concurrent.Futures.addCallback(Futures.java:1322)
  at com.google.common.util.concurrent.Futures.addCallback(Futures.java:1258)
  at org.opendaylight.netconf.sal.connect.netconf.NetconfDevice$RecursiveSchemaSetup.setUpSchema(NetconfDevice.java:489)
  at org.opendaylight.netconf.sal.connect.netconf.NetconfDevice$RecursiveSchemaSetup.run(NetconfDevice.java:411)
  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  at java.lang.Thread.run(Thread.java:745)