https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/NETCONF-905 netconf <p>The fix for <a href="https://jira.opendaylight.org/browse/NETCONF-827" title="Mount loop when setting too low connection-timeout" class="issue-link" data-issue-key="NETCONF-827"><del>NETCONF-827</del></a> addressed the race between channel message dispatch &amp; timeout. However, NETCONF session negotiation can still fail putting NetconfSessionPromise in the failed state. Once this happens following reconnection attempts initiated by ReconnectStrategy all fail because of the failed state.</p> <p>For example, the following log shows a case where session negotiation fails because NETCONF channel has been closed before hello message is sent:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2022-10-13T21:47:49,161 | INFO | globalWorkerGroup-3-2 | AbstractNetconfSessionNegotiator | 258 - org.opendaylight.netconf.netty-util - 4.0.3.SNAPSHOT | Failed to send message &lt;hello xmlns="urn:ietf:params:xml:ns:netconf:base:1.0"&gt; &lt;capabilities&gt; &lt;capability&gt;urn:ietf:params:netconf:capability:exi:1.0&lt;/capability&gt; &lt;capability&gt;urn:ietf:params:netconf:base:1.1&lt;/capability&gt; &lt;capability&gt;urn:ietf:params:netconf:base:1.0&lt;/capability&gt; &lt;/capabilities&gt; &lt;/hello&gt; on channel [id: 0x738232cc] java.lang.IllegalStateException: Channel closed at org.opendaylight.netconf.nettyutil.handler.ssh.client.AsyncSshHandlerWriter.write(AsyncSshHandlerWriter.java:59) ~[bundleFile:?] at org.opendaylight.netconf.nettyutil.handler.ssh.client.AsyncSshHandler.write(AsyncSshHandler.java:186) ~[bundleFile:?] at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717) ~[bundleFile:4.1.76.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:709) ~[bundleFile:4.1.76.Final] at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:792) ~[bundleFile:4.1.76.Final] at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:702) ~[bundleFile:4.1.76.Final] at io.netty.handler.codec.MessageToByteEncoder.write(MessageToByteEncoder.java:113) ~[bundleFile:4.1.76.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717) ~[bundleFile:4.1.76.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:709) ~[bundleFile:4.1.76.Final] at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:792) ~[bundleFile:4.1.76.Final] at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:702) ~[bundleFile:4.1.76.Final] at io.netty.handler.codec.MessageToByteEncoder.write(MessageToByteEncoder.java:113) ~[bundleFile:4.1.76.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717) ~[bundleFile:4.1.76.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:764) ~[bundleFile:4.1.76.Final] at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:790) ~[bundleFile:4.1.76.Final] at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:758) ~[bundleFile:4.1.76.Final] at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:808) ~[bundleFile:4.1.76.Final] at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1025) ~[bundleFile:4.1.76.Final] at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:306) ~[bundleFile:4.1.76.Final] at org.opendaylight.netconf.nettyutil.AbstractNetconfSessionNegotiator.sendMessage(AbstractNetconfSessionNegotiator.java:340) ~[bundleFile:?] at org.opendaylight.netconf.nettyutil.AbstractNetconfSessionNegotiator.start(AbstractNetconfSessionNegotiator.java:150) ~[bundleFile:?] at org.opendaylight.netconf.nettyutil.AbstractNetconfSessionNegotiator.startNegotiation(AbstractNetconfSessionNegotiator.java:131) ~[bundleFile:?] at org.opendaylight.netconf.nettyutil.AbstractNetconfSessionNegotiator.channelActive(AbstractNetconfSessionNegotiator.java:355) ~[bundleFile:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:230) ~[bundleFile:4.1.76.Final] at io.netty.channel.AbstractChannelHandlerContext.access$200(AbstractChannelHandlerContext.java:61) ~[bundleFile:4.1.76.Final] at io.netty.channel.AbstractChannelHandlerContext$3.run(AbstractChannelHandlerContext.java:221) ~[bundleFile:4.1.76.Final] at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) ~[bundleFile:4.1.76.Final] at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469) ~[bundleFile:4.1.76.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:503) ~[bundleFile:4.1.76.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) ~[bundleFile:4.1.76.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[bundleFile:4.1.76.Final] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[bundleFile:4.1.76.Final] at java.lang.Thread.run(Thread.java:833) ~[?:?] </pre> </div></div> <p>Once this event happens, NETCONF session cannot be established because the promise cannot be set to 'SUCCESS' even if all the negotiation has been completed successfully:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2022-10-13T21:48:43,976 | WARN | globalWorkerGroup-3-2 | AbstractNetconfSessionNegotiator | 258 - org.opendaylight.netconf.netty-util - 4.0.3.SNAPSHOT | An exception occurred during negotiation with null on channel [id: 0x738232cc] java.lang.IllegalStateException: complete already: NetconfSessionPromise@37c85c76(failure: java.lang.IllegalStateException: Channel closed) at io.netty.util.concurrent.DefaultPromise.setFailure(DefaultPromise.java:112) ~[bundleFile:4.1.76.Final] at org.opendaylight.netconf.nettyutil.AbstractNetconfSessionNegotiator.negotiationFailed(AbstractNetconfSessionNegotiator.java:330) ~[bundleFile:?] at org.opendaylight.netconf.nettyutil.AbstractNetconfSessionNegotiator.channelRead(AbstractNetconfSessionNegotiator.java:370) ~[bundleFile:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[bundleFile:4.1.76.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[bundleFile:4.1.76.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[bundleFile:4.1.76.Final] at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327) ~[bundleFile:4.1.76.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299) ~[bundleFile:4.1.76.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[bundleFile:4.1.76.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[bundleFile:4.1.76.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[bundleFile:4.1.76.Final] at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327) ~[bundleFile:4.1.76.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299) ~[bundleFile:4.1.76.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[bundleFile:4.1.76.Final] at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:61) ~[bundleFile:4.1.76.Final] at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:370) ~[bundleFile:4.1.76.Final] at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) ~[bundleFile:4.1.76.Final] at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469) ~[bundleFile:4.1.76.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:503) ~[bundleFile:4.1.76.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) ~[bundleFile:4.1.76.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[bundleFile:4.1.76.Final] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[bundleFile:4.1.76.Final] at java.lang.Thread.run(Thread.java:833) ~[?:?] Caused by: java.lang.IllegalStateException: complete already: NetconfSessionPromise@37c85c76(failure: java.lang.IllegalStateException: Channel closed) at io.netty.util.concurrent.DefaultPromise.setSuccess(DefaultPromise.java:99) ~[bundleFile:4.1.76.Final] at org.opendaylight.netconf.nettyutil.NetconfSessionPromise.setSuccess(NetconfSessionPromise.java:88) ~[bundleFile:?] at org.opendaylight.netconf.nettyutil.NetconfSessionPromise.setSuccess(NetconfSessionPromise.java:26) ~[bundleFile:?] at org.opendaylight.netconf.nettyutil.AbstractNetconfSessionNegotiator.negotiationSuccessful(AbstractNetconfSessionNegotiator.java:324) ~[bundleFile:?] at org.opendaylight.netconf.client.NetconfClientSessionNegotiator.handleMessage(NetconfClientSessionNegotiator.java:88) ~[bundleFile:?] at org.opendaylight.netconf.nettyutil.AbstractNetconfSessionNegotiator.channelRead(AbstractNetconfSessionNegotiator.java:367) ~[bundleFile:?] ... 20 more </pre> </div></div> NETCONF-905

NetconfSessionPromise reconnection failure

Bug Medium Resolved Done Robert Varga Sangwook Ha Thu, 13 Oct 2022 22:18:51 +0000 Mon, 31 Oct 2022 07:40:57 +0000 Thu, 20 Oct 2022 09:43:24 +0000 3.0.6 4.0.2 3.0.7 4.0.3 2.0.17 netconf 0 4 <p>So something like this would work: <a href="https://git.opendaylight.org/gerrit/c/netconf/+/102590" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/c/netconf/+/102590</a><br/> But I'm not sure why session negotiation failure due to timeout requires a special treatment where promise is not failed, so negotiation is retried with the same promise. It seems to be just one of the failure scenarios - since timer does not start until channel becomes active, it does not cover pure reachability issue different from other exceptions.</p> <p>Yeah, but that patch seems to have other issues. Gawking at the stack trace, it seems a tad strange: the channel was closed before channelActive() was invoked?!</p> <p>Can you post logs with AsyncSshHandler at debug, please?</p> <p>I wonder whether AsyncSshHandler.handleSshChanelOpened() is executing outside the event loop and we end up somehow circling to AsyncSshHandler.safelyDisconnect() before channelActive() is processed...</p> <p>Post karaf log at the DEBUG level. The error was triggered when device configuration was reapplied a few times forcing controller to establish new NETCONF connections while interrupting the negotiation process at breakpoints set in AbstractNetconfSessionNegotiator (channelActive, channelRead).</p> <p>Looks like the device initiated disconnection probably because of the long idle time before it receives hello back:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2022-10-14T21:20:14,016 | DEBUG | globalWorkerGroup-3-5 | AbstractNetconfSessionNegotiator | 258 - org.opendaylight.netconf.netty-util - 4.0.3.SNAPSHOT | Starting session negotiation on channel [id: 0x1d4d427c] 2022-10-14T21:20:40,839 | DEBUG | sshd-NetconfSshClient[a8b2d7e]-nio2-thread-6 | NettyAwareChannelSubsystem | 265 - org.opendaylight.netconf.shaded-sshd - 4.0.3.SNAPSHOT | handleEof(NettyAwareChannelSubsystem[id=0, recipient=43]-NetconfClientSessionImpl[netconf@/10.100.18.43:830][netconf]) SSH_MSG_CHANNEL_EOF 2022-10-14T21:20:40,840 | DEBUG | sshd-NetconfSshClient[a8b2d7e]-nio2-thread-6 | NettyAwareChannelSubsystem | 265 - org.opendaylight.netconf.shaded-sshd - 4.0.3.SNAPSHOT | handleClose(NettyAwareChannelSubsystem[id=0, recipient=43]-NetconfClientSessionImpl[netconf@/10.100.18.43:830][netconf]) SSH_MSG_CHANNEL_CLOSE 2022-10-14T21:20:40,840 | DEBUG | sshd-NetconfSshClient[a8b2d7e]-nio2-thread-6 | NettyAwareChannelSubsystem | 265 - org.opendaylight.netconf.shaded-sshd - 4.0.3.SNAPSHOT | handleClose(NettyAwareChannelSubsystem[id=0, recipient=43]-NetconfClientSessionImpl[netconf@/10.100.18.43:830][netconf]) prevent sending EOF 2022-10-14T21:20:40,840 | DEBUG | sshd-NetconfSshClient[a8b2d7e]-nio2-thread-6 | NettyAwareChannelSubsystem | 265 - org.opendaylight.netconf.shaded-sshd - 4.0.3.SNAPSHOT | close(NettyAwareChannelSubsystem[id=0, recipient=43]-NetconfClientSessionImpl[netconf@/10.100.18.43:830][netconf]) Closing gracefully 2022-10-14T21:20:40,841 | DEBUG | sshd-NetconfSshClient[a8b2d7e]-nio2-thread-6 | Window | 265 - org.opendaylight.netconf.shaded-sshd - 4.0.3.SNAPSHOT | Closing Window[client/local](NettyAwareChannelSubsystem[id=0, recipient=43]-NetconfClientSessionImpl[netconf@/10.100.18.43:830][netconf]) 2022-10-14T21:20:40,841 | DEBUG | sshd-NetconfSshClient[a8b2d7e]-nio2-thread-6 | Window | 265 - org.opendaylight.netconf.shaded-sshd - 4.0.3.SNAPSHOT | Closing Window[client/remote](NettyAwareChannelSubsystem[id=0, recipient=43]-NetconfClientSessionImpl[netconf@/10.100.18.43:830][netconf]) 2022-10-14T21:20:40,842 | DEBUG | sshd-NetconfSshClient[a8b2d7e]-nio2-thread-6 | ChannelSession$1 | 265 - org.opendaylight.netconf.shaded-sshd - 4.0.3.SNAPSHOT | close([NettyAwareChannelSubsystem[id=0, recipient=43]-NetconfClientSessionImpl[netconf@/10.100.18.43:830][netconf]] cmd=SSH_MSG_CHANNEL_DATA) Closing gracefully 2022-10-14T21:20:40,842 | DEBUG | sshd-NetconfSshClient[a8b2d7e]-nio2-thread-6 | NettyAwareChannelSubsystem | 265 - org.opendaylight.netconf.shaded-sshd - 4.0.3.SNAPSHOT | sendEof(NettyAwareChannelSubsystem[id=0, recipient=43]-NetconfClientSessionImpl[netconf@/10.100.18.43:830][netconf]) already sent (state=Graceful) 2022-10-14T21:20:40,842 | DEBUG | sshd-NetconfSshClient[a8b2d7e]-nio2-thread-6 | ChannelSession$1 | 265 - org.opendaylight.netconf.shaded-sshd - 4.0.3.SNAPSHOT | close([NettyAwareChannelSubsystem[id=0, recipient=43]-NetconfClientSessionImpl[netconf@/10.100.18.43:830][netconf]] cmd=SSH_MSG_CHANNEL_DATA][Graceful] - operationComplete() closed 2022-10-14T21:20:40,842 | DEBUG | sshd-NetconfSshClient[a8b2d7e]-nio2-thread-6 | ChannelAsyncInputStream | 265 - org.opendaylight.netconf.shaded-sshd - 4.0.3.SNAPSHOT | close(ChannelAsyncInputStream[NettyAwareChannelSubsystem[id=0, recipient=43]-NetconfClientSessionImpl[netconf@/10.100.18.43:830][netconf]]) Closing gracefully 2022-10-14T21:20:44,837 | DEBUG | sshd-NetconfSshClient[a8b2d7e]-nio2-thread-6 | ChannelAsyncInputStream | 265 - org.opendaylight.netconf.shaded-sshd - 4.0.3.SNAPSHOT | close(ChannelAsyncInputStream[NettyAwareChannelSubsystem[id=0, recipient=43]-NetconfClientSessionImpl[netconf@/10.100.18.43:830][netconf]]][Graceful] - operationComplete() closed 2022-10-14T21:20:44,838 | DEBUG | sshd-NetconfSshClient[a8b2d7e]-nio2-thread-6 | ChannelAsyncInputStream | 265 - org.opendaylight.netconf.shaded-sshd - 4.0.3.SNAPSHOT | close(ChannelAsyncInputStream[NettyAwareChannelSubsystem[id=0, recipient=43]-NetconfClientSessionImpl[netconf@/10.100.18.43:830][netconf]]) Closing gracefully 2022-10-14T21:20:44,838 | DEBUG | sshd-NetconfSshClient[a8b2d7e]-nio2-thread-6 | ChannelAsyncInputStream | 265 - org.opendaylight.netconf.shaded-sshd - 4.0.3.SNAPSHOT | close(ChannelAsyncInputStream[NettyAwareChannelSubsystem[id=0, recipient=43]-NetconfClientSessionImpl[netconf@/10.100.18.43:830][netconf]]][Graceful] - operationComplete() closed </pre> </div></div> <p>Okay so we have some inter-thread handover:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2022-10-14T21:19:38,167 | DEBUG | sshd-NetconfSshClient[a8b2d7e]-nio2-thread-2 | AsyncSshHandler                  | 258 - org.opendaylight.netconf.netty-util - 4.0.3.SNAPSHOT | SSH session authenticated on channel: [id: 0x1d4d427c], server version: SSH-2.0-libssh_0.9.5 [...] 2022-10-14T21:20:14,013 | DEBUG | sshd-NetconfSshClient[a8b2d7e]-nio2-thread-5 | NettyAwareChannelSubsystem       | 265 - org.opendaylight.netconf.shaded-sshd - 4.0.3.SNAPSHOT | handleData(NettyAwareChannelSubsystem[id=0, recipient=43]- NetconfClientSessionImpl[netconf@/10.100.18.43:830][netconf]) SSH_MSG_CHANNEL_DATA len=6 2022-10-14T21:20:14,016 | DEBUG | globalWorkerGroup-3-5 | AbstractNetconfSessionNegotiator | 258 - org.opendaylight.netconf.netty-util - 4.0.3.SNAPSHOT | Starting session negotiation on channel [id: 0x1d4d427c] </pre> </div></div> <p>and then:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2022-10-14T21:20:46,512 | DEBUG | sshd-NetconfSshClient[a8b2d7e]-nio2-thread-6 | AsyncSshHandler                  | 258 - org.opendaylight.netconf.netty-util - 4.0.3.SNAPSHOT | SSH session closed on channel: [id: 0x1d4d427c] [...] 2022-10-14T21:20:51,595 | DEBUG | globalWorkerGroup-3-5 | AbstractNetconfSessionNegotiator | 258 - org.opendaylight.netconf.netty-util - 4.0.3.SNAPSHOT | Session negotiation started with hello message &lt;hello xmlns="urn:ietf:params:xml:ns:netconf:base:1.0"&gt;    &lt;capabilities&gt;        &lt;capability&gt;urn:ietf:params:netconf:capability:exi:1.0&lt;/capability&gt;        &lt;capability&gt;urn:ietf:params:netconf:base:1.1&lt;/capability&gt;        &lt;capability&gt;urn:ietf:params:netconf:base:1.0&lt;/capability&gt;    &lt;/capabilities&gt; &lt;/hello&gt; on channel [id: 0x1d4d427c] [...] 2022-10-14T21:21:10,178 | INFO  | globalWorkerGroup-3-5 | AbstractNetconfSessionNegotiator | 258 - org.opendaylight.netconf.netty-util - 4.0.3.SNAPSHOT | Failed to send message &lt;hello xmlns="urn:ietf:params:xml:ns:netconf:base:1.0"&gt;    &lt;capabilities&gt;        &lt;capability&gt;urn:ietf:params:netconf:capability:exi:1.0&lt;/capability&gt;        &lt;capability&gt;urn:ietf:params:netconf:base:1.1&lt;/capability&gt;        &lt;capability&gt;urn:ietf:params:netconf:base:1.0&lt;/capability&gt;    &lt;/capabilities&gt; &lt;/hello&gt; on channel [id: 0x1d4d427c] java.lang.IllegalStateException: Channel closed        at org.opendaylight.netconf.nettyutil.handler.ssh.client.AsyncSshHandlerWriter.write(AsyncSshHandlerWriter.java:59) ~[bundleFile:?]</pre> </div></div> <p>note how we have things executing on sshd-nio2 threads and on globalWorkerGroup threads.</p> <p>Also, there is a 30 second delay in negotiation start – but note also that "Session negotiation started" is triggered after the channel is closed – pointing towards a problem in state checking. I think we need to handle the case where channelActive is triggered and there is also a channelInactive is staged in the executor,</p> <p>The interesting thing is the delay of <font color="#000000">37,579 </font>seconds between "starting" and "started" messages on the global worker thread. It is during this delay that the session goes down. This seems to indicate that AbstractNetconfSessionNegotiator.start() needs to check whether the channel is still open before embarking on sending the message.</p> <p>We also need to figure out how AsyncSshHandler.safelyDisconnect() -&gt; AsyncSshHandlerWriter.close() -&gt; channel.close() is synchronized w.r.t. the the negotiator.</p> <p>Rather than mucking around with lifecycle and trying to synchronize things, the fix is to run the two methods interacting with channel lifecycle on the corresponding executor – thus providing natural sequencing.</p> <p>karaf-netconf-905.log: karaf log for <a href="https://git.opendaylight.org/gerrit/c/netconf/+/102740/5" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/c/netconf/+/102740/</a></p> <p>So here we are catching this transition:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>java.lang.IllegalStateException: Channel closed at org.opendaylight.netconf.nettyutil.handler.ssh.client.AsyncSshHandlerWriter.write(AsyncSshHandlerWriter.java:59) ~[bundleFile:?] at org.opendaylight.netconf.nettyutil.handler.ssh.client.AsyncSshHandler.write(AsyncSshHandler.java:186) ~[bundleFile:?] at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717) ~[bundleFile:4.1.76.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:709) ~[bundleFile:4.1.76.Final] at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:792) ~[bundleFile:4.1.76.Final] at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:702) ~[bundleFile:4.1.76.Final] at io.netty.handler.codec.MessageToByteEncoder.write(MessageToByteEncoder.java:113) ~[bundleFile:4.1.76.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717) ~[bundleFile:4.1.76.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:709) ~[bundleFile:4.1.76.Final] at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:792) ~[bundleFile:4.1.76.Final] at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:702) ~[bundleFile:4.1.76.Final] at io.netty.handler.codec.MessageToByteEncoder.write(MessageToByteEncoder.java:113) ~[bundleFile:4.1.76.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717) ~[bundleFile:4.1.76.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:764) ~[bundleFile:4.1.76.Final] at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:790) ~[bundleFile:4.1.76.Final] at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:758) ~[bundleFile:4.1.76.Final] at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:808) ~[bundleFile:4.1.76.Final] at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1025) ~[bundleFile:4.1.76.Final] at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:306) ~[bundleFile:4.1.76.Final] at org.opendaylight.netconf.nettyutil.AbstractNetconfSessionNegotiator.sendMessage(AbstractNetconfSessionNegotiator.java:340) ~[bundleFile:?] at org.opendaylight.netconf.nettyutil.AbstractNetconfSessionNegotiator.start(AbstractNetconfSessionNegotiator.java:150) ~[bundleFile:?] at org.opendaylight.netconf.nettyutil.AbstractNetconfSessionNegotiator.startNegotiation(AbstractNetconfSessionNegotiator.java:131) ~[bundleFile:?] at org.opendaylight.netconf.nettyutil.AbstractNetconfSessionNegotiator.channelActive(AbstractNetconfSessionNegotiator.java:355) ~[bundleFile:?]</pre> </div></div> <p>and also this:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>java.io.EOFException: Channel is being closed at org.opendaylight.netconf.shaded.sshd.common.channel.AbstractChannel$2.&lt;init&gt;(AbstractChannel.java:782) ~[bundleFile:?] at org.opendaylight.netconf.shaded.sshd.common.channel.AbstractChannel.writePacket(AbstractChannel.java:780) ~[bundleFile:?] at org.opendaylight.netconf.shaded.sshd.common.channel.throttle.DefaultChannelStreamWriter.writeData(DefaultChannelStreamWriter.java:46) ~[bundleFile:?] at org.opendaylight.netconf.shaded.sshd.common.channel.ChannelAsyncOutputStream.doWriteIfPossible(ChannelAsyncOutputStream.java:209) ~[bundleFile:?] at org.opendaylight.netconf.shaded.sshd.common.channel.ChannelAsyncOutputStream.writeBuffer(ChannelAsyncOutputStream.java:106) ~[bundleFile:?] at org.opendaylight.netconf.nettyutil.handler.ssh.client.AsyncSshHandlerWriter.writeWithPendingDetection(AsyncSshHandlerWriter.java:95) ~[bundleFile:?] at org.opendaylight.netconf.nettyutil.handler.ssh.client.AsyncSshHandlerWriter.write(AsyncSshHandlerWriter.java:78) ~[bundleFile:?] at org.opendaylight.netconf.nettyutil.handler.ssh.client.AsyncSshHandler.write(AsyncSshHandler.java:125) ~[bundleFile:?] at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717) ~[bundleFile:4.1.76.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:709) ~[bundleFile:4.1.76.Final] at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:792) ~[bundleFile:4.1.76.Final] at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:702) ~[bundleFile:4.1.76.Final] at io.netty.handler.codec.MessageToByteEncoder.write(MessageToByteEncoder.java:113) ~[bundleFile:4.1.76.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717) ~[bundleFile:4.1.76.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:709) ~[bundleFile:4.1.76.Final] at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:792) ~[bundleFile:4.1.76.Final] at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:702) ~[bundleFile:4.1.76.Final] at io.netty.handler.codec.MessageToByteEncoder.write(MessageToByteEncoder.java:113) ~[bundleFile:4.1.76.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:717) ~[bundleFile:4.1.76.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:764) ~[bundleFile:4.1.76.Final] at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:790) ~[bundleFile:4.1.76.Final] at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:758) ~[bundleFile:4.1.76.Final] at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:808) ~[bundleFile:4.1.76.Final] at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1025) ~[bundleFile:4.1.76.Final] at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:306) ~[bundleFile:4.1.76.Final] at org.opendaylight.netconf.nettyutil.AbstractNetconfSessionNegotiator.sendMessage(AbstractNetconfSessionNegotiator.java:340) ~[bundleFile:?] at org.opendaylight.netconf.nettyutil.AbstractNetconfSessionNegotiator.start(AbstractNetconfSessionNegotiator.java:150) ~[bundleFile:?] at org.opendaylight.netconf.nettyutil.AbstractNetconfSessionNegotiator.startNegotiation(AbstractNetconfSessionNegotiator.java:131) ~[bundleFile:?] at org.opendaylight.netconf.nettyutil.AbstractNetconfSessionNegotiator.channelActive(AbstractNetconfSessionNegotiator.java:355) ~[bundleFile:?]</pre> </div></div> <p>Both of these are a failure to send initial hello message. Since these are in the Netty-&gt;SSHD direction, there simply are SSHD transitions we have not observed yet (as they are running concurrently with Netty) – a closed or closing channel may be one of those.</p> <p>If this ever happens it is equivalent to the session never being established, as the failure occurred when attempting IDLE-&gt;OPEN_WAIT transition and hence the transition becomes IDLE-&gt;FAILED instead.</p> <ul> <li>karaf-netconf-905-102751-4.log: Karaf log for <a href="https://git.opendaylight.org/gerrit/c/netconf/+/102751/4" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/c/netconf/+/102751/4</a></li> <li>karaf-netconf-905-102751-4-mod.log: Karaf log after making a change to allow state transition from <tt>IDLE</tt> to <tt>FAILED</tt>:</li> </ul> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>--- a/netconf/netconf-netty-util/src/main/java/org/opendaylight/netconf/nettyutil/AbstractNetconfSessionNegotiator.java +++ b/netconf/netconf-netty-util/src/main/java/org/opendaylight/netconf/nettyutil/AbstractNetconfSessionNegotiator.java @@ -327,6 +327,9 @@ public abstract class AbstractNetconfSessionNegotiator&lt;S extends AbstractNetconf if (state == State.OPEN_WAIT &amp;&amp; newState == State.ESTABLISHED) { return true; } + if (state == State.IDLE &amp;&amp; newState == State.FAILED) { + return true; + } if (state == State.OPEN_WAIT &amp;&amp; newState == State.FAILED) { return true; } </pre> </div></div> Issue split NETCONF-827 Development Rank 0|i04353: