https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/NETCONF-830 netconf <p>We use ODL netconf topology. It failed to establish connection to Netconf server because of folowing error.<br/>  </p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>AbstractNetconfSessionNegotiator - Failed to send message &lt;hello xmlns="urn:ietf:params:xml:ns:netconf:base:1.0"&gt;    &lt;capabilities&gt;        &lt;capability&gt;urn:ietf:params:netconf:capability:exi:1.0&lt;/capability&gt;        &lt;capability&gt;urn:ietf:params:netconf:base:1.1&lt;/capability&gt;        &lt;capability&gt;urn:ietf:params:netconf:base:1.0&lt;/capability&gt;    &lt;/capabilities&gt; &lt;/hello&gt; java.lang.IllegalStateException: Channel closed at org.opendaylight.netconf.nettyutil.handler.ssh.client.AsyncSshHandlerWriter.write(AsyncSshHandlerWriter.java:64) ~[netconf-netty-util-1.13.4.jar:?] at org.opendaylight.netconf.nettyutil.handler.ssh.client.AsyncSshHandler.write(AsyncSshHandler.java:188) ~[netconf-netty-util-1.13.4.jar:?] </pre> </div></div> <p>Then logs were flooded with exceptions:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[WARN ] 2021-11-09 17:09:36.562 [nioEventLoopGroup-3-1] AbstractChannelHandlerContext - An exception 'java.lang.IllegalStateException: complete already: NetconfSessionPromise@546106c5(failure: java.lang.IllegalStateException: Channel closed)' [enable DEBUG level for full   stacktrace] was thrown by a user handler's exceptionCaught() method while handling the following exception: java.lang.IllegalStateException: complete already: NetconfSessionPromise@546106c5(failure: java.lang.IllegalStateException: Channel closed)        at io.netty.util.concurrent.DefaultPromise.setFailure(DefaultPromise.java:112) ~[netty-common-4.1.65.Final.jar:4.1.65.Final]        at org.opendaylight.netconf.nettyutil.AbstractNetconfSessionNegotiator.negotiationFailed(AbstractNetconfSessionNegotiator.java:290) ~[netconf-netty-util-1.13.4.jar:?]        at org.opendaylight.netconf.nettyutil.AbstractNetconfSessionNegotiator.channelRead(AbstractNetconfSessionNegotiator.java:330) ~[netconf-netty-util-1.13.4.jar:?]        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.65.Final.jar:4.1.65.Final]        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.65.Final.jar:4.1.65.Final]        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.65.Final.jar:4.1.65.Final]        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) [netty-codec-4.1.65.Final.jar:4.1.65.Final]        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) [netty-codec-4.1.65.Final.jar:4.1.65.Final]        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.65.Final.jar:4.1.65.Final]        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.65.Final.jar:4.1.65.Final]        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.65.Final.jar:4.1.65.Final]        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) [netty-codec-4.1.65.Final.jar:4.1.65.Final]        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) [netty-codec-4.1.65.Final.jar:4.1.65.Final]        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.65.Final.jar:4.1.65.Final]        at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:61) [netty-transport-4.1.65.Final.jar:4.1.65.Final]        at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:370) [netty-transport-4.1.65.Final.jar:4.1.65.Final]        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) [netty-common-4.1.65.Final.jar:4.1.65.Final]        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) [netty-common-4.1.65.Final.jar:4.1.65.Final]        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) [netty-transport-4.1.65.Final.jar:4.1.65.Final]        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.65.Final.jar:4.1.65.Final]        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.65.Final.jar:4.1.65.Final]        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.65.Final.jar:4.1.65.Final]        at java.lang.Thread.run(Thread.java:829) [?:?] Caused by: java.lang.IllegalStateException: complete already: NetconfSessionPromise@546106c5(failure: java.lang.IllegalStateException: Channel closed)        at io.netty.util.concurrent.DefaultPromise.setSuccess(DefaultPromise.java:99) ~[netty-common-4.1.65.Final.jar:4.1.65.Final]        at org.opendaylight.netconf.nettyutil.NetconfSessionPromise.setSuccess(NetconfSessionPromise.java:79) ~[netconf-netty-util-1.13.4.jar:?]        at org.opendaylight.netconf.nettyutil.NetconfSessionPromise.setSuccess(NetconfSessionPromise.java:27) ~[netconf-netty-util-1.13.4.jar:?]        at org.opendaylight.netconf.nettyutil.AbstractNetconfSessionNegotiator.negotiationSuccessful(AbstractNetconfSessionNegotiator.java:284) ~[netconf-netty-util-1.13.4.jar:?]        at org.opendaylight.netconf.client.NetconfClientSessionNegotiator.handleMessage(NetconfClientSessionNegotiator.java:93) ~[netconf-client-1.13.4.jar:?]        at org.opendaylight.netconf.nettyutil.AbstractNetconfSessionNegotiator.channelRead(AbstractNetconfSessionNegotiator.java:327) ~[netconf-netty-util-1.13.4.jar:?]        ... 20 more Nov 09, 2021 5:09:36 PM com.google.common.util.concurrent.AbstractFuture executeListener SEVERE: RuntimeException while executing runnable CallbackListener{org.opendaylight.netconf.sal.connect.netconf.NetconfDevice$1@7230cd8c} with executor MoreExecutors.directExecutor() java.lang.IllegalStateException: Already initialized        at com.google.common.base.Preconditions.checkState(Preconditions.java:508)        at org.opendaylight.netconf.sal.connect.netconf.sal.NetconfDeviceSalProvider$MountInstance.onTopologyDeviceConnected(NetconfDeviceSalProvider.java:127)        at org.opendaylight.netconf.sal.connect.netconf.sal.NetconfDeviceSalFacade.onDeviceConnected(NetconfDeviceSalFacade.java:87)        at org.opendaylight.netconf.sal.connect.netconf.sal.NetconfDeviceSalFacade.onDeviceConnected(NetconfDeviceSalFacade.java:43)        at org.opendaylight.netconf.sal.connect.netconf.NetconfDevice.handleSalInitializationSuccess(NetconfDevice.java:262)        at org.opendaylight.netconf.sal.connect.netconf.NetconfDevice$1.onSuccess(NetconfDevice.java:184)        at org.opendaylight.netconf.sal.connect.netconf.NetconfDevice$1.onSuccess(NetconfDevice.java:181)        at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1089)        at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)        at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1174)        at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:969)        at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:800)        at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.setResult(AbstractTransformFuture.java:224)        at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.setResult(AbstractTransformFuture.java:202)        at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:163)        at com.google.common.util.concurrent.MoreExecutors$5$1.run(MoreExecutors.java:1091)        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)        at java.base/java.lang.Thread.run(Thread.java:829) </pre> </div></div> <p>It wasn't possible to make netconf connect again with deleting and creating mount point again.</p> <p>We checked, that netconf server is running and we were able to send netconf RPCs via ssh.</p> NETCONF-830

Flood of "IllegalStateException: complete already"

Bug Medium Resolved Duplicate Ivan Hrasko Andrej Mak pt Wed, 10 Nov 2021 10:19:17 +0000 Thu, 18 Nov 2021 11:43:25 +0000 Thu, 18 Nov 2021 11:42:58 +0000 1.13.4 2.0.4 1.13.6 netconf 0 4 <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=andmak" class="user-hover" rel="andmak">andmak</a> Are there steps how to reproduce the issue?</p> <p>There are no simple steps. Issue happens only on some of tens of devices.</p> <p>I think it is related to reconnecting - if something goes wrong, netconf tries to reconnect, but fails to update netconf mountpoint session state(java.lang.IllegalStateException: Already initialized).</p> <p>Are you using clustered NETCONF topology with 3-node cluster?</p> <p>Is it possible that your leader node has changed during the process?</p> <p>You write: It wasn't possible to make netconf connect again with deleting and creating mount point again.</p> <p>Have you also checked OPERATIONAL data?</p> <p>When you delete device from CONFIG data is it deleted from OPERATIONAL as well?</p> <p>When you create device in OPERATIONAL is it created in OPERATIONAL as well?</p> <p><a href="https://jira.opendaylight.org/browse/NETCONF-784" title="ReconnectPromise keep reconnecting after device unregistered" class="issue-link" data-issue-key="NETCONF-784"><del>NETCONF-784</del></a> might be <a href="https://github.com/opendaylight/netconf/commit/86ea92e3769dc513b4c488726d318d939434eed7#diff-1742b8a062a6c3d8e746addd6d1a348b7667e8fddc15efeabde1bf917ca467b0R172" class="external-link" target="_blank" rel="nofollow noopener">the same thing</a>.</p> <p> <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=andmak" class="user-hover" rel="andmak">andmak</a> can you repro with netconf-2.0.8? </p> <p>We don't use cluster.</p> <p>No, I haven't checked operational data. Reconnect just deletes config and then waits until MountPoint listener says, that mountpoint was removed. Then it creates same neconf node in config datastore again.</p> <p>I see, that fix version in linked issue is 2.0.4. Would testing with 2.0.5 be enough?</p> <p>The latest, the better - but probably yes <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/biggrin.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>I looks you are not the only one complaining on "complete already": <a href="https://git.opendaylight.org/gerrit/c/netconf/+/98273" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/c/netconf/+/98273</a></p> <p>OK, this patch is misleading.</p> <p>It is difficult to reproduce this issue but we can see that the problem with the first connection was on line:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> AsyncSshHandlerWriter.java:64</pre> </div></div> <p>It means the following code: </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> promise.setFailure(<span class="code-keyword">new</span> IllegalStateException(<span class="code-quote">"Channel closed"</span>));</pre> </div></div> <p>Thus it is possible to simulate the issue by injecting a code that will invoke the same code as above when we are trying to establish connection to the device for the first time.</p> <p>Really, it was possible to reproduce the issue with the version 1.13.4, please see attached: <span class="nobr"><a href="https://jira.opendaylight.org/secure/attachment/17005/17005_1.13.4-reproduced.karaf.log" title="1.13.4-reproduced.karaf.log attached to NETCONF-830">1.13.4-reproduced.karaf.log^{<img class="rendericon" src="https://jira.opendaylight.org/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span></p> <p>Steps to reproduce are:</p> <ol> <li>start the device</li> <li>start controller (containing injected code) with installed netconf topology and restconf features</li> <li>create device node in netconf topology</li> <li>device is unable to connect (or sometimes successfully connected)</li> <li>delete device from netconf topology</li> <li>create device node in netconf topology</li> <li>device is in connecting state on forever now</li> </ol> <ul> <li>sometimes we need to repeat steps 5 and 6 two times</li> </ul> <p>With the current master branch - version 2.0.10-SNAPSHOT it is possible to observe behavior as we expect:</p> <ol> <li>device is unable to connect - because we injected the code to make first attempt to fail</li> <li>after deleting device node from topology and creating it again the device is connected successfully</li> </ol> <p>For example the version 2.0.4 is working, too.</p> <p>The cherry-picked code available <a href="https://git.opendaylight.org/gerrit/c/netconf/+/98573" class="external-link" target="_blank" rel="nofollow noopener">here</a> for <a href="https://jira.opendaylight.org/browse/NETCONF-784" title="ReconnectPromise keep reconnecting after device unregistered" class="issue-link" data-issue-key="NETCONF-784"><del>NETCONF-784</del></a> alone does not solve the problem on the Silicon branch (netconf 1.13.x branch).</p> <p>When the code available <a href="https://git.opendaylight.org/gerrit/c/netconf/+/98605" class="external-link" target="_blank" rel="nofollow noopener">here</a> is applied the problem is fixed. It helps even when it is applied alone without any other cherry-pick on 1.13.4 version.</p> <p>It means that this issue can be fixed in the 1.13.6 version in Silicon SR4 and it is already fixed in the 2.0.4 version in Phosphorus-SR0.</p> <p>.</p> Duplicate NETCONF-784 Development Rank 0|i040lj: