https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/NETCONF-880 netconf <p>We have an ODL-based app which manages netconf devices in a DC.<br/> Recently, we've been experiencing netconf connectivity/sync issues between ODL app and netconf devices in some of our prod environments.<br/> So here's what has been happening:</p> <p>1. We start the ODL app which connects to devices and fetches data from them.<br/>     All is good, devices are connected properly and are in sync (data was fetched correctly).<br/> 2. After a short time, ODL lose connection to some devices.<br/> 3. ODL netconf attempts automatic reconnect and the device gets reconnected.<br/> 4. ODL app tries to load data from device again using a netconf get RPC call.<br/> 5. ODL netconf (not the device) returns "transport error" and ODL app now marks the        device with sync-failed status. See the attachment "netconf-rpc-transport-error.txt"<br/> 6. User then calls ODL app RPC for reconnecting the device.<br/>    This RPC first deletes the node from ODL netconf-topology and then recreates it.<br/>    We added a 5 seconds delay (for investigation purposes only) between the topology  node deletion and recreation to be sure ODL netconf has enough time to dismount and mount the device again.<br/>    We added a check if the ODL netconf mountpoint exists (using DOMMountPointService) after the topology node was deleted and before it's going to be recreated again.<br/> 7. ODL app asks ODL netconf to close the netconf session.<br/>    Then we see the ODL netconf logs as they are in the attachment "netconf-logs-after-device-reconnect-rpc-call.txt".<br/> 8. ODL app asks ODL netconf to start the netconf session again, but it seems that the previous session is stuck.<br/>    Our check if the mountpoint still exists, throws an exception because it still exists, even though it should not at this point.<br/>    Devops guys from customer also reported lots of hanging netconf session between the device and ODL.<br/> 9. Afterwards we see lots of logs "netconf session established" followed by "netconf session promise complete already" errors in short intervals (few secs).<br/>    See the logs in the attachment "netconf-session-promise-complete-already-error.txt"</p> NETCONF-880

Netconf does not close device mountpoint properly

Bug Medium Resolved Duplicate Robert Varga Igor Foltin Fri, 27 May 2022 14:08:24 +0000 Tue, 31 May 2022 19:28:50 +0000 Tue, 31 May 2022 19:27:53 +0000 2.0.14 netconf 0 5 <p>I added more logs:<br/> <span class="nobr"><a href="https://jira.opendaylight.org/secure/attachment/17718/17718_reproducer-debug-logs.txt" title="reproducer-debug-logs.txt attached to NETCONF-880">reproducer-debug-logs.txt^{<img class="rendericon" src="https://jira.opendaylight.org/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span></p> <p>They represent a scenario when a user starts the device reconnection via ODL app RPC (mountpoint removal and recreation).</p> <p>The device name is: DCS:DCS:DCS-N9K-LEAF04<br/> I tried to remove all parts of the logs which were related to other devices present in the DC, so hopefully there should not be any irrelevant logs there.</p> <p>Are you able to reproduce with netconf 3?</p> <p>So if I am reading this right, the session goes up after the node has already been deleted:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[DEBUG] 2022-05-30 09:27:50.266 [opendaylight-cluster-data-notification-dispatcher-44] AbstractNetconfTopology - Disconnecting RemoteDevice{DCS:DCS:DCS-N9K-LEAF04} [TRACE] 2022-05-30 09:27:50.267 [opendaylight-cluster-data-notification-dispatcher-44] NetconfDeviceSalProvider - RemoteDevice{DCS:DCS:DCS-N9K-LEAF04}: Not removing TOPOLOGY mountpoint from MD-SAL, mountpoint was not registered yet [...] [TRACE] 2022-05-30 09:27:50.269 [opendaylight-cluster-data-notification-dispatcher-44] NetconfDeviceSalProvider - RemoteDevice{DCS:DCS:DCS-N9K-LEAF04}: TransactionChain(org.opendaylight.mdsal.binding.dom.adapter.BindingDOMTransactionChainAdapter@7b6c02b7) SUCCESSFUL [DEBUG] 2022-05-30 09:27:51.714 [nioEventLoopGroup-3-1] NetconfDeviceCommunicator - RemoteDevice{DCS:DCS:DCS-N9K-LEAF04}: Session established </pre> </div></div> <p>I think the problem lies somewhere between NetconfSessionPromise's retry logic and AsyncSshHandler. Most notably the former does not check for isFailure() and the latter calls explicit setFailure(). The setFailure() is problematic because it renders the Future.cancel() inoperative and hence reconnection attempts are not stopped when the node is deleted.</p> <p>We need debug logs for netconf-netty-util including initial connection setup to ascertain what went down.</p> <p>The recorded failure in NetconfSessionPromise is the same as the one reported in <a href="https://jira.opendaylight.org/browse/NETCONF-827" title="Mount loop when setting too low connection-timeout" class="issue-link" data-issue-key="NETCONF-827"><del>NETCONF-827</del></a>, hence I believe these are duplicates.</p> Duplicate NETCONF-827 Development Rank 0|i042hj: