lenapeuk What is the NETCONF version you are using? Are you able to reproduce this issue with netopeer/netopeer2? Can you reproduce the issue when device is connected "manually" by adding its configuration into netconf topology or the problem appears only when using call-home feature?
ivanhrasko: Thanks for you response. Actually the NetworkFunction we are working with is not using netopeer instead confd is used. The issue / error that the global requests are unsupported also occures if we mount the device manually, but basically the connection is not dropped but this is related to the implementation on the device side.
I am afraid it can be a problem on the device. I am running the following command with running ODL Aluminium SR3 (NETCONF version 1.9.3) controller:
ssh -vvv -p 2830 -o ServerAliveInterval=2 -o ServerAliveCountMax=3 admin@localhost -s netconf
And I am getting the response as follows:
debug3: send packet: type 80 debug3: receive packet: type 81 debug3: send packet: type 80 debug3: receive packet: type 81 debug3: send packet: type 80 debug3: receive packet: type 81 debug3: send packet: type 80 debug3: receive packet: type 81 debug3: send packet: type 80 debug3: receive packet: type 81
If my assumptions are correct then 80 stands for SSH_MSG_GLOBAL_REQUEST and 81 stands for SSH_MSG_REQUEST_SUCCESS. Thus I think that ODL controller works as expected.
ivanhrasko Thanks for the info. I will also check this on our side and analyze in more details 
Thanks, it would really help if you are able to check the versions you're using.
https://datatracker.ietf.org/doc/html/rfc4254#section-9 :
SSH_MSG_GLOBAL_REQUEST 80
SSH_MSG_REQUEST_SUCCESS 81
SSH_MSG_REQUEST_FAILURE 82
SSH_MSG_CHANNEL_OPEN 90
SSH_MSG_CHANNEL_OPEN_CONFIRMATION 91
SSH_MSG_CHANNEL_OPEN_FAILURE 92
SSH_MSG_CHANNEL_WINDOW_ADJUST 93
SSH_MSG_CHANNEL_DATA 94
SSH_MSG_CHANNEL_EXTENDED_DATA 95
SSH_MSG_CHANNEL_EOF 96
SSH_MSG_CHANNEL_CLOSE 97
SSH_MSG_CHANNEL_REQUEST 98
SSH_MSG_CHANNEL_SUCCESS 99
SSH_MSG_CHANNEL_FAILURE 100
ivanhrasko It is version 1.0
So NETCONF version is 1.0.0-Beryllium? Are you still using ODL Beryllium?
Is that the version of the VNF or the version of OpenDaylight stack (or ONAP SDNC, or what is it?)
This is the version of the NF-device. We actually use ONAP SDNC istanbul release with ODL silicon-version.
I have checked and the command:
ssh -vvv -p 2830 -o ServerAliveInterval=2 -o ServerAliveCountMax=3 admin@localhost -s netconf
works as expected also with ODL Silicon SR2.
lenapeuk Sure, some configuration has to be done on both the controller and the device to setup the call-home correctly. For example the controller needs to have device's SSH public key and device needs to have call-home feature enabled as well. Do you remember the steps how you configured the both? Which requests have you send to ODL?
POST /restconf/config/odl-netconf-callhome-server:netconf-callhome-server/allowed-devices
{
"device": {
"unique-id": "ssh-device-to-discover",
"ssh-client-params": {
"host-key": "device's public key here"
}
}
}
Hi ivanhrasko, thanks for the info. We have exactly used these Endpoint to add the the device to the allowed device by passing the required parameters (SSH-public-key, id). Basically this works fine as expected, so we can see that hello-capabilities are exchanged, yand-models are loaded and that the device is mounted under the specified unique-id. But after a few minutes / seconds we run into this exception, that the ssh-session is closed.
Actually we try to validate if it is possible to reproduce it via ssh / other netconf-client.
lenapeuk I have setup SSH Call-Home session with Netopeer2 device and I have observed the logs like this:
2021-12-03T14:55:21,530 | DEBUG | nioEventLoopGroup-4-2 | AsyncSshHandlerReader | 281 - org.opendaylight.netconf.netty-util - 1.13.4 | Ssh session dropped on channel: netconf org.opendaylight.netconf.shaded.sshd.common.SshException: Closed at org.opendaylight.netconf.shaded.sshd.common.channel.ChannelAsyncInputStream.preClose(ChannelAsyncInputStream.java:97) ~[bundleFile:?] at org.opendaylight.netconf.shaded.sshd.common.util.closeable.AbstractCloseable.close(AbstractCloseable.java:94) ~[bundleFile:?] at org.opendaylight.netconf.shaded.sshd.common.util.closeable.ParallelCloseable.doClose(ParallelCloseable.java:65) ~[bundleFile:?] at org.opendaylight.netconf.shaded.sshd.common.util.closeable.SimpleCloseable.close(SimpleCloseable.java:63) ~[bundleFile:?]
To appear every 10 minutes. Every 10 minutes the connection is terminated and established new one on different port.
I expect you hit this idle timeout/disconnection and the fact that you see SSH global requests unsupported/failed is:
a) not related to the disconnection itself
b) it is a consequence of terminated connection.
Is this explanation applicable to your problem? Do you see anything else/different happening?
Other netconf clients work?
Its also possible that timeout appears as a consequence of not correctly implemented heart-beat as you said 
Anyway, I think that also unsupported response should be enough as heart-beat and it can be a problem of the device that it does not accept it.
Hi lenapeuk . I assume the CALL HOME is coming from devices behind NAT. Can you please check with your router admin to validate the default NAT entries timeout. It could be related to NAT entries being timed out which consequently terminates the TCP sessions for every 10 minutes also. I guess, we can consider increasing the NAT timeout and check if that helps here.
Most router settings will change NAT default timer to few seconds to few minutes. This will end up closing existing TCP connections behind NAT. It seems to be the case with your device for every 10 minutes also.
lenapeuk I have tried call home with ConfD Basic (for free) and I have observed that it sends no SSH_MSG_GLOBAL_REQUEST. Have you configured it to send it somehow?
In general, every device is different, nor Netopeer2 nor ConfD in default configuration sends SSH_MSG_GLOBAL_REQUEST.
I believe this is the reason why NETCONF solves the problem of keeping the connection to device alive by its own mechanism.
This mechanism was turned off for call home mounted devices - I have enabled it by the gerrit change: https://git.opendaylight.org/gerrit/c/netconf/+/98872
With gerrit change NETCONF sends keep-alive message to the call home device every two minutes.
IMO the SSH_MSG_REQUEST_FAILURE response is good enough according to Section 4 of RFC 4254. Its just not needed/supported - that's not reason for the device to drop connection.
When you apply the proposed fix you can turn off sending SSH_MSG_GLOBAL_REQUEST and connection stays alive.
If you anyway would to use SSH_MSG_GLOBAL_REQUEST instead of NETCONF own keep-alive mechanism - let me know - we need to discuss it - because it is a new feature then.
Yes, that is one of the reasons why we need some keep-alive - heartbeats mechanism to keep connection alive. The current 10 minutes in ODL are configured here: https://github.com/apache/mina-sshd/blob/2772c7c8f6afb8c53546ca803501f52118bd0491/sshd-core/src/main/java/org/apache/sshd/core/CoreModuleProperties.java#L319
ivanhrasko Thanks you very much for the update. From our understanding it is configured on Device-side. Actually we try to check the device in more details. With regards to Section 4 of RFC 4254, we have a simular understanding, so maybe there might be some issues on device side.
Hi ivanhrasko, after some detailed testing and some further investigations, we found out the issue seems not be related to ODL or confd and we had some deeper problems with network setup of the netconf-device. It took some time to identify this because e.g. with python netconf-client it works probably and we could only see this behaviour and the drop of the connection inside ODL. So with the fix on network side after the callhome was triggered, the connection remains stable (we still see the warnings related to the global-requests, but we see it just as a warning but nothing related to our inital problem). Thank you very much for you support and your investigations