[NETCONF-1195] Device Remain Connected after it get stopped Created: 07/Nov/23  Updated: 23/Jan/24

Status: Confirmed
Project: netconf
Component/s: netconf-topology
Affects Version/s: 6.0.4, 5.0.9
Fix Version/s: 6.0.7, 7.0.1

Type: Bug Priority: Medium
Reporter: Vlad Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Zip Archive test10.zip    
Issue Links:
Relates
relates to NETCONF-966 Long delay in connection status updat... Resolved

 Description   

Opendaylight latest docker images FEATURES=odl-restconf,odl-netconf-topology

 

Netconf Device remained connected even after I stopped the device. TCP connection is established for the IP 10.8.120.122

 

Started the controller in docker the latest version 0.18.2 is available on the docker hub

 

Once started I mounted 5000 devices and I noticed that some of the devices remain connected in network topology even if that device died for a while... I waited 24 hours to see if anything changed on the network topology status but it remained connected

 

I would like to understand how I can proceed to make sure that the ODL has the proper state.

Network topology
"node-id": "10.8.120.122:830",
                        "netconf-node-topology:sleep-factor": "1.1",
                        "netconf-node-topology:connection-timeout-millis": 20000,
                        "netconf-node-topology:port": 830,
                        "netconf-node-topology:tcp-only": false,
                        "netconf-node-topology:host": "10.8.120.122",
                        "netconf-node-topology:connection-status": "connected",
                        "netconf-node-topology:session-id": 4,
                        "netconf-node-topology:max-connection-attempts": 3,
                        "netconf-node-topology:username": "admin",
                        "netconf-node-topology:password": "admin",
                        "netconf-node-topology:between-attempts-timeout-millis": 2000,
                        "netconf-node-topology:reconnect-on-changed-schema": true,
                        "netconf-node-topology:keepalive-delay": 2000,
 

Logs here

++ dirname ./start_docker.sh

+ BASEDIR=.

+ sed -i 's/(featuresBoot= |featuresBoot = )/featuresBoot = odl-restconf-all,odl-netconf-clustered-topology,odl-mdsal-distributed-datastore,/g' ./etc/org.apache.karaf.features.cfg

+ cat ./etc/org.apache.karaf.features.cfg

+ ./bin/karaf run

Oct 11, 2023 6:52:16 AM org.apache.karaf.main.Main launch

INFO: Installing and starting initial bundles

Oct 11, 2023 6:52:16 AM org.apache.karaf.main.Main launch

INFO: All initial bundles installed and set to start

Oct 11, 2023 6:52:16 AM org.apache.karaf.main.lock.SimpleFileLock lock

INFO: Trying to lock /opt/opendaylight/lock

Oct 11, 2023 6:52:16 AM org.apache.karaf.main.lock.SimpleFileLock lock

INFO: Lock acquired

Oct 11, 2023 6:52:16 AM org.apache.karaf.main.Main$KarafLockCallback lockAcquired

INFO: Lock acquired. Setting startlevel to 100

2023-10-11T06:58:47,160 | INFO  | opendaylight-cluster-data-akka.actor.default-dispatcher-21 | NetconfTopologyContext           | 281 - org.opendaylight.netconf.topology-singleton - 5.0.6 | Master was selected: IpAddress{ipv4Address=Ipv4Address{value=10.8.120.122}}

2023-10-11T06:58:47,161 | INFO  | opendaylight-cluster-data-akka.actor.default-dispatcher-21 | RemoteDeviceConnectorImpl        | 281 - org.opendaylight.netconf.topology-singleton - 5.0.6 | RemoteDeviceId[name=10.8.120.122:830, address=/10.8.120.122:830]: Adding keepalive facade.

2023-10-11T06:58:47,162 | INFO  | opendaylight-cluster-data-akka.actor.default-dispatcher-21 | RemoteDeviceConnectorImpl        | 281 - org.opendaylight.netconf.topology-singleton - 5.0.6 | RemoteDeviceId[name=10.8.120.122:830, address=/10.8.120.122:830]: Concurrent rpc limit is smaller than 1, no limit will be enforced.

2023-10-11T06:58:47,200 | WARN  | sshd-NetconfSshClient[4adaf1d1]-nio2-thread-3 | AcceptAllServerKeyVerifier       | 279 - org.opendaylight.netconf.shaded-sshd - 5.0.6 | Server at /10.8.120.122:830 presented unverified EC key: SHA256:mieSv5hSk1LUum30BBwQ47khcl35jUQsFBKCL32mUF0

2023-10-11T06:58:48,070 | WARN  | remote-connector-processing-executor-11 | DeviceSourcesResolver            | 276 - org.opendaylight.netconf.sal-netconf-connector - 5.0.6 | RemoteDeviceId[name=10.8.120.122:830, address=/10.8.120.122:830]: Netconf device does not provide all yang models reported in hello message capabilities, required but not provided: [(urn:sysrepo:plugind?revision=2022-03-10)sysrepo-plugind]

2023-10-11T06:58:48,070 | WARN  | remote-connector-processing-executor-11 | DeviceSourcesResolver            | 276 - org.opendaylight.netconf.sal-netconf-connector - 5.0.6 | RemoteDeviceId[name=10.8.120.122:830, address=/10.8.120.122:830]: Attempting to build schema context from required sources

2023-10-11T06:58:48,070 | WARN  | remote-connector-processing-executor-11 | DeviceSourcesResolver            | 276 - org.opendaylight.netconf.sal-netconf-connector - 5.0.6 | RemoteDeviceId[name=10.8.120.122:830, address=/10.8.120.122:830]: Netconf device provides additional yang models not reported in hello message capabilities: [(urn:ieee:std:802.1Q:yang:ieee802-dot1q-types?revision=2022-05-19)ieee802-dot1q-types, (urn:ietf:params:xml:ns:yang:ietf-netconf-server?revision=2019-07-02)ietf-netconf-server, (urn:ietf:params:xml:ns:yang:ietf-tcp-common?revision=2019-07-02)ietf-tcp-common, (urn:ietf:params:xml:ns:yang:iana-hardware?revision=2018-03-13)iana-hardware, (urn:onf:yang:air-interface-2-0?revision=2022-07-29)air-interface-2-0, (urn:onf:params:xml:ns:yang:cera-radioprotection-profile?revision=2022-10-03)cera-radioprotection-profile, (urn:ieee:std:802.3:yang:ieee802-ethernet-interface?revision=2019-06-21)ieee802-ethernet-interface, (urn:ietf:params:xml:ns:yang:ietf-origin?revision=2018-02-14)ietf-origin, (urn:ietf:params:xml:ns:yang:ietf-restconf?revision=2017-01-26)ietf-restconf, (urn:onf:yang:core-model-1-4?revision=2019-11-27)core-model-1-4, (urn:ietf:params:xml:ns:yang:ietf-interface-protection?revision=2019-06-19)ietf-interface-protection, (urn:ietf:params:xml:ns:yang:ietf-ssh-common?revision=2019-07-02)ietf-ssh-common, (urn:ietf:params:xml:ns:yang:ietf-yang-library?revision=2019-01-04)ietf-yang-library, (urn:ietf:params:xml:ns:yang:ietf-microwave-radio-link?revision=2019-06-19)ietf-microwave-radio-link, (urn:ietf:params:xml:ns:yang:ietf-crypto-types?revision=2019-07-02)ietf-crypto-types, (urn:ietf:params:xml:ns:yang:ietf-datastores?revision=2018-02-14)ietf-datastores, (urn:ietf:params:xml:ns:yang:ietf-tls-server?revision=2019-07-02)ietf-tls-server, (urn:onf:yang:wire-interface-2-0?revision=2022-04-06)wire-interface-2-0, (urn:ietf:params:xml:ns:yang:ietf-microwave-types?revision=2019-06-19)ietf-microwave-types, (urn:ietf:params:xml:ns:yang:ietf-subscribed-notifications?revision=2019-09-09)ietf-subscribed-notifications, (urn:ietf:params:xml:ns:yang:ietf-hardware?revision=2018-03-13)ietf-hardware, (urn:onf:params:xml:ns:yang:cera-common?revision=2022-09-28)cera-common, (urn:ietf:params:xml:ns:yang:ietf-yang-patch?revision=2017-02-22)ietf-yang-patch, (urn:onf:yang:co-channel-profile-1-0?revision=2020-01-27)co-channel-profile-1-0, (urn:ieee:std:802.1Q:yang:ieee802-dot1ab-types?revision=2021-07-21)ieee802-dot1ab-types, (urn:ietf:params:xml:ns:yang:ietf-ssh-server?revision=2019-07-02)ietf-ssh-server, (urn:onf:params:xml:ns:yang:cera-linkbonding-profile?revision=2022-08-19)cera-linkbonding-profile, (urn:ieee:std:802.1Q:yang:ieee802-dot1q-bridge?revision=2022-05-19)ieee802-dot1q-bridge, (urn:onf:yang:firmware-1-0?revision=2021-04-01)firmware-1-0, (urn:ietf:params:xml:ns:yang:ietf-network-instance?revision=2019-01-21)ietf-network-instance, (urn:onf:yang:pure-ethernet-structure-2-0?revision=2022-03-24)pure-ethernet-structure-2-0, (urn:ietf:params:xml:ns:yang:ietf-tls-common?revision=2019-07-02)ietf-tls-common, (urn:ietf:params:xml:ns:yang:ietf-netconf-nmda?revision=2019-01-07)ietf-netconf-nmda, (urn:onf:params:xml:ns:yang:cera-microwave-model?revision=2023-04-03)cera-microwave-model, (urn:ietf:params:xml:ns:yang:ietf-tcp-client?revision=2019-07-02)ietf-tcp-client, (urn:onf:yang:ethernet-container-2-0?revision=2022-04-05)ethernet-container-2-0, (urn:onf:params:xml:ns:yang:cera-lag-profile?revision=2022-10-05)cera-lag-profile, (urn:ietf:params:xml:ns:yang:ietf-ip?revision=2018-02-22)ietf-ip, (urn:ietf:params:xml:ns:yang:ietf-truststore?revision=2019-07-02)ietf-truststore, (urn:ietf:params:xml:ns:yang:ietf-routing?revision=2018-03-13)ietf-routing, (urn:ietf:params:xml:ns:yang:ietf-interfaces?revision=2018-02-20)ietf-interfaces, (urn:ietf:params:xml:ns:yang:ietf-tcp-server?revision=2019-07-02)ietf-tcp-server, (urn:onf:params:xml:ns:yang:cera-abc-profile?revision=2022-08-19)cera-abc-profile, (urn:ieee:std:802.1AB:yang:ieee802-dot1ab-lldp?revision=2021-09-14)ieee802-dot1ab-lldp, (urn:ietf:params:xml:ns:yang:ietf-keystore?revision=2019-07-02)ietf-keystore, (urn:onf:yang:alarms-1-0?revision=2022-03-02)alarms-1-0, (urn:ietf:params:xml:ns:yang:ietf-yang-schema-mount?revision=2019-01-14)ietf-yang-schema-mount, (urn:onf:params:xml:ns:yang:cera-co-channel-profile?revision=2022-10-10)cera-co-channel-profile, (urn:ietf:params:xml:ns:yang:ietf-yang-push?revision=2019-09-09)ietf-yang-push]

2023-10-11T06:58:48,071 | WARN  | remote-connector-processing-executor-11 | DeviceSourcesResolver            | 276 - org.opendaylight.netconf.sal-netconf-connector - 5.0.6 | RemoteDeviceId[name=10.8.120.122:830, address=/10.8.120.122:830]: Adding provided but not required sources as required to prevent failures

2023-10-11T06:58:48,468 | INFO  | remote-connector-processing-executor-1 | MasterSalFacade                  | 281 - org.opendaylight.netconf.topology-singleton - 5.0.6 | Device RemoteDeviceId[name=10.8.120.122:830, address=/10.8.120.122:830] connected - registering master mount point

2023-10-11T06:58:48,469 | INFO  | remote-connector-processing-executor-1 | NetconfDevice                    | 276 - org.opendaylight.netconf.sal-netconf-connector - 5.0.6 | RemoteDeviceId[name=10.8.120.122:830, address=/10.8.120.122:830]: Netconf connector initialized successfully

2023-10-11T07:58:59,502 | INFO  | globalWorkerGroup-3-10 | NetconfClientSessionImpl         | 279 - org.opendaylight.netconf.shaded-sshd - 5.0.6 | requestNewKeysExchange(NetconfClientSessionImpl[admin@/10.8.120.122:830]) Initiating key re-exchange

2023-10-11T07:58:59,522 | WARN  | sshd-NetconfSshClient[4adaf1d1]-nio2-thread-4 | AcceptAllServerKeyVerifier       | 279 - org.opendaylight.netconf.shaded-sshd - 5.0.6 | Server at /10.8.120.122:830 presented unverified EC key: SHA256:mieSv5hSk1LUum30BBwQ47khcl35jUQsFBKCL32mUF0

2023-10-11T08:59:10,987 | INFO  | globalWorkerGroup-3-10 | NetconfClientSessionImpl         | 279 - org.opendaylight.netconf.shaded-sshd - 5.0.6 | requestNewKeysExchange(NetconfClientSessionImpl[admin@/10.8.120.122:830]) Initiating key re-exchange

2023-10-11T08:59:11,007 | WARN  | sshd-NetconfSshClient[4adaf1d1]-nio2-thread-8 | AcceptAllServerKeyVerifier       | 279 - org.opendaylight.netconf.shaded-sshd - 5.0.6 | Server at /10.8.120.122:830 presented unverified EC key: SHA256:mieSv5hSk1LUum30BBwQ47khcl35jUQsFBKCL32mUF0



 Comments   
Comment by Vlad [ 15/Dec/23 ]

Hi ivanhrasko,

 

I just deployed one of the latest versions of the karaf -> and I still facing this issue: mounted 6k(5k real device and 1k simulated devices) - (stopped 1k simulated and 1k real device and the behavior is:

  • real devices 902 get unable to connect and 98 remains connected network topology 
  • simulated devices 1000 in status connecting in network topology 

 

https://nexus.opendaylight.org/content/repositories/opendaylight.snapshot/org/opendaylight/integration/karaf/0.20.0-SNAPSHOT/

Generated at Wed Feb 07 20:16:54 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.