[NETCONF-1161] Callhome Issues with ODL Netconf and Callhome Codebase

Relates — Wed, 18 Oct 2023 11:43:26 +0000

While trying to establish a connection between a callhome device and the SMO, we’re facing issues while trying to delete the device from the SMO. The following is a summary of the problem that is being faced and observations that have been made after looking at the logs in the SMO and in the callhome device.

Observations: Connecting the callhome device to the SMO

We’re trying to connect a callhome device simulated by a O-RAN-SC simulator

The callhome device is configured with the following well-known callhome xml as part of its sysrepo configuration:

 <call-home>
  <netconf-client>
    <name> default-client </name>
    <endpoints>
      <endpoint>
        <name> default-ssh </name>
        <ssh>
          <tcp-client-parameters>
            <remote-address>172.17.0.7</remote-address>
            <remote-port>6666</remote-port>
            <keepalives>
              <idle-time>1</idle-time>
              <max-probes>10</max-probes>
              <probe-interval>5</probe-interval>
            </keepalives>
          </tcp-client-parameters>
          <ssh-server-parameters>
            <server-identity>
              <host-key>
                <name>default-key</name>
                <public-key>
                  <keystore-reference>genkey</keystore-reference>
                </public-key>
              </host-key>
            </server-identity>
            <client-authentication>
              <supported-authentication-methods>
                <passsword/>
                <other>interactive</other>
              </supported-authentication-methods>
              <users/>
            </client-authentication>
          </ssh-server-parameters>
        </ssh>
      </endpoint>
    </endpoints>
    <connection-type>
      <persistent/>
    </connection-type>
  </netconf-client>
</call-home>

The SDNR/SMO is configured with the following northbound configurations so as to be able to authenticate the callhome device:

# Configure ODL to accept all ssh keys from Callhome Clients

curl -v -u admin:Kp8bJ4SXszM0WXlhak3eHlcse2gAw84vaoGGmJvUy2U -X PUT http://192.168.101.242:30181/rests/data/odl-netconf-callhome-server:netconf-callhome-server/global -H "accept: */*" -H 'content-type: application/json' -d '{ "global": {"accept-all-ssh-keys": "true" } }'

# Configure ODL to accept Callhome devices with credentials 
# user/pwd --> netconf/netconf

curl -v -u admin:Kp8bJ4SXszM0WXlhak3eHlcse2gAw84vaoGGmJvUy2U -X PUT "http://192.168.101.242:30181/rests/data/odl-netconf-callhome-server:netconf-callhome-server/global/credentials" -H "accept: */*" -H "Content-Type:application/json" -d "{\"credentials\":{\"username\":\"netconf\",\"passwords\":[\"netconf\"]}}"

When connecting the callhome device to the SMO, the device connects properly and can be traced in the SDNR karaf.log and also viewed in the ODL GUI. Below is a comparison of the log that is seen in the callhome device and a normal device (i.e. a device that is connected via NETCONF from the SMO)

Opendaylight karaf.log (logs of relevance for the callhome device)

2023-09-13T13:42:08,152 | DEBUG | opendaylight-cluster-data-notification-dispatcher-42 | NetworkElementConnectionEntitiyUtil | 213 - org.onap.ccsdk.features.sdnr.wt.sdnr-wt-devicemanager-core-provider - 1.3.0 |  -  | mount-method: null (null)

...

2023-09-13T13:42:08,119 | INFO  | nioEventLoopGroup-4-1 | NetconfDevice                    | 51 - com.highstreet-technologies.netconf.sal-netconf-connector - 2.0.11 |  -  | RemoteDevice{10.32.0.27:37388}: Netconf connector initialized successfully

Opendaylight karaf.log (logs of relevance for normal device)

2023-09-11T06:38:46,109 | DEBUG | opendaylight-cluster-data-notification-dispatcher-37 | NetworkElementConnectionEntitiyUtil | 213 - org.onap.ccsdk.features.sdnr.wt.sdnr-wt-devicemanager-core-provider - 1.3.0 |  -  | mount-method: LoginPassword (org.opendaylight.yang.gen.v1.urn.opendaylight.netconf.node.topology.rev150114.netconf.node.credentials.credentials.LoginPassword$$$codecImpl)

...

2023-09-11T06:38:49,934 | INFO  | globalWorkerGroup-3-1 | NetconfDevice                    | 51 - com.highstreet-technologies.netconf.sal-netconf-connector - 2.0.11 |  -  | RemoteDevice{samit-du}: Netconf connector initialized successfully

When comparing both sets of logs we see that the callhome device is initialized with a mount-method “null”, whereas the normal device is connected with a mount-method “LoginPassword”. In both cases the devices are connected successfully as it can be seen that an instance of the class “com.highstreet-technologies.netconf.sal-netconf-connector” is created, suggesting that the connection to the device is successful.

Further analysis of the karaf.log suggests that the callhome device is connected properly and that the “hello” routine and the sharing of device capabilities is successful, however, the device capabilities are showing up as empty in case of the callhome device, but not so for the normal device:

Opendaylight karaf.log (logs of relevance for the callhome device)

2023-09-13T13:42:03,297 | INFO  | opendaylight-cluster-data-notification-dispatcher-43 | DeviceManagerNetconfNotConnectHandler | 213 - org.onap.ccsdk.features.sdnr.wt.sdnr-wt-devicemanager-core-provider - 1.3.0 |  -  | onCreated Uri{_value=10.32.0.27:37388}
2023-09-13T13:42:03,298 | INFO  | nioEventLoopGroup-4-2 | CallHomeSessionContext           | 331 - org.opendaylight.netconf.callhome-protocol - 2.0.11 |  -  | Activating Netconf channel for /10.32.0.27:37388 with org.opendaylight.netconf.callhome.mount.CallHomeMountSessionContext$1@67f9daac
2023-09-13T13:42:03,316 | INFO  | opendaylight-cluster-data-notification-dispatcher-43 | Capabilities                     | 228 - org.onap.ccsdk.features.sdnr.wt.sdnr-wt-netconfnode-state-service-model - 1.3.0 |  -  | GetAvailableCapabilities for node
2023-09-13T13:42:03,327 | DEBUG | opendaylight-cluster-data-notification-dispatcher-43 | Capabilities                     | 228 - org.onap.ccsdk.features.sdnr.wt.sdnr-wt-netconfnode-state-service-model - 1.3.0 |  -  | empty capabilites
2023-09-13T13:42:03,327 | INFO  | opendaylight-cluster-data-notification-dispatcher-43 | Capabilities                     | 228 - org.onap.ccsdk.features.sdnr.wt.sdnr-wt-netconfnode-state-service-model - 1.3.0 |  -  | GetUnavailableCapabilities for node
2023-09-13T13:42:03,334 | DEBUG | opendaylight-cluster-data-notification-dispatcher-43 | Capabilities                     | 228 - org.onap.ccsdk.features.sdnr.wt.sdnr-wt-netconfnode-state-service-model - 1.3.0 |  -  | empty capabilites

...

2023-09-13T13:42:03,360 | DEBUG | opendaylight-cluster-data-notification-dispatcher-43 | UpdateRequest                    | 208 - org.onap.ccsdk.features.sdnr.wt.sdnr-wt-common - 1.3.0 |  -  | update payload: {"upsert":{"is-required":false,"node-id":"10.32.0.27:37388","core-model-capability":"Unsupported","port":37388,"device-type":"Unknown","host":"10.32.0.27","node-details":{"available-capabilities":[],"unavailable-capabilities":[]},"id":"10.32.0.27:37388","status":"Connecting"},"script":{"source":"ctx._source['node-id']=\"10.32.0.27:37388\";ctx._source['core-model-capability']=\"Unsupported\";ctx._source['port']=37388;ctx._source['device-type']=\"Unknown\";ctx._source['host']=\"10.32.0.27\";ctx._source['node-details']=params.p0;ctx._source['id']=\"10.32.0.27:37388\";ctx._source['status']=\"Connecting\";","lang":"painless","params":{"p0":{"available-capabilities":[],"unavailable-capabilities":[]}}}}

Opendaylight karaf.log (logs of relevance for normal device)

2023-09-11T06:38:49,941 | INFO  | opendaylight-cluster-data-notification-dispatcher-34 | Capabilities                     | 228 - org.onap.ccsdk.features.sdnr.wt.sdnr-wt-netconfnode-state-service-model - 1.3.0 |  -  | GetAvailableCapabilities for node
2023-09-11T06:38:49,943 | INFO  | opendaylight-cluster-data-notification-dispatcher-34 | DeviceManagerNetconfConnectHandler | 213 - org.onap.ccsdk.features.sdnr.wt.sdnr-wt-devicemanager-core-provider - 1.3.0 |  -  | onEnterConnected - starting Event listener on Netconf for mountpoint samit-du
2023-09-11T06:38:49,943 | INFO  | opendaylight-cluster-data-notification-dispatcher-34 | DeviceManagerNetconfConnectHandler | 213 - org.onap.ccsdk.features.sdnr.wt.sdnr-wt-devicemanager-core-provider - 1.3.0 |  -  | Master mountpoint samit-du
2023-09-11T06:38:49,943 | INFO  | opendaylight-cluster-data-notification-dispatcher-34 | DeviceManagerNetconfConnectHandler | 213 - org.onap.ccsdk.features.sdnr.wt.sdnr-wt-devicemanager-core-provider - 1.3.0 |  -  | update ConnectedState for device :: Name : samit-du ConnectionStatus Connected

2023-09-11T06:38:49,947 | INFO  | opendaylight-cluster-data-notification-dispatcher-34 | Capabilities                     | 228 - org.onap.ccsdk.features.sdnr.wt.sdnr-wt-netconfnode-state-service-model - 1.3.0 |  -  | GetAvailableCapabilities for node

2023-09-11T06:38:49,947 | INFO  | opendaylight-cluster-data-notification-dispatcher-34 | Capabilities                     | 228 - org.onap.ccsdk.features.sdnr.wt.sdnr-wt-netconfnode-state-service-model - 1.3.0 |  -  | GetUnavailableCapabilities for node

...

2023-09-11T06:38:49,954 | INFO  | opendaylight-cluster-data-notification-dispatcher-34 | HtDatabaseEventsService          | 211 - org.onap.ccsdk.features.sdnr.wt.sdnr-wt-data-provider-provider - 1.3.0 |  -  | update networkelement-connection for samit-du with data NetworkElementConnection{coreModelCapability=Unsupported, deviceType=Unknown, host=10.32.0.28, id=samit-du, isRequired=false, mountMethod=LoginPassword, nodeDetails=NodeDetails{availableCapabilities=[urn:ietf:params:netconf:capability:notification:1.0, urn:ietf:params:netconf:capability:with-defaults:1.0?basic-mode=explicit&also-supported=report-all,report-all-tagged,trim,explicit, urn:ietf:params:netconf:capability:confirmed-commit:1.1, urn:ietf:params:netconf:capability:xpath:1.0, urn:ietf:params:netconf:capability:writable-running:1.0, urn:ietf:params:netconf:capability:interleave:1.0, urn:ietf:params:netconf:capability:rollback-on-error:1.0, urn:ietf:params:netconf:base:1.0, urn:ietf:params:netconf:base:1.1, urn:ietf:params:netconf:capability:startup:1.0, urn:ietf:params:netconf:capability:candidate:1.0, urn:ietf:params:netconf:capability:validate:1.1, urn:ietf:params:netconf:capability:yang-library:1.1?revision=2019-01-04&content-id=59, (urn:ietf:params:xml:ns:yang:ietf-yang-schema-mount?revision=2019-01-14)ietf-yang-schema-mount, (urn:aarna-yang-augment-5a?revision=2022-08-07)aarna-yang-augment-5a, (urn:3gpp:sa5:_3gpp-common-subnetwork?revision=2020-03-11)_3gpp-common-subnetwork, (urn:aarna-yang-augment-4?revision=2022-08-06)aarna-yang-augment-4, (urn:3gpp:sa5:_3gpp-nr-nrm-nrcelldu?revision=2020-02-14)_3gpp-nr-nrm-nrcelldu, (urn:3gpp:sa5:_3gpp-common-ep-rp?revision=2019-06-17)_3gpp-common-ep-rp, (urn:aarna-z5a?revision=2022-08-06)aarna-z5a, (urn:ietf:params:xml:ns:yang:ietf-ssh-server?revision=2019-07-02)ietf-ssh-server, (urn:3gpp:sa5:_3gpp-nr-nrm-nrnetwork-nrfrequency?revision=2019-10-28)_3gpp-nr-nrm-nrfrequency, (urn:ietf:params:xml:ns:yang:ietf-ssh-common?revision=2019-07-02)ietf-ssh-common, (urn:o-ran:antcal:1.0?revision=2021-12-01)o-ran-antenna-calibration, (urn:aarna-yang-features-4?revision=2022-08-17)aarna-yang-features-4, (urn:3gpp:sa5:_3gpp-common-yang-types?revision=2020-03-10)_3gpp-common-yang-types, (urn:o-ran:oran-gnbdufunction?revision=2020-09-07)o-ran_3gpp-nr-nrm-gnbdufunction, (urn:ietf:params:xml:ns:yang:ietf-yang-patch?revision=2017-02-22)ietf-yang-patch, (urn:3gpp:sa5:3gpp-nr-nrm-common?revision=2020-02-14)_3gpp-nr-nrm-common, (urn:ietf:params:xml:ns:yang:iana-crypt-hash?revision=2014-08-06)iana-crypt-hash, (urn:o-ran:oran-nrcelldu?revision=2020-09-07)o-ran_3gpp-nr-nrm-nrcelldu, (urn:ietf:params:xml:ns:netmod:notification?revision=2008-07-14)nc-notifications, (urn:ietf:params:xml:ns:yang:ietf-crypto-types?revision=2019-07-02)ietf-crypto-types, (urn:o-ran:trace:1.0?revision=2019-07-03)o-ran-trace, (urn:3gpp:sa5:_3gpp-nr-nrm-ep?revision=2020-03-02)_3gpp-nr-nrm-ep, (urn:aarna-yang-features-2?revision=2022-09-09)aarna-yang-features-2, (urn:ietf:params:xml:ns:yang:iana-hardware?revision=2018-03-13)iana-hardware, (urn:3gpp:sa5:_3gpp-nr-nrm-eutrancellrelation?revision=2019-10-28)_3gpp-nr-nrm-eutrancellrelation, (urn:3gpp:sa5:_3gpp-nr-nrm-eutranfreqrelation?revision=2019-10-28)_3gpp-nr-nrm-eutranfreqrelation, (urn:aarna-yang-augment-1?revision=2022-08-12)aarna-yang-augment-1, (urn:ietf:params:xml:ns:yang:ietf-yang-metadata?revision=2016-08-05)ietf-yang-metadata, (urn:3gpp:sa5:_3gpp-nr-nrm-nrnetwork-beam?revision=2019-11-22)_3gpp-nr-nrm-beam, (urn:3gpp:sa5:_3gpp-common-measurements?revision=2020-03-11)_3gpp-common-measurements, (urn:3gpp:sa5:_3gpp-nr-nrm-nrnetwork-nrsectorcarrier?revision=2019-10-28)_3gpp-nr-nrm-nrsectorcarrier, (urn:ietf:params:xml:ns:yang:ietf-datastores?revision=2018-02-14)ietf-datastores, (urn:3gpp:sa5:_3gpp-common-fm?revision=2020-02-24)_3gpp-common-fm, (urn:ietf:params:xml:ns:yang:ietf-tcp-server?revision=2019-07-02)ietf-tcp-server, (http://www.sysrepo.org/yang/sysrepo?revision=2021-10-08)sysrepo, (urn:ietf:params:xml:ns:yang:ietf-netconf-nmda?revision=2019-01-07)ietf-netconf-nmda, (urn:o-ran:hardware:1.0?revision=2019-07-03)o-ran-hardware, (urn:aarna-top-2?revision=2022-08-07)aarna-top-2, (urn:3gpp:sa5:_3gpp-nr-nrm-gnbdufunction?revision=2020-03-12)_3gpp-nr-nrm-gnbdufunction, (urn:ietf:params:xml:ns:yang:ietf-ip?revision=2018-02-22)ietf-ip, (urn:ietf:params:xml:ns:yang:ietf-yang-library?revision=2019-01-04)ietf-yang-library, (urn:ietf:params:xml:ns:netconf:base:1.0?revision=2013-09-29)ietf-netconf, (urn:o-ran:beamforming:1.0?revision=2021-12-01)o-ran-beamforming, (urn:ietf:params:xml:ns:yang:ietf-netconf-acm?revision=2018-02-14)ietf-netconf-acm, (urn:o-ran:c-plane-tnl:1.0?revision=2020-09-08)o-ran-c-plane-tnl, (urn:o-ran:rlc:1.0?revision=2020-09-04)o-ran-rlc, (urn:ietf:params:xml:ns:yang:ietf-tls-common?revision=2019-07-02)ietf-tls-common, (urn:aarna-yang-types?revision=2022-09-02)aarna-yang-types, (urn:3gpp:sa5:_3gpp-nr-nrm-nrfreqrelation?revision=2019-10-28)_3gpp-nr-nrm-nrfreqrelation, (urn:o-ran:fm:1.0?revision=2019-02-04)o-ran-fm, (urn:3gpp:sa5:_3gpp-common-managed-element?revision=2020-02-24)_3gpp-common-managed-element, (urn:aarna-yang-features-1?revision=2022-09-22)aarna-yang-features-1, (urn:o-ran:compression-factors:1.0?revision=2021-12-01)o-ran-compression-factors, (urn:o-ran:delay:1.0?revision=2021-12-01)o-ran-delay-management, (urn:ietf:params:xml:ns:yang:ietf-interfaces?revision=2018-02-20)ietf-interfaces, (urn:3gpp:sa5:3gpp-nr-nrm-nrnetwork-rrmpolicy?revision=2020-02-14)_3gpp-nr-nrm-rrmpolicy, (urn:o-ran:interfaces:1.0?revision=2021-12-01)o-ran-interfaces, (urn:ietf:params:xml:ns:yang:ietf-yang-types?revision=2013-07-15)ietf-yang-types, (urn:3gpp:sa5:_3gpp-common-subscription-control?revision=2019-11-29)_3gpp-common-subscription-control, (urn:ietf:params:xml:ns:yang:ietf-hardware?revision=2018-03-13)ietf-hardware, (urn:o-ran:uplane-conf:1.0?revision=2021-12-01)o-ran-uplane-conf, (urn:3gpp:sa5:_3gpp-common-managed-function?revision=2019-11-21)_3gpp-common-managed-function, (urn:ietf:params:xml:ns:yang:ietf-tls-server?revision=2019-07-02)ietf-tls-server, (urn:ietf:params:xml:ns:yang:ietf-restconf?revision=2017-01-26)ietf-restconf, (urn:ietf:params:xml:ns:yang:ietf-keystore?revision=2019-07-02)ietf-keystore, (urn:o-ran:wg4feat:1.0?revision=2021-12-01)o-ran-wg4-features, (urn:ietf:params:xml:ns:yang:ietf-subscribed-notifications?revision=2019-09-09)ietf-subscribed-notifications, (urn:ietf:params:xml:ns:yang:ietf-x509-cert-to-name?revision=2014-12-10)ietf-x509-cert-to-name, (urn:3gpp:sa5:_3gpp-nr-nrm-bwp?revision=2019-10-28)_3gpp-nr-nrm-bwp, (urn:ietf:params:xml:ns:yang:smiv2:InternetGatewayDeviceDU?revision=2012-10-05)InternetGatewayDeviceDU, (urn:3gpp:sa5:_3gpp-nr-nrm-gnbcuupfunction?revision=2020-03-12)_3gpp-nr-nrm-gnbcuupfunction, (urn:aarna-yang-features-3?revision=2022-09-03)aarna-yang-features-3, (urn:ietf:params:xml:ns:yang:ietf-inet-types?revision=2013-07-15)ietf-inet-types, (urn:3gpp:sa5:_3gpp-nr-nrm-gnbcucpfunction?revision=2020-02-14)_3gpp-nr-nrm-gnbcucpfunction, (urn:ietf:params:xml:ns:yang:ietf-yang-push?revision=2019-09-09)ietf-yang-push, (urn:ietf:params:xml:ns:yang:ietf-netconf-monitoring?revision=2010-10-04)ietf-netconf-monitoring, (http://www.sysrepo.org/yang/sysrepo-monitoring?revision=2022-08-19)sysrepo-monitoring, (urn:ietf:params:xml:ns:yang:iana-if-type?revision=2019-02-08)iana-if-type, (urn:ietf:params:xml:ns:yang:ietf-tcp-common?revision=2019-07-02)ietf-tcp-common, (urn:aarna-yang-elements-1?revision=2022-08-11)aarna-yang-elements-1, (urn:ietf:params:xml:ns:yang:1?revision=2022-06-16)yang, (urn:ietf:params:xml:ns:yang:ietf-netconf-with-defaults?revision=2011-06-01)ietf-netconf-with-defaults, (urn:o-ran:operations:1.0?revision=2021-12-01)o-ran-operations, (urn:3gpp:sa5:_3gpp-nr-nrm-nrcellcu?revision=2020-02-14)_3gpp-nr-nrm-nrcellcu, (urn:o-ran:software-management:1.0?revision=2019-07-03)o-ran-software-management, (urn:ietf:params:xml:ns:yang:ietf-network-instance?revision=2019-01-21)ietf-network-instance, (urn:o-ran:module-cap:1.0?revision=2021-12-01)o-ran-module-cap, (urn:o-ran:file-management:1.0?revision=2019-07-03)o-ran-file-management, (urn:ietf:params:xml:ns:netconf:notification:1.0?revision=2008-07-14)notifications, (urn:ietf:params:xml:ns:yang:ietf-netconf-server?revision=2019-07-02)ietf-netconf-server, (urn:aarna-yang-augment-2?revision=2022-08-05)aarna-yang-augment-2, (urn:o-ran:udpecho:1.0?revision=2019-02-04)o-ran-udp-echo, (urn:3gpp:sa5:_3gpp-nr-nrm-nrcellrelation?revision=2019-10-28)_3gpp-nr-nrm-nrcellrelation, (urn:3gpp:sa5:_3gpp-nr-nrm-nrnetwork?revision=2019-06-17)_3gpp-nr-nrm-nrnetwork, (urn:ietf:params:xml:ns:yang:smiv2:InternetGatewayDevice?revision=2012-09-05)InternetGatewayDevice, (urn:o-ran:processing-element:1.0?revision=2021-12-01)o-ran-processing-element, (urn:o-ran:u-plane-tnl:1.0?revision=2020-09-08)o-ran-u-plane-tnl, (urn:sysrepo:plugind?revision=2022-08-26)sysrepo-plugind, (urn:capgemini:o-ran-file-mgmt?revision=2022-01-20)capgemini-o-ran-file-management, (urn:ietf:params:xml:ns:yang:ietf-origin?revision=2018-02-14)ietf-origin, (urn:ietf:params:xml:ns:yang:ietf-yang-structure-ext?revision=2020-06-17)ietf-yang-structure-ext, (urn:3gpp:sa5:_3gpp-common-top?revision=2019-06-17)_3gpp-common-top, (urn:o-ran:qos:1.0?revision=2020-09-08)o-ran-qos, (urn:o-ran:sync:1.0?revision=2019-07-03)o-ran-sync, (urn:ietf:params:xml:ns:yang:ietf-truststore?revision=2019-07-02)ietf-truststore, (urn:ietf:params:xml:ns:yang:ietf-tcp-client?revision=2019-07-02)ietf-tcp-client, (urn:3gpp:sa5:_3gpp-nr-nrm-nrnetwork-commonbeamformingfunction?revision=2019-11-22)_3gpp-nr-nrm-commonbeamformingfunction, (urn:ietf:params:xml:ns:yang:ietf-netconf-notifications?revision=2012-02-06)ietf-netconf-notifications, (urn:aarna-yang-augment-3?revision=2022-08-05)aarna-yang-augment-3], unavailableCapabilities=[]}, nodeId=samit-du, password=netconf, port=830, status=Connected, username=netconf}

Observations: Disconnecting the callhome Device from the SMO

After the device is connected and we try to remove the device, we see that the NETCONF/SSH connection between the SMO and callhome device doesn’t get removed and the mount point that we’re trying to remove using the northbound interface is not found. We encounter the following error in the GUI suggesting that the callhome device is not found: "failed to unmount [10.32.0.27:50802]" suggesting that the device referenced from the callhome provider is not found in topology:netconf.

When we analyze the problem further, we see that the reason for not finding the device is that the capabilities are not getting stored in the callhome implementation. However, if we query the device from the northbound RESTCONF interface in the topology-netconf, we’re able to see the device with its capabilities (For simplicity, the output of the RESTCONF output has been truncated):

curl -X GET "http://192.168.101.242:30181/rests/data/network-topology:network-topology/topology=topology-netconf/node=10.32.0.27%3A50802" -H "accept: application/xml"

Output of curl command to get the details of the callhome device

<node xmlns="urn:TBD:params:xml:ns:yang:network-topology">
  <node-id>10.32.0.27:50802</node-id>
  <connection-status xmlns="urn:opendaylight:netconf-node-topology">connected</connection-status>
  <port xmlns="urn:opendaylight:netconf-node-topology">50802</port>
  <available-capabilities xmlns="urn:opendaylight:netconf-node-topology">
    <available-capability>
      <capability>urn:ietf:params:netconf:capability:validate:1.1</capability>
      <capability-origin>device-advertised</capability-origin>
    </available-capability>
    <available-capability>      

...      

      <capability>(urn:3gpp:sa5:3gpp-nr-nrm-nrnetwork-rrmpolicy?revision=2020-02-14)_3gpp-nr-nrm-rrmpolicy</capability>
    </available-capability>
    <available-capability>
      <capability>(urn:o-ran:interfaces:1.0?revision=2021-12-01)o-ran-interfaces</capability>
    </available-capability>
    <available-capability>
      <capability>(urn:ietf:params:xml:ns:yang:ietf-hardware?revision=2018-03-13)ietf-hardware</capability>
    </available-capability>
  </available-capabilities>
  <unavailable-capabilities xmlns="urn:opendaylight:netconf-node-topology"/>
  <host xmlns="urn:opendaylight:netconf-node-topology">10.32.0.27</host>
</node>

What we can deduce from the above is that the Opendaylight callhome implementation which is “org.opendaylight.netconf.callhome-provider” seems to be having trouble loading the capabilities of the callhome device. As a result of which, the “in memory” representation of the device and the actual storage of the device differ. If we change the representation of the device after loading it to only contain the fields by which the callhome provider represents the device, by executing the following RESTCONF API:

curl -X PUT "http://192.168.101.242:30181/rests/data/network-topology:network-topology/topology=topology-netconf/node=10.32.0.27%3A57652" -H "accept: */*" -H "Content-Type: application/json" -d "{\"node\":[{\"node-id\":\"10.32.0.27:57652\",\"port\":57652,\"host\":\"10.32.0.27\",\"username\":\"netconf\",\"password\":\"netconf\"}]}"

What we have done is change the fields of the device to contain the minimal set by which it can be recognized by the callhome provider. Below is a clearer representation of the callhome device used in the RESTCONF call above:

{
  "node": [
    {
      "node-id": "10.32.0.27:57652",
      "port": 57652,
      "host": "10.32.0.27",
      "username": "netconf",
      "password": "netconf"
    }
  ]
}

Notice that this device does not have any capabilities and it only contains the extra “username” and “password” by which the callhome provider can identify it. Now if we remove the device from the SMO GUI, then there is no unmount error.

On removal of the device, though the device gets removed from the GUI and the database, however, the NETCONF session between the callhome device and the SMO remains intact, which is not the expected behavior. If we compare this behavior with that of a normal device, we see different activity in the logs. If we compare the logs below, we see that there is an entry in the logs where the “com.highstreet-technologies.netconf.sal-netconf-connector” removes the session between the device and the SMO in the case of a normal device, but this is not seen to happen with the callhome device.{}

Opendaylight karaf.log (logs of relevance for the callhome device)

2023-09-19T14:17:46,404 | DEBUG | qtp1197466357-1030 | DataProviderServiceImpl          | 211 - org.onap.ccsdk.features.sdnr.wt.sdnr-wt-data-provider-provider - 1.3.0 |  -  | RPC Request: deleteNetworkElementConnection with input DeleteNetworkElementConnectionInput{id=10.32.0.27:51890}
2023-09-19T14:17:46,411 | DEBUG | qtp1197466357-1030 | BaseRequest                      | 208 - org.onap.ccsdk.features.sdnr.wt.sdnr-wt-common - 1.3.0 |  -  | create request DELETE /networkelement-connection/networkelement-connection/10.32.0.27%3A51890 with refresh=true
2023-09-19T14:17:46,411 | DEBUG | qtp1197466357-1030 | BaseRequest                      | 208 - org.onap.ccsdk.features.sdnr.wt.sdnr-wt-common - 1.3.0 |  -  | create request DELETE /networkelement-connection/_doc/10.32.0.27%3A51890 with refresh=true
2023-09-19T14:17:46,499 | DEBUG | qtp1197466357-1030 | BaseResponse                     | 208 - org.onap.ccsdk.features.sdnr.wt.sdnr-wt-common - 1.3.0 |  -  | parsing response={"_index":"networkelement-connection-v7","_type":"_doc","_id":"10.32.0.27:51890","_version":5,"result":"deleted","forced_refresh":true,"_shards":{"total":2,"successful":1,"failed":0},"_seq_no":315,"_primary_term":1}

Opendaylight karaf.log (logs of relevance for normal device)

2023-09-11T06:39:17,482 | DEBUG | qtp1654343517-587 | DataProviderServiceImpl          | 211 - org.onap.ccsdk.features.sdnr.wt.sdnr-wt-data-provider-provider - 1.3.0 |  -  | RPC Request: deleteNetworkElementConnection with input DeleteNetworkElementConnectionInput{id=samit-du}
2023-09-11T06:39:17,486 | DEBUG | qtp1654343517-587 | BaseRequest                      | 208 - org.onap.ccsdk.features.sdnr.wt.sdnr-wt-common - 1.3.0 |  -  | create request DELETE /networkelement-connection/networkelement-connection/samit-du with refresh=true
2023-09-11T06:39:17,487 | DEBUG | qtp1654343517-587 | BaseRequest                      | 208 - org.onap.ccsdk.features.sdnr.wt.sdnr-wt-common - 1.3.0 |  -  | create request DELETE /networkelement-connection/_doc/samit-du with refresh=true
2023-09-11T06:39:17,571 | DEBUG | qtp1654343517-587 | BaseResponse                     | 208 - org.onap.ccsdk.features.sdnr.wt.sdnr-wt-common - 1.3.0 |  -  | parsing response={"_index":"networkelement-connection-v7","_type":"_doc","_id":"samit-du","_version":6,"result":"deleted","forced_refresh":true,"_shards":{"total":2,"successful":1,"failed":0},"_seq_no":167,"_primary_term":1}
2023-09-11T06:39:17,802 | WARN  | opendaylight-cluster-data-notification-dispatcher-38 | NetconfDeviceCommunicator        | 51 - com.highstreet-technologies.netconf.sal-netconf-connector - 2.0.11 |  -  | RemoteDevice{samit-du}: Session terminated Session closed
2023-09-11T06:39:17,812 | WARN  | opendaylight-cluster-data-notification-dispatcher-37 | CallhomeStatusReporter           | 332 - org.opendaylight.netconf.callhome-provider - 2.0.11 |  -  | No corresponding callhome device found - exiting.
2023-09-11T06:39:17,813 | INFO  | opendaylight-cluster-data-notification-dispatcher-37 | NetconfNodeStateServiceImpl      | 229 - org.onap.ccsdk.features.sdnr.wt.sdnr-wt-netconfnode-state-service-provider - 1.3.0 |  -  | L2 TreeChange enter changes:1
2023-09-11T06:39:17,813 | INFO  | opendaylight-cluster-data-notification-dispatcher-37 | NetconfNodeStateServiceImpl      | 229 - org.onap.ccsdk.features.sdnr.wt.sdnr-wt-netconfnode-state-service-provider - 1.3.0 |  -  | L2 TreeChange leave
2023-09-11T06:39:17,815 | WARN  | opendaylight-cluster-data-notification-dispatcher-37 | CallhomeStatusReporter           | 332 - org.opendaylight.netconf.callhome-provider - 2.0.11 |  -  | No corresponding callhome device found - exiting.
2023-09-11T06:39:17,816 | WARN  | opendaylight-cluster-data-notification-dispatcher-37 | CallhomeStatusReporter           | 332 - org.opendaylight.netconf.callhome-provider - 2.0.11 |  -  | No corresponding callhome device found - exiting.
2023-09-11T06:39:17,826 | WARN  | opendaylight-cluster-data-notification-dispatcher-37 | CallhomeStatusReporter           | 332 - org.opendaylight.netconf.callhome-provider - 2.0.11 |  -  | No corresponding callhome device found - exiting.
2023-09-11T06:39:17,826 | INFO  | opendaylight-cluster-data-notification-dispatcher-37 | NetconfNodeStateServiceImpl      | 229 - org.onap.ccsdk.features.sdnr.wt.sdnr-wt-netconfnode-state-service-provider - 1.3.0 |  -  | L2 TreeChange enter changes:1
2023-09-11T06:39:17,826 | INFO  | opendaylight-cluster-data-notification-dispatcher-37 | NetconfNodeStateServiceImpl      | 229 - org.onap.ccsdk.features.sdnr.wt.sdnr-wt-netconfnode-state-service-provider - 1.3.0 |  -  | L2 TreeChange leave2023-09-11T06:39:17,831 | WARN  | opendaylight-cluster-data-notification-dispatcher-37 | CallhomeStatusReporter           | 332 - org.opendaylight.netconf.callhome-provider - 2.0.11 |  -  | No corresponding callhome device found - exiting.
2023-09-11T06:39:17,832 | WARN  | opendaylight-cluster-data-notification-dispatcher-38 | CallhomeStatusReporter           | 332 - org.opendaylight.netconf.callhome-provider - 2.0.11 |  -  | No corresponding callhome device found - exiting.

OpenDaylight JIRA

[NETCONF-1161] Callhome Issues with ODL Netconf and Callhome Codebase

Observations: Connecting the callhome device to the SMO

Observations: Disconnecting the callhome Device from the SMO