[NETCONF-1114] Incorrect operational state of device configuration with Invalid encrypted password Created: 03/Aug/23  Updated: 03/Nov/23

Status: In Progress
Project: netconf
Component/s: netconf
Affects Version/s: 6.0.0, 5.0.7, 4.0.8
Fix Version/s: 7.0.0

Type: Bug Priority: Medium
Reporter: Sangwook Ha Assignee: Peter Suna
Resolution: Unresolved Votes: 0
Labels: pt
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

Some device configuration with invalid encrypted password causes failure while processing the device configuration and creates incorrect operational state for the device.

For example, the following request which directly creates/updates device configuration with unencrypted password:

PUT /rests/data/network-topology:network-topology/topology=topology-netconf/node=netconf-mdsal

{
    "network-topology:node": [
        {
            "node-id": "netconf-mdsal",
            "netconf-node-topology:concurrent-rpc-limit": 0,
            "netconf-node-topology:schema-cache-directory": "netconf-mdsal",
            "netconf-node-topology:login-password": {
                "username": "admin",
                "password": "admin"
            },
            "netconf-node-topology:default-request-timeout-millis": 1800000,
            "netconf-node-topology:port": 2830,
            "netconf-node-topology:tcp-only": false,
            "netconf-node-topology:host": "127.0.0.1",
            "netconf-node-topology:actor-response-wait-time": 600,
            "netconf-node-topology:keepalive-delay": 600
        }
    ]
}

triggers this exception:

00:51:16.260 ERROR [opendaylight-cluster-data-notification-dispatcher-46] member-1-shard-topology-config: Error notifying listener org.opendaylight.mdsal.binding.dom.adapter.BindingDOMDataTreeChangeListenerAdapter@4abd7f00
java.lang.IllegalArgumentException: Last unit does not have enough valid bits
	at java.util.Base64$Decoder.decode0(Base64.java:867) ~[?:?]
	at java.util.Base64$Decoder.decode(Base64.java:566) ~[?:?]
	at java.util.Base64$Decoder.decode(Base64.java:589) ~[?:?]
	at org.opendaylight.aaa.encrypt.impl.AAAEncryptionServiceImpl.decrypt(AAAEncryptionServiceImpl.java:151) ~[?:?]
	at org.opendaylight.netconf.topology.spi.DefaultNetconfClientConfigurationBuilderFactory.getHandlerFromCredentials(DefaultNetconfClientConfigurationBuilderFactory.java:96) ~[bundleFile:?]
	at org.opendaylight.netconf.topology.spi.DefaultNetconfClientConfigurationBuilderFactory.createClientConfigurationBuilder(DefaultNetconfClientConfigurationBuilderFactory.java:68) ~[bundleFile:?]
	at org.opendaylight.netconf.topology.spi.NetconfNodeHandler.<init>(NetconfNodeHandler.java:143) ~[bundleFile:?]
	at org.opendaylight.netconf.topology.spi.AbstractNetconfTopology.setupConnection(AbstractNetconfTopology.java:142) ~[bundleFile:?]
	at org.opendaylight.netconf.topology.spi.AbstractNetconfTopology.lockedEnsureNode(AbstractNetconfTopology.java:108) ~[bundleFile:?]
	at org.opendaylight.netconf.topology.spi.AbstractNetconfTopology.ensureNode(AbstractNetconfTopology.java:96) ~[bundleFile:?]
	at org.opendaylight.netconf.topology.impl.NetconfTopologyImpl.onDataTreeChanged(NetconfTopologyImpl.java:145) ~[?:?]
	at org.opendaylight.mdsal.binding.dom.adapter.BindingDOMDataTreeChangeListenerAdapter.onDataTreeChanged(BindingDOMDataTreeChangeListenerAdapter.java:44) ~[bundleFile:?]
	at org.opendaylight.controller.cluster.datastore.DataTreeChangeListenerActor.dataTreeChanged(DataTreeChangeListenerActor.java:90) ~[bundleFile:?]
	at org.opendaylight.controller.cluster.datastore.DataTreeChangeListenerActor.handleReceive(DataTreeChangeListenerActor.java:45) ~[bundleFile:?]
	at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:24) ~[bundleFile:?]
	at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:20) ~[bundleFile:?]
	at scala.PartialFunction.applyOrElse(PartialFunction.scala:214) ~[bundleFile:?]
	at scala.PartialFunction.applyOrElse$(PartialFunction.scala:213) ~[bundleFile:?]
	at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:20) ~[bundleFile:?]
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:269) ~[bundleFile:?]
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:270) ~[bundleFile:?]
	at akka.actor.Actor.aroundReceive(Actor.scala:537) ~[bundleFile:?]
	at akka.actor.Actor.aroundReceive$(Actor.scala:535) ~[bundleFile:?]
	at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:220) ~[bundleFile:?]
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:579) ~[bundleFile:?]
	at akka.actor.ActorCell.invoke(ActorCell.scala:547) ~[bundleFile:?]
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270) ~[bundleFile:?]
	at akka.dispatch.Mailbox.run(Mailbox.scala:231) ~[bundleFile:?]
	at akka.dispatch.Mailbox.exec(Mailbox.scala:243) ~[bundleFile:?]
	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373) ~[?:?]
	at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182) ~[?:?]
	at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655) ~[?:?]
	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622) ~[?:?]
	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165) ~[?:?]

The exception thrown from Base64$Decoder.decode while attempting to decrypt the password is not handled while processing the device configuration, hence the process is abruptly aborted and premature operational state is created with connection-status of connecting even though there is no connection attempt:

GET /rests/data/network-topology:network-topology/topology=topology-netconf/node=netconf-mdsal?content=nonconfig

{
    "network-topology:node": [
        {
            "node-id": "netconf-mdsal",
            "netconf-node-topology:port": 2830,
            "netconf-node-topology:connection-status": "connecting",
            "netconf-node-topology:host": "127.0.0.1"
        }
    ]
}

Also, the operational data is not cleaned up even after the device configuration is removed.

Another side effect of this issue is that if there are multiple devices created/updated together, e.g. when controller is restarted with multiple devices configured, then several of them may not be activated even if there just one device with the configuration issue because processing is aborted for the rest once this error is encountered.

 Comments   
Comment by Ruslan Kashapov [ 28/Aug/23 ]

Root cause description – see AbstractNetconfTopology#setupConnection(...)

  • on initialization of deviceSalFacade variable the instance of NetconfDeviceTopologyAdapter is created which immediately (out of constructor) writes device connection status "connecting" to operational datastore.
  • on next variable nodeHandler initialization the instance of NetconfNodeHandler is created, however building a client configuration (within constructor) RuntimeException occurs
  • as result nodeHandler variable is not created and no handler being mapped to nodeId within activeConnectors map
  • when node deletion is requested no associated handler is found; due to operational data cleanup is performed via nodeHandler.close() – which is transferred to NetconfDeviceTopologyAdapter#close() – then handler absence causes node data remaining in operational data store as garbage
Comment by Robert Varga [ 28/Aug/23 ]

So the problem is that the password is allowed to be stored through the PUT request. As the expectation is to have if Base64-encoded, this should by expressed in the YANG model (via a pattern) and the datastore would reject it.

Comment by Ruslan Kashapov [ 29/Aug/23 ]

the password encryption is out of scope for current task, the issue to be solved via NETCONF-1115

The issue with client configuration build failure can be also caused by missing credentials (NPE on building clientConfig). This also lead garbage data remain on node removal

Comment by Ivan Hrasko [ 17/Oct/23 ]

Yes, we have handled any possible exception that can occur. Fix is working but maybe in the future our logic would need more refactoring and better layering to avoid catching plain java's Exception class.

Generated at Wed Feb 07 20:16:42 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.