[CONTROLLER-717] Remote Netconf Server Initialization Failed With NullPointerException Created: 22/Aug/14  Updated: 25/Jul/23  Resolved: 03/Sep/14

Status: Resolved
Project: controller
Component/s: netconf
Affects Version/s: None
Fix Version/s: None

Type: Bug
Reporter: Sachi Gupta Assignee: Maros Marsalek
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: Linux
Platform: PC


Attachments: File Opendaylight.log:26-08-2014     Text File snapshot.txt    
External issue ID: 1601
Priority: Highest

 Description   

Remote Netconf Server Connection With new cloned Opendaylight controller(Cloned on Aug 21, 2013) failed with java null pointer exception.

Steps followed to replicate the issue:

1. Clone new Opnedaylight controller
2. Created the xml file and added the file in configuration/initial folder of the distribution that we are running
3. Added the remote device configuration yang files in cache/schema directory
4. Run the controller

Running the controller failed to initialize sal and throws java null pointer exception as below

2014-08-22 10:59:21.743 IST [remote-connector-processing-executor-2] WARN o.o.c.s.c.netconf.NetconfDevice - RemoteDevice

{m10sachi}: Unexpected error resolving device sources: java.lang.NullPointerException
2014-08-22 10:59:21.770 IST [remote-connector-processing-executor-2] ERROR o.o.c.s.c.netconf.NetconfDevice - RemoteDevice{m10sachi}

: Initialization in sal failed, disconnecting from device
java.lang.NullPointerException: null
at org.opendaylight.controller.sal.connect.netconf.NetconfStateSchemas.create(NetconfStateSchemas.java:128) ~[na:na]
at org.opendaylight.controller.sal.connect.netconf.NetconfStateSchemas.create(NetconfStateSchemas.java:118) ~[na:na]
at org.opendaylight.controller.sal.connect.netconf.NetconfStateSchemas.access$000(NetconfStateSchemas.java:35) ~[na:na]
at org.opendaylight.controller.sal.connect.netconf.NetconfStateSchemas$NetconfStateSchemasResolverImpl.resolve(NetconfStateSchemas.java:53) ~[na:na]
at org.opendaylight.controller.sal.connect.netconf.NetconfDevice$DeviceSourcesResolver.call(NetconfDevice.java:222) ~[na:na]
at org.opendaylight.controller.sal.connect.netconf.NetconfDevice$DeviceSourcesResolver.call(NetconfDevice.java:202) ~[na:na]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_55]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_55]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_55]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_55]
2014-08-22 10:59:21.771 IST [remote-connector-processing-executor-2] WARN o.o.c.s.c.n.l.NetconfDeviceCommunicator - RemoteDevice

{m10sachi}

: Session terminated Session closed



 Comments   
Comment by Maros Marsalek [ 22/Aug/14 ]

Hi, I have fixed the NullPointer exception. But it looks like your device does not respond correctly to get request with filter netconf-state/schemas. So it looks like your device supports netconf monitoring, but does not expose any schemas.

Here is the commit to fix null pointer:
https://git.opendaylight.org/gerrit/#/c/10158/

But since your device does not support netconf monitoring, you would have to place your schemas into cache folder before ODL start.

Comment by Sachi Gupta [ 22/Aug/14 ]

Hi,

The null pointer exception is resolved with your patch

Have added the schema files in cache/schema folder but still getting one more exception as below:

2014-08-22 15:01:49.289 IST [remote-connector-processing-executor-2] WARN o.o.c.s.c.n.NetconfStateSchemas - RemoteDevice

{m10sachi}: Unable to detect available schemas, get to (urn:ietf:params:xml:ns:yang:ietf-netconf-monitoring?revision=2010-10-04)netconf-state/(urn:ietf:params:xml:ns:yang:ietf-netconf-monitoring?revision=2010-10-04)schemas was empty
2014-08-22 15:01:49.313 IST [remote-connector-processing-executor-3] ERROR o.o.c.s.c.netconf.NetconfDevice - RemoteDevice{m10sachi}

: Initialization in sal failed, disconnecting from device
java.lang.IllegalStateException: RemoteDevice

{m10sachi}: No more sources for schema context
at org.opendaylight.controller.sal.connect.netconf.NetconfDevice$RecursiveSchemaSetup.setUpSchema(NetconfDevice.java:305) [bundlefile:na]
at org.opendaylight.controller.sal.connect.netconf.NetconfDevice$RecursiveSchemaSetup.run(NetconfDevice.java:294) [bundlefile:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_55]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_55]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_55]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_55]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_55]
2014-08-22 15:01:49.316 IST [remote-connector-processing-executor-3] WARN o.o.c.s.c.n.l.NetconfDeviceCommunicator - RemoteDevice{m10sachi}

: Session terminated Session closed

Comment by Maros Marsalek [ 25/Aug/14 ]

I see, can you show me the hello message produced by your device ?

Comment by Sachi Gupta [ 26/Aug/14 ]

Hi Maros,

The hello message produced by the device is as follows:

<hello xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
<capabilities>
<capability>urn:ietf:params:netconf:base:1.0</capability>
<capability>urn:ietf:params:netconf:capability:candidate:1.0</capability>
<capability>urn:ietf:params:netconf:capability:confirmed-commit:1.0</capability>
<capability>urn:ietf:params:netconf:capability:validate:1.0</capability>
<capability>urn:ietf:params:netconf:capability:url:1.0?scheme=http,ftp,file</capability>
<capability>urn:ietf:params:xml:ns:netconf:base:1.0</capability>
<capability>urn:ietf:params:xml:ns:netconf:capability:candidate:1.0</capability>
<capability>urn:ietf:params:xml:ns:netconf:capability:confirmed-commit:1.0</capability>
<capability>urn:ietf:params:xml:ns:netconf:capability:validate:1.0</capability>
<capability>urn:ietf:params:xml:ns:netconf:capability:url:1.0?protocol=http,ftp,file</capability>
<capability>http://xml.juniper.net/netconf/junos/1.0</capability>
<capability>http://yang.juniper.net/yang/1.1/jc?module=configuration</capability>
<capability>urn:ietf:params:xml:ns:yang:ietf-netconf-monitoring</capability>
<capability>http://xml.juniper.net/dmi/system/1.0</capability>
</capabilities>
<session-id>27216</session-id>
</hello>

Comment by Maros Marsalek [ 26/Aug/14 ]

Looking at the hello message, not many yang based capabilities there. Except maybe: <capability>http://yang.juniper.net/yang/1.1/jc?module=configuration</capability>. What yang modules did you put under cache/schema folder ?

Or what yang modules do you expect to be downloaded from the device and used ?

Because your device provides no schemas, and in hello message there is maybe one yang schema reported. In case you need to override the capabilities from hello message, there is a way via netconf-connector configuretion.

Comment by Sachi Gupta [ 26/Aug/14 ]

Hi Maros,

I have already added configuration.yang file in cache/schema folder, but faced the above mentioned issue.

Note: The same setup was working fine with the older odl code(Cloned on June 19,2014) but showing the issues with new odl code(Cloned on Aug 21, 2014)

Comment by Maros Marsalek [ 26/Aug/14 ]

Can you show me the whole log ? or can I connect to your device from my PC to debug ?

Comment by Sachi Gupta [ 26/Aug/14 ]

Hi Maros,

Device does not have internet connectivity, so you will not be able to connect to my netconf device.

Sharing with you the log file.

Comment by Sachi Gupta [ 26/Aug/14 ]

Attachment Opendaylight.log:26-08-2014 has been added with description: Log File Attached

Comment by Maros Marsalek [ 27/Aug/14 ]

Ok,

So your device has this capability in hello message
http://yang.juniper.net/yang/1.1/jc?module=configuration
without revision.

Netconf-connector currently ignores capabilities without revision parameter.

What you can do as a workaround:

1. add revision to the schema in cache/schema folder.
2. change configuration of netconf-connector, by adding this list property:
<yang-module-capabilities>
<capability>
http://yang.juniper.net/yang/1.1/jc?module=configuration?revision=YYYY-MM-DD
</capability>
</yang-module-capabilities>

Comment by Sachi Gupta [ 27/Aug/14 ]

Hi Maros,
Are you suggesting to put the capability as added in the last line of attached file(snapshot.txt) of 99-netconf-connector.xml?
i have added configuration@1970-01-01.yang file in cache/schema folder but getting the exception

Exception:
2014-08-27 19:07:14.139 IST [config-pusher] ERROR o.o.c.l.b.i.UncaughtExceptionPolicy - Thread Thread[config-pusher,5,main] died because of an uncaught exception
java.lang.IllegalStateException: Max wait for capabilities reached.Not enough capabilities for ConfigSnapshot

{configSnapshot='<data xmlns="urn:ietf:params:xml:ns:netconf:base:1.0"><modules xmlns="urn:opendaylight:params:xml:ns:yang:controller:config"><module><type xmlns:prefix="urn:opendaylight:params:xml:ns:yang:controller:md:sal:connector:netconf">prefix:sal-netconf-connector</type><name>sachi</name><address xmlns="urn:opendaylight:params:xml:ns:yang:controller:md:sal:connector:netconf">1.1.1.7</address><port xmlns="urn:opendaylight:params:xml:ns:yang:controller:md:sal:connector:netconf">830</port><username xmlns="urn:opendaylight:params:xml:ns:yang:controller:md:sal:connector:netconf">tcs</username><password xmlns="urn:opendaylight:params:xml:ns:yang:controller:md:sal:connector:netconf">tcs@1234</password><tcp-only xmlns="urn:opendaylight:params:xml:ns:yang:controller:md:sal:connector:netconf">false</tcp-only><event-executor xmlns="urn:opendaylight:params:xml:ns:yang:controller:md:sal:connector:netconf"><type xmlns:prefix="urn:opendaylight:params:xml:ns:yang:controller:netty">prefix:netty-event-executor</type><name>global-event-executor</name> </event-executor><binding-registry xmlns="urn:opendaylight:params:xml:ns:yang:controller:md:sal:connector:netconf"><type xmlns:prefix="urn:opendaylight:params:xml:ns:yang:controller:md:sal:binding">prefix:binding-broker-osgi-registry</type><name>binding-osgi-broker</name> </binding-registry><dom-registry xmlns="urn:opendaylight:params:xml:ns:yang:controller:md:sal:connector:netconf"><type xmlns:prefix="urn:opendaylight:params:xml:ns:yang:controller:md:sal:dom">prefix:dom-broker-osgi-registry</type><name>dom-broker</name> </dom-registry><client-dispatcher xmlns="urn:opendaylight:params:xml:ns:yang:controller:md:sal:connector:netconf"><type xmlns:prefix="urn:opendaylight:params:xml:ns:yang:controller:config:netconf">prefix:netconf-client-dispatcher</type><name>global-netconf-dispatcher</name> </client-dispatcher><processing-executor xmlns="urn:opendaylight:params:xml:ns:yang:controller:md:sal:connector:netconf"><type xmlns:prefix="urn:opendaylight:params:xml:ns:yang:controller:threadpool">prefix:threadpool</type><name>global-netconf-processing-executor</name> </processing-executor> </module> </modules> </data> ', capabilities=[http://yang.juniper.net/yang/1.1/jc?module=configuration&revision=1970-01-01, urn:opendaylight:params:xml:ns:yang:controller:md:sal:connector:netconf?module=odl-sal-netconf-connector- cfg&revision=2013-10-28]}

. Expected but not found: [urn:opendaylight:params:xml:ns:yang:controller:md:sal:connector:netconf?module=odl-sal-netconf-connector- cfg&revision=2013-10-28, http://yang.juniper.net/yang/1.1/jc?module=configuration&revision=1970-01-01]
at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.getOperationServiceWithRetries(ConfigPusherImpl.java:158) ~[na:na]
at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.pushConfigWithConflictingVersionRetries(ConfigPusherImpl.java:134) ~[na:na]
at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.internalPushConfigs(ConfigPusherImpl.java:111) ~[na:na]
at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.process(ConfigPusherImpl.java:78) ~[na:na]
at org.opendaylight.controller.netconf.persist.impl.osgi.ConfigPersisterActivator$InnerCustomizer$1.run(ConfigPersisterActivator.java:177) ~[na:na]
at java.lang.Thread.run(Thread.java:744) ~[na:1.7.0_55]
Caused by: org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl$NotEnoughCapabilitiesException: Not enough capabilities for ConfigSnapshot

{configSnapshot='<data xmlns="urn:ietf:params:xml:ns:netconf:base:1.0"><modules xmlns="urn:opendaylight:params:xml:ns:yang:controller:config"><module><type xmlns:prefix="urn:opendaylight:params:xml:ns:yang:controller:md:sal:connector:netconf">prefix:sal-netconf-connector</type><name>sachi</name><address xmlns="urn:opendaylight:params:xml:ns:yang:controller:md:sal:connector:netconf">1.1.1.7</address><port xmlns="urn:opendaylight:params:xml:ns:yang:controller:md:sal:connector:netconf">830</port><username xmlns="urn:opendaylight:params:xml:ns:yang:controller:md:sal:connector:netconf">tcs</username><password xmlns="urn:opendaylight:params:xml:ns:yang:controller:md:sal:connector:netconf">tcs@1234</password><tcp-only xmlns="urn:opendaylight:params:xml:ns:yang:controller:md:sal:connector:netconf">false</tcp-only><event-executor xmlns="urn:opendaylight:params:xml:ns:yang:controller:md:sal:connector:netconf"><type xmlns:prefix="urn:opendaylight:params:xml:ns:yang:controller:netty">prefix:netty-event-executor</type><name>global-event-executor</name> </event-executor><binding-registry xmlns="urn:opendaylight:params:xml:ns:yang:controller:md:sal:connector:netconf"><type xmlns:prefix="urn:opendaylight:params:xml:ns:yang:controller:md:sal:binding">prefix:binding-broker-osgi-registry</type><name>binding-osgi-broker</name> </binding-registry><dom-registry xmlns="urn:opendaylight:params:xml:ns:yang:controller:md:sal:connector:netconf"><type xmlns:prefix="urn:opendaylight:params:xml:ns:yang:controller:md:sal:dom">prefix:dom-broker-osgi-registry</type><name>dom-broker</name> </dom-registry><client-dispatcher xmlns="urn:opendaylight:params:xml:ns:yang:controller:md:sal:connector:netconf"><type xmlns:prefix="urn:opendaylight:params:xml:ns:yang:controller:config:netconf">prefix:netconf-client-dispatcher</type><name>global-netconf-dispatcher</name> </client-dispatcher><processing-executor xmlns="urn:opendaylight:params:xml:ns:yang:controller:md:sal:connector:netconf"><type xmlns:prefix="urn:opendaylight:params:xml:ns:yang:controller:threadpool">prefix:threadpool</type><name>global-netconf-processing-executor</name> </processing-executor> </module> </modules> </data> ', capabilities=[http://yang.juniper.net/yang/1.1/jc?module=configuration&revision=1970-01-01, urn:opendaylight:params:xml:ns:yang:controller:md:sal:connector:netconf?module=odl-sal-netconf-connector- cfg&revision=2013-10-28]}

. Expected but not found: [urn:opendaylight:params:xml:ns:yang:controller:md:sal:connector:netconf?module=odl-sal-netconf-connector- cfg&revision=2013-10-28, http://yang.juniper.net/yang/1.1/jc?module=configuration&revision=1970-01-01]
at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.getOperationService(ConfigPusherImpl.java:196) ~[na:na]
at org.opendaylight.controller.netconf.persist.impl.ConfigPusherImpl.getOperationServiceWithRetries(ConfigPusherImpl.java:151) ~[na:na]
... 5 common frames omitted

and if i add the <yang-module-capabilities>
<capability>
http://yang.juniper.net/yang/1.1/jc?module=configuration?revision=YYYY-MM-DD
</capability>
</yang-module-capabilities> in required capabilities or after the required one's i am getting xml exception.
Kindly guide me how can i move ahead with correct configuration.

Thanks in advance.

Comment by Sachi Gupta [ 27/Aug/14 ]

Attachment snapshot.txt has been added with description: netconf-connector snapshot

Comment by Sachi Gupta [ 02/Sep/14 ]

Hi Maros,

I have hard coded the capabilities in NetconfClientSession.java file(as mentioned in CONTROLLER-256) after which we don't need to add yang capabilities in xml explicitly and it skips the above mentioned exception.

After doing this also, I am facing the same error of "Sal Initialization Failure"

2014-09-02 11:16:36.317 IST [remote-connector-processing-executor-2] WARN o.o.c.s.c.n.NetconfStateSchemas - RemoteDevice

{sachi}: Unable to detect available schemas, get to (urn:ietf:params:xml:ns:yang:ietf-netconf-monitoring?revision=2010-10-04)netconf-state/(urn:ietf:params:xml:ns:yang:ietf-netconf-monitoring?revision=2010-10-04)schemas was empty
2014-09-02 11:16:36.319 IST [remote-connector-processing-executor-3] INFO o.o.c.s.c.netconf.NetconfDevice - schema required resources
2014-09-02 11:16:36.324 IST [remote-connector-processing-executor-3] ERROR o.o.c.s.c.netconf.NetconfDevice - RemoteDevice{sachi}

: Initialization in sal failed, disconnecting from device
java.lang.IllegalStateException: RemoteDevice

{sachi}: No more sources for schema context
at org.opendaylight.controller.sal.connect.netconf.NetconfDevice$RecursiveSchemaSetup.setUpSchema(NetconfDevice.java:306) [bundlefile:na]
at org.opendaylight.controller.sal.connect.netconf.NetconfDevice$RecursiveSchemaSetup.run(NetconfDevice.java:295) [bundlefile:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_55]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_55]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_55]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_55]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_55]
2014-09-02 11:16:36.325 IST [remote-connector-processing-executor-3] WARN o.o.c.s.c.n.l.NetconfDeviceCommunicator - RemoteDevice{sachi}

: Session terminated Session closed

Comment by Maros Marsalek [ 02/Sep/14 ]

Ok, so you have file configuration@1970-01-01.yang in cache/schema. Does the file contain revision statement with 1970-01-01 revision ?

If so, adding this configuration under your netconf-connector:

<yang-module-capabilities>
<capability>
http://yang.juniper.net/yang/1.1/jc?module=configuration?revision=1970-01-01
</capability>
</yang-module-capabilities>

should do the trick.

Also are you running latest controller ?

Comment by Sachi Gupta [ 02/Sep/14 ]

Hi Maros,

1. Yes, configuration@1970-01-01.yang in cache/schema contains 1970-01-01 revision statement as below:

module configuration {
namespace "http://yang.juniper.net/yang/1.1/jc";
prefix jc;

revision 1970-01-01

{ description "Initial revision."; }

2. Added configuration under your netconf-connector as below:

<required-capabilities> <capability>urn:opendaylight:params:xml:ns:yang:controller:md:sal:connector:netconf?module=odl-sal-netconf-connector-cfg&revision=2013-10-28</capability>
</required-capabilities>
<yang-module-capabilities>
<capability>
http://yang.juniper.net/yang/1.1/jc?module=configuration&amp;revision=1970-01-01
</capability>
</yang-module-capabilities>

3. Yes, I am running the new controller code cloned on Sept-02.

After doing all the changes, its not working.
Please suggest.

Thanks in advance
Sachi

Comment by Maros Marsalek [ 02/Sep/14 ]

This xml:

<yang-module-capabilities>
<capability>
http://yang.juniper.net/yang/1.1/jc?module=configuration&amp;revision=1970-01-01
</capability>
</yang-module-capabilities>

Needs to be under module element. It is part of netconf-connector configuration.

Comment by Sachi Gupta [ 02/Sep/14 ]

Hi Maros,

Have added the above as part of netconf-connector configuration.

Now, I am not getting any error and device is not getting initialized also.

Anything else I need to add.

Thanks

Comment by Maros Marsalek [ 02/Sep/14 ]

Could you please rerun with detailed logs ?

Add these lines to configuration/logback.xml before you start ODL. Delete logs/ folder and rerun:

<logger name="org.opendaylight.protocol.framework" level="TRACE"/>
<logger name="org.opendaylight.controller.netconf" level="TRACE"/>
<logger name="org.opendaylight.controller.sal.connect.netconf" level="TRACE"/>
<logger name="org.opendaylight.yangtools.yang.model.repo.util" level="TRACE"/>
<logger name="org.opendaylight.yangtools.yang.parser.repo" level="TRACE"/>

And please attach produced opendaylight.log file afterwards.

And what do you mean by not getting initialized ? There is nothing under nodes/node from restconf ? or just no data are transfered ?

Comment by Sachi Gupta [ 02/Sep/14 ]

Send the logs on mmarsale@cisco.com.

The size of the file was larger so was not able to attach it on bugzilla.

The log says "RemoteDevice

{controller-config}

: Netconf connector initialized successfully" but it does not show any log for my "sachi-m10" netconf connector.

Generated at Wed Feb 07 19:53:42 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.