Operating System: All
Platform: All

Please can you attach whole log or stack trace. Thanks.

Attachment karaf.log has been added with description: logs (with initial mounting deleted)

Hi Giles,

The problem seems to be that your device does not conform to provided models.

Possible workaround:
You can modify models to conform to data that comes from device and then side-load this modified models.

What can be done in netconf, is that we can distinguish RPC failures like this one from RPC failures like non-reachable device and fail only on latter ones. But if your device sends you bad data, something else will eventually fail, e.g RESTCONF, RPCService, etc.

Hi Jakub,

yes - I'm well aware that the device doesn't conform to the models.

what we're looking for in this instance is simply to ignore cases where devices return content that isn't in the model. That way we can still see the rest of the config, and can still write to the device.

Giles

So you just want NETCONF not to reconnect on RPC failures like this one or you want NETCONF to do less strict parsing and just ignore non-conforming data overall?

Hi Jakub

in this case (where the device returns data that's not in the model) I'd want ODL just to ignore that data (so parse the rest of the data and present it through the Java and RESTCONF APIs).

Giles

We can make something with this reconnecting stuff, but I don't know if we can and want to allow less strict parsing and ignoring non-conforming data. I think this should be discussed thoroughly

well - Postel's Law would tend to suggest ignoring it is the right thing to do. But sure, we have to check it won't break anything...

Ignoring invalid data is not something we can do on netconf level, all parsing is handled by yangtools.

sure - though there's a "strict" switch that NETCONF can set (or unset) when having yangtools parse NETCONF modules. I tried setting it to false but it still didn't work - so yeah maybe needs fixing in Yangtools.

Guess you are talking about NetconfMessageTransformer and its paramater strictParsing.

What could be done in NETCONF is to add configuration option for mountpoints, whether to use strict parsing or don't. But if you have tested this and it does not work, this should be consulted with YANGTOOLS developers. So maybe we should move this bug back to YANGTOOLS project and you can add report what you have observed during testing this feature.

Also we should open a new bug for adding new configuration option for mountpoints in Netconf project.

Ok my bad i forgotten about the switch, what was the issue when you tried to hardcode it to false? Did it still fail with unknown child nodes?

Yes - I think the failure was exactly the same. Maybe I got something wrong when I built it?

I am moving this bug back to YANGTOOLS project. I think now it is clear, what the problem is.

The problem should be in DomToNormalizedNodeParserFactory's strictParsing switch is not working correctly and is not documented properly.

are we still expecting a fix for this in Carbon? I'm seeing issues with this with Boron-SR1 and XR 6.1.2, and a colleague is seeing the same with Beryllium-SR3 and XR 6.0.1.

Hello Giles,

yes, the fix should be done in Carbon. Alas I cannot say when exactly will that be as we are currently busy with other issues.

thanks Igor.

I've bumped the importance up to "major" at any rate - this is blocking various customers.

Giles

Hello Giles, I tried parsing xml data containing nodes not defined in the YANG model with the DomToNormalizedNodeParserFactory's strictParsing flag set to false and it worked correctly. No exception was thrown. I tested this in stable/beryllium, stable/boron and current master.

Could you please try testing this one more time ?

Hi Giles, if the problem still occurs, it would be really helpful if you provided the yang modules and yang data that you used when the exception was thrown. I think that the actual problem might be something different than the strictParsing flag. In fact, the code flow does not even come to the part with the exception when strictParsing is set to false.

Hi Igor,

I think the issue may be to do with where the node is in the hierarchy (it's a list entry).

the stack trace starts with:

2017-03-08 07:13:06,069 | WARN | oupCloseable-3-4 | ReadOnlyTx | 190 - org.opendaylight.netconf.sal-netconf-connector - 1.4.3.SNAPSHOT | RemoteDevice

: Reading data failed
java.lang.IllegalStateException: Unknown child(ren) node(s) detected, identified by: (http://openconfig.net/yang/interfaces)ip, in: list address
at com.google.common.base.Preconditions.checkState(Preconditions.java:197)[41:com.google.guava:18.0.0]
at org.opendaylight.yangtools.yang.data.impl.schema.SchemaUtils.findSchemaForChild(SchemaUtils.java:105)[57:org.opendaylight.yangtools.yang-data-impl:1.0.3.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.SchemaUtils.findSchemaForChild(SchemaUtils.java:87)[57:org.opendaylight.yangtools.yang-data-impl:1.0.3.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.SchemaUtils.findSchemaForChild(SchemaUtils.java:93)[57:org.opendaylight.yangtools.yang-data-impl:1.0.3.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.ListEntryNodeBaseParser.getSchemaForChild(ListEntryNodeBaseParser.java:45)[57:org.opendaylight.yangtools.yang-data-impl:1.0.3.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.ListEntryNodeBaseParser.getSchemaForChild(ListEntryNodeBaseParser.java:28)[57:org.opendaylight.yangtools.yang-data-impl:1.0.3.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.BaseDispatcherParser.parse(BaseDispatcherParser.java:160)[57:org.opendaylight.yangtools.yang-data-impl:1.0.3.SNAPSHOT]

ListEntryNodeBaseParser.java extends BaseDispatcherParser.java. In BaseDispatcherParser.java. but BaseDispatcherParser has a protected method strictParsing() that always returns "true".

i'll attach both the YANG model and the ODL log (starting with the instance data).

Yes, but in case of parsing a list entry, the strictParsing() method is actually called in the subclass MapEntryNodeDomParser which overrides it. I have tested this on list entries too and it worked as expected.

ah ok.

well at any rate this isn't behaving as expected - I'm certainly seeing "strict" parsing here. Is there any way to configure ODL not to be strict - or do i have to edit the code? Or do you think the strict flag isn't the issue here?

Giles

Does the problem come up even if you edit the strictParsing flag in the NetconfMessageTransformer's constructor (line 88) to false ?

this is in the second constructor (the one with 3 parameters?) I have that at line 89:
parserFactory = DomToNormalizedNodeParserFactory.getInstance(XmlUtils.DEFAULT_XML_CODEC_PROVIDER, schemaContext, strictParsing);

So change that to:

parserFactory = DomToNormalizedNodeParserFactory.getInstance(XmlUtils.DEFAULT_XML_CODEC_PROVIDER, schemaContext, strictParsing);

correct?

if I edit that the config get still fails but with a different error:

{
"errors": {
"error": [
{
"error-type": "application",
"error-tag": "operation-failed",
"error-message": "read execution failed",
"error-info": "org.opendaylight.yangtools.yang.data.impl.schema.builder.impl.valid.DataValidationException$IllegalListKeyException: Key value not present for key: (http://openconfig.net/yang/interfaces/ip?revision=2015-11-20)ip, in: (http://openconfig.net/yang/interfaces/ip?revision=2015-11-20)address[{}]\n\tat org.opendaylight.yangtools.yang.data.impl.schema.builder.impl.valid.DataValidationException.checkListKey(DataValidationException.java:74)\n\tat org.opendaylight.yangtools.yang.data.impl.schema.builder.impl.ImmutableMapEntryNodeSchemaAwareBuilder.constructNodeIdentifier(ImmutableMapEntryNodeSchemaAwareBuilder.java:63)\n\tat org.opendaylight.yangtools.yang.data.impl.schema.builder.impl.ImmutableMapEntryNodeSchemaAwareBuilder.build(ImmutableMapEntryNodeSchemaAwareBuilder.java:46)\n\tat org.opendaylight.yangtools.yang.data.impl.schema.builder.impl.ImmutableMapEntryNodeSchemaAwareBuilder.build(ImmutableMapEntryNodeSchemaAwareBuilder.java:23)\n\tat org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.BaseDispatcherParser$SimpleBuildingStrategy.build(BaseDispatcherParser.java:238)\n\tat org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.BaseDispatcherParser$SimpleBuildingStrategy.build(BaseDispatcherParser.java:235)\n\tat org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.BaseDispatcherParser.parse(BaseDispatcherParser.java:204)\n\tat org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.ListEntryNodeBaseParser.parse(ListEntryNodeBaseParser.java:61)\n\tat org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.ListEntryNodeBaseParser.parse(ListEntryNodeBaseParser.java:28)\n\tat org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.ListNodeBaseParser.parse(ListNodeBaseParser.java:46)\n\tat org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.ListNodeBaseParser.parse(ListNodeBaseParser.java:26)\n\tat org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.NodeParserDispatcher$BaseNodeParserDispatcher.dispatchChildElement(NodeParserDispatcher.java:61)\n\tat org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.BaseDispatcherParser.parse(BaseDispatcherParser.java:178)\n\tat org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.ContainerNodeBaseParser.parse(ContainerNodeBaseParser.java:47)\n\tat org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.ContainerNodeBaseParser.parse(ContainerNodeBaseParser.java:29)\n\tat org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.NodeParserDispatcher$BaseNodeParserDispatcher.dispatchChildElement(NodeParserDispatcher.java:49)\n\tat org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.BaseDispatcherParser.parse(BaseDispatcherParser.java:178)\n\tat org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.BaseDispatcherParser.parse(BaseDispatcherParser.java:34)\n\tat org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.NodeParserDispatcher$BaseNodeParserDispatcher.dispatchChildElement(NodeParserDispatcher.java:66)\n\tat org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.BaseDispatcherParser.parse(BaseDispatcherParser.java:198)\n\tat org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.ListEntryNodeBaseParser.parse(ListEntryNodeBaseParser.java:61)\n\tat org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.ListEntryNodeBaseParser.parse(ListEntryNodeBaseParser.java:28)\n\tat org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.ListNodeBaseParser.parse(ListNodeBaseParser.java:46)\n\tat org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.ListNodeBaseParser.parse(ListNodeBaseParser.java:26)\n\tat org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.NodeParserDispatcher$BaseNodeParserDispatcher.dispatchChildElement(NodeParserDispatcher.java:61)\n\tat org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.BaseDispatcherParser.parse(BaseDispatcherParser.java:178)\n\tat org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.ContainerNodeBaseParser.parse(ContainerNodeBaseParser.java:47)\n\tat org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.ContainerNodeBaseParser.parse(ContainerNodeBaseParser.java:29)\n\tat org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.NodeParserDispatcher$BaseNodeParserDispatcher.dispatchChildElement(NodeParserDispatcher.java:49)\n\tat org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.BaseDispatcherParser.parse(BaseDispatcherParser.java:178)\n\tat org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.ListEntryNodeBaseParser.parse(ListEntryNodeBaseParser.java:61)\n\tat org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.ListEntryNodeBaseParser.parse(ListEntryNodeBaseParser.java:28)\n\tat org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.ListNodeBaseParser.parse(ListNodeBaseParser.java:46)\n\tat org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.ListNodeBaseParser.parse(ListNodeBaseParser.java:26)\n\tat org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.NodeParserDispatcher$BaseNodeParserDispatcher.dispatchChildElement(NodeParserDispatcher.java:61)\n\tat org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.BaseDispatcherParser.parse(BaseDispatcherParser.java:178)\n\tat org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.ContainerNodeBaseParser.parse(ContainerNodeBaseParser.java:47)\n\tat org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.ContainerNodeBaseParser.parse(ContainerNodeBaseParser.java:29)\n\tat org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.NodeParserDispatcher$BaseNodeParserDispatcher.dispatchChildElement(NodeParserDispatcher.java:49)\n\tat org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.BaseDispatcherParser.parse(BaseDispatcherParser.java:178)\n\tat org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.ContainerNodeBaseParser.parse(ContainerNodeBaseParser.java:47)\n\tat org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.ContainerNodeBaseParser.parse(ContainerNodeBaseParser.java:29)\n\tat org.opendaylight.netconf.sal.connect.netconf.schema.mapping.NetconfMessageTransformer.toRpcResult(NetconfMessageTransformer.java:186)\n\tat org.opendaylight.netconf.sal.connect.netconf.schema.mapping.NetconfMessageTransformer.toRpcResult(NetconfMessageTransformer.java:57)\n\tat org.opendaylight.netconf.sal.connect.netconf.sal.NetconfDeviceRpc$2.apply(NetconfDeviceRpc.java:68)\n\tat org.opendaylight.netconf.sal.connect.netconf.sal.NetconfDeviceRpc$2.apply(NetconfDeviceRpc.java:64)\n\tat com.google.common.util.concurrent.Futures$2.apply(Futures.java:760)\n\tat com.google.common.util.concurrent.Futures$ChainingListenableFuture.run(Futures.java:906)\n\tat com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)\n\tat com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)\n\tat com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)\n\tat com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:185)\n\tat org.opendaylight.netconf.sal.connect.netconf.listener.UncancellableFuture.set(UncancellableFuture.java:44)\n\tat org.opendaylight.netconf.sal.connect.netconf.listener.NetconfDeviceCommunicator.processMessage(NetconfDeviceCommunicator.java:325)\n\tat org.opendaylight.netconf.sal.connect.netconf.listener.NetconfDeviceCommunicator.onMessage(NetconfDeviceCommunicator.java:262)\n\tat org.opendaylight.netconf.sal.connect.netconf.listener.NetconfDeviceCommunicator.onMessage(NetconfDeviceCommunicator.java:49)\n\tat org.opendaylight.netconf.nettyutil.AbstractNetconfSession.handleMessage(AbstractNetconfSession.java:64)\n\tat org.opendaylight.netconf.nettyutil.AbstractNetconfSession.handleMessage(AbstractNetconfSession.java:35)\n\tat org.opendaylight.protocol.framework.AbstractProtocolSession.channelRead0(AbstractProtocolSession.java:53)\n\tat io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:343)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:336)\n\tat io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293)\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:343)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:336)\n\tat io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293)\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357)\n\tat io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35)\n\tat io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:348)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399)\n\tat io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:446)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)\n\tat io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)\n\tat java.lang.Thread.run(Thread.java:745)\n"
}
]
}
}

Yes, correct. Now, regarding the new exception - basically, it means that you have invalid data. A keyed list entry must have the key leaf data present.
I would like to have a look at the yang model and the xml data. Would it be possible to attach them here ?

Attachment openconfig-interfaces@2015-11-20.yang has been added with description: YANG model with missing instance data

these are correct - the previous log was the wrong file

Attachment short.log has been added with description: logs (with initial mounting deleted)

Cut-down Logs (starting with the instance data) and YANG model attached now

I managed to replicate the original problem you had reported, i.e. getting the exception because of unknown nodes even if strictParsing was set to false.
This problem occurs when parsing an unkeyed list entry.

The following patch should fix the issue - https://git.opendaylight.org/gerrit/#/c/53149/.

Could you please test it again using the patch ?

boron: https://git.opendaylight.org/gerrit/53174

I'm still seeing the problem with the patch applied...

maybe we can take a look tomorrow?

Are you seeing the problem with unknown child nodes or the one with DataValidationException$IllegalListKeyException ?

still seeing:

{
"errors": {
"error": [
{
"error-type": "application",
"error-tag": "operation-failed",
"error-message": "read execution failed",
"error-info": "java.lang.IllegalStateException: Unknown child(ren) node(s) detected, identified by: (http://openconfig.net/yang/interfaces)ip, in: list address\n\tat com.google.common.base.Preconditions.checkState(Preconditions.java:197)\n\tat org.opendaylight.yangtools.yang.data.impl.schema.SchemaUtils.findSchemaForChild(SchemaUtils.java:105...

Ok, so does this exception still occur even if the strictParsing in the second constructor of NetconfMessageTransformer is hard-coded to false ?

ah ok. So if I change that .jar too then I get:

{
"errors": {
"error": [
{
"error-type": "application",
"error-tag": "operation-failed",
"error-message": "read execution failed",
"error-info": "org.opendaylight.yangtools.yang.data.impl.schema.builder.impl.valid.DataValidationException$IllegalListKeyException: Key value not present for key: (http://openconfig.net/yang/interfaces/ip?revision=2015-11-20)ip, in: (http://openconfig.net/yang/interfaces/ip?revision=2015-11-20
...

So I think the offender is:

<subinterface>
<index>0</index>
<ipv4 xmlns="http://openconfig.net/yang/interfaces/ip">
<address>
<ip>192.168.4.2</ip>
<config>
<ip>192.168.4.2</ip>
<prefix-length>24</prefix-length>
</config>
</address>
</ipv4>
<ipv6 xmlns="http://openconfig.net/yang/interfaces/ip">
<address>
<ip xmlns="http://openconfig.net/yang/interfaces">2001::2</ip>
<config>
<ip>2001::2</ip>
<prefix-length>64</prefix-length>
</config>
</address>
</ipv6>
</subinterface>

note the difference in ipv4 and ipv6's <ip> member. Both are defined to point to the same entity:
type leafref

but their encoded namespaces differ. There is no 'ip' defined in openconfig-interfaces.yang, so the ipv6 case is being encoded badly by the device.

The only recovery we can attempt here is an exhaustive enumeration of data and making wild assumptions about data interpretation – i.e. that a leaf which is not defined in the schema and present can be used as a substitute for the key. That is a very very bad assumption to make.

Giles, this really needs to be escalated to the vendor team.

good catch

So I just tested that by removing all IPv6 interface config from the device, and that worked.

I can raise the bug with the device guys.

But at any rate I guess the question in such cases is do we:

1) just ignore the instance data we don't understand

2) throw an error.

You know my preference on that one

Giles