[YANGTOOLS-587] Parser: XSD regular expressions are interpreted as Java regexes Created: 24/Feb/16  Updated: 10/Apr/22  Resolved: 20/Jul/17

Status: Resolved
Project: yangtools
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug
Reporter: Bhavesh Kumar Assignee: Peter Kajsa
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: Linux
Platform: PC


Attachments: Text File karaf.log    
Issue Links:
Blocks
blocks YANGTOOLS-658 Escape Meta-characters like $, ^ in R... Resolved
Duplicate
is duplicated by YANGTOOLS-610 Parser error with regex Resolved
is duplicated by YANGTOOLS-658 Escape Meta-characters like $, ^ in R... Resolved
External issue ID: 5410
Priority: High

 Description   

I am using integration(stable/lithium), have registered a netconf device (device1) successfully.
When I try to mount the device it throws error as below :
http://127.0.0.1:8181/restconf/config/opendaylight-inventory:nodes/node/device1/yang-ext:mount/

applicationoperation-failedProblem to get data from transaction.ReadFailedException{message=read execution failed, errorList=[RpcError [message=read execution failed, severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=java.lang.IllegalArgumentException: Value $6$AnrKGc0V$B/0/A.pWg4HrrA6YiEJOtFGibQ9Fmm5.4rI/00gEz3QeB7joSxBU3YtbHDm6NSkS1dKTQy3BWhwKKDS8nB5S// does not match regular expression <^^$0$.*|$1$[a-zA-Z0-9./]

{1,8}$[a-zA-Z0-9./]{22}|$5$(rounds=\d+$)?[a-zA-Z0-9./]{1,16}$[a-zA-Z0-9./]{43}|$6$(rounds=\d+$)?[a-zA-Z0-9./]{1,16}$[a-zA-Z0-9./]{86}$$>]]} at org.opendaylight.controller.md.sal.common.api.data.ReadFailedException$1.newWithCause(ReadFailedException.java:26) at org.opendaylight.controller.md.sal.common.api.data.ReadFailedException$1.newWithCause(ReadFailedException.java:23) at org.opendaylight.yangtools.util.concurrent.ExceptionMapper.apply(ExceptionMapper.java:80) at org.opendaylight.yangtools.util.concurrent.ExceptionMapper.apply(ExceptionMapper.java:31) at org.opendaylight.yangtools.util.concurrent.MappingCheckedFuture.mapException(MappingCheckedFuture.java:60) at org.opendaylight.yangtools.util.concurrent.MappingCheckedFuture.wrapInExecutionException(MappingCheckedFuture.java:64) at org.opendaylight.yangtools.util.concurrent.MappingCheckedFuture.get(MappingCheckedFuture.java:77) at org.opendaylight.controller.sal.restconf.impl.BrokerFacade.readDataViaTransaction(BrokerFacade.java:198) at org.opendaylight.controller.sal.restconf.impl.BrokerFacade.readConfigurationData(BrokerFacade.java:87) at org.opendaylight.controller.sal.restconf.impl.RestconfImpl.readConfigurationData(RestconfImpl.java:658) at org.opendaylight.controller.sal.restconf.impl.StatisticsRestconfServiceWrapper.readConfigurationData(StatisticsRestconfServiceWrapper.java:95) at org.opendaylight.controller.sal.rest.impl.RestconfCompositeWrapper.readConfigurationData(RestconfCompositeWrapper.java:64) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185) at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.ResourceObjectRule.accept(ResourceObjectRule.java:100) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1511) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1442) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1391) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1381) at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:538) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:716) at javax.servlet.http.HttpServlet.service(HttpServlet.java:668) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1496) at org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:247) at org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:210) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467) at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:82) at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:294) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501) at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:69) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:240) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:429) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:75) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.eclipse.jetty.server.Server.handle(Server.java:370) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494) at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.IllegalArgumentException: Value $6$AnrKGc0V$B/0/A.pWg4HrrA6YiEJOtFGibQ9Fmm5.4rI/00gEz3QeB7joSxBU3YtbHDm6NSkS1dKTQy3BWhwKKDS8nB5S// does not match regular expression <^^$0$.*|$1$[a-zA-Z0-9./]{1,8}

$[a-zA-Z0-9./]

{22}

|$5$(rounds=\d+$)?[a-zA-Z0-9./]

{1,16}$[a-zA-Z0-9./]{43}|$6$(rounds=\d+$)?[a-zA-Z0-9./]{1,16}

$[a-zA-Z0-9./]

{86}

$$> at com.google.common.base.Preconditions.checkArgument(Preconditions.java:145) at org.opendaylight.yangtools.yang.data.impl.codec.CompiledPatternContext.validate(CompiledPatternContext.java:31) at org.opendaylight.yangtools.yang.data.impl.codec.StringPatternCheckingCodec.validate(StringPatternCheckingCodec.java:39) at org.opendaylight.yangtools.yang.data.impl.codec.StringStringCodec.deserialize(StringStringCodec.java:36) at org.opendaylight.yangtools.yang.data.impl.codec.StringStringCodec.deserialize(StringStringCodec.java:14) at org.opendaylight.yangtools.yang.data.impl.schema.transform.dom.DomUtils.parseXmlValue(DomUtils.java:58) at org.opendaylight.yangtools.yang.data.impl.schema.transform.dom.DomUtils.parseXmlValue(DomUtils.java:136) at org.opendaylight.yangtools.yang.data.impl.schema.transform.dom.parser.LeafNodeDomParser.parseLeaf(LeafNodeDomParser.java:45) at org.opendaylight.yangtools.yang.data.impl.schema.transform.dom.parser.LeafNodeDomParser.parseLeaf(LeafNodeDomParser.java:22) at org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.LeafNodeBaseParser.parse(LeafNodeBaseParser.java:46) at org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.LeafNodeBaseParser.parse(LeafNodeBaseParser.java:27) at org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.NodeParserDispatcher$BaseNodeParserDispatcher.dispatchChildElement(NodeParserDispatcher.java:50) at org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.BaseDispatcherParser.parse(BaseDispatcherParser.java:177) at org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.ListEntryNodeBaseParser.parse(ListEntryNodeBaseParser.java:61) at org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.ListEntryNodeBaseParser.parse(ListEntryNodeBaseParser.java:28) at org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.ListNodeBaseParser.parse(ListNodeBaseParser.java:46) at org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.ListNodeBaseParser.parse(ListNodeBaseParser.java:26) at org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.NodeParserDispatcher$BaseNodeParserDispatcher.dispatchChildElement(NodeParserDispatcher.java:60) at org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.BaseDispatcherParser.parse(BaseDispatcherParser.java:177) at org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.ContainerNodeBaseParser.parse(ContainerNodeBaseParser.java:47) at org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.ContainerNodeBaseParser.parse(ContainerNodeBaseParser.java:29) at org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.NodeParserDispatcher$BaseNodeParserDispatcher.dispatchChildElement(NodeParserDispatcher.java:48) at org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.BaseDispatcherParser.parse(BaseDispatcherParser.java:177) at org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.ContainerNodeBaseParser.parse(ContainerNodeBaseParser.java:47) at org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.ContainerNodeBaseParser.parse(ContainerNodeBaseParser.java:29) at org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.NodeParserDispatcher$BaseNodeParserDispatcher.dispatchChildElement(NodeParserDispatcher.java:48) at org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.BaseDispatcherParser.parse(BaseDispatcherParser.java:177) at org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.ContainerNodeBaseParser.parse(ContainerNodeBaseParser.java:47) at org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.ContainerNodeBaseParser.parse(ContainerNodeBaseParser.java:29) at org.opendaylight.controller.sal.connect.netconf.schema.mapping.NetconfMessageTransformer.toRpcResult(NetconfMessageTransformer.java:271) at org.opendaylight.controller.sal.connect.netconf.schema.mapping.NetconfMessageTransformer.toRpcResult(NetconfMessageTransformer.java:71) at org.opendaylight.controller.sal.connect.netconf.sal.NetconfDeviceRpc$2.apply(NetconfDeviceRpc.java:69) at org.opendaylight.controller.sal.connect.netconf.sal.NetconfDeviceRpc$2.apply(NetconfDeviceRpc.java:65) at com.google.common.util.concurrent.Futures$2.apply(Futures.java:760) at com.google.common.util.concurrent.Futures$ChainingListenableFuture.run(Futures.java:906) at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457) at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:185) at org.opendaylight.controller.sal.connect.netconf.listener.UncancellableFuture.set(UncancellableFuture.java:45) at org.opendaylight.controller.sal.connect.netconf.listener.NetconfDeviceCommunicator.processMessage(NetconfDeviceCommunicator.java:268) at org.opendaylight.controller.sal.connect.netconf.listener.NetconfDeviceCommunicator.onMessage(NetconfDeviceCommunicator.java:213) at org.opendaylight.controller.sal.connect.netconf.listener.NetconfDeviceCommunicator.onMessage(NetconfDeviceCommunicator.java:46) at org.opendaylight.controller.netconf.nettyutil.AbstractNetconfSession.handleMessage(AbstractNetconfSession.java:61) at org.opendaylight.controller.netconf.nettyutil.AbstractNetconfSession.handleMessage(AbstractNetconfSession.java:32) at org.opendaylight.protocol.framework.AbstractProtocolSession.channelRead0(AbstractProtocolSession.java:53) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:32) at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:329) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) ... 1 more

Logs attached.



 Comments   
Comment by Bhavesh Kumar [ 24/Feb/16 ]

Attachment karaf.log has been added with description: karaf log of integration while mounting the device

Comment by Robert Varga [ 24/Feb/16 ]

Input YANG file contains invalid regular expression, hence the error. Note that YANG follows XSD regular expressions, as defined in https://www.w3.org/TR/2004/REC-xmlschema-2-20041028/#regexs. These do not allow ^/$ anchors (as anchoring is implicit) – which are obviously present.

Error on invalid input, downgrading to improvement/normal, targetting to SR4. What yangtools can do is catch the case when an expression starts/ends with an anchor, complain loudly and ignore the anchor points.

Comment by Robert Varga [ 24/Feb/16 ]

Also, a YANG snippet defining the leaf would be nice.

Comment by Bhavesh Kumar [ 24/Feb/16 ]

Thanks for your support Robert.
I'll look into my yang files and try to resolve the anchors which are not allowed.

Comment by Robert Varga [ 24/Feb/16 ]

I am not sure, but I think one problem with regex matching may be that it does not interpret $ correctly, we'll continue to investigate.

Comment by Igor Foltin [ 25/Feb/16 ]

fix pushed: https://git.opendaylight.org/gerrit/#/c/35399/

stable/lithium: https://git.opendaylight.org/gerrit/#/c/35407/

Comment by Robert Varga [ 12/May/16 ]

THe agreed plan of action is to import the XSD RegEx engine from Xerces, under third-party/xsd-regex. Note the import should include only required classes and those need to be imported in the org.opendaylight.yangtools.xsd.regex.

These will then need to be modified, such that RegularExpression has a:

public java.util.Pattern toPattern() throws PatternSyntaxException;

method, which will produce a compiled Pattern. If an expression cannot be converted to a Pattern, PatternSyntaxException should be reported.

This pattern will then used in Java to validate inputs.

Comment by Robert Varga [ 20/May/16 ]

import of Xerces regex engine: https://git.opendaylight.org/gerrit/39168
cleanup to comply with ODL coding standards: https://git.opendaylight.org/gerrit/39169

Comment by Robert Varga [ 05/Sep/16 ]

master: https://git.opendaylight.org/gerrit/45145

Comment by Robert Varga [ 05/Sep/16 ]

While using Xerces' RegularExpression works, it has a very undesirable side-effect of expanding regexes – which renders debugging and error reporting a horror.

Hence I think it will be more feasible to create a simple XSD RegEx tokenizer, which will emit minimally-modified strings.

Comment by Robert Varga [ 21/Nov/16 ]

Alternate approach: https://git.opendaylight.org/gerrit/46990

Comment by Viera Zelcamova [ 17/Feb/17 ]

RCO delivery

Comment by Peter Kajsa [ 08/Mar/17 ]

(In reply to Robert Varga from comment #1)
> What yangtools can do is catch the case when an expression starts/ends
> with an anchor, complain loudly and ignore the anchor points.

I am not sure whether we are really able to do that, because for an example consider the following XSD pattern "x$". Are both chars '' and '$' added mistakenly as anchor points and should therefore be omitted or are they added intentionally in order to match string "^x$" ?

Comment by Peter Kajsa [ 08/Mar/17 ]

Adapted XSD regex grammar does not work for ANTLR correctly. I spent some time trying to fix it, but there are more issues in the grammar like ambiguity of rules, order of rules, etc. Therefore, I have proposed the following patch based on iteration over regex chars: https://git.opendaylight.org/gerrit/#/c/52999/

I think it is sufficient for our needs...

Comment by Peter Kajsa [ 08/Mar/17 ]

Patch for xsd-regex removal from Yangtools:
https://git.opendaylight.org/gerrit/#/c/53000/

It needs to be fixed (yangtools-carbon-distribution-check job fails). Resolving in progress ...

Comment by Robert Varga [ 11/Mar/17 ]

We should actually backport this to Boron

Comment by Dana Kutenicsova [ 19/Jul/17 ]

Reopening as problem persists in Boron-SR3. The comments below suggests, that if a regexp contains anchors, yangtools should ignore them. Instead of that, yangtools is escaping them and adding one more set of anchors, generating a compilable but unusable regexp pattern:

Yang (https://github.com/openconfig/public/blob/master/release/models/types/openconfig-yang-types.yang) :

typedef dotted-quad {
type string {
pattern '^(([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|' +
'25[0-5])\.)

{3}([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4]' +
'[0-9]|25[0-5])$';
}
}

- note all openconfig regexps contain anchors ^ and $

Generated java regexp:

public static final List<String> PATTERN_CONSTANTS = ImmutableList.of("^\\^(([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])
.){3}

([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])
$$");

Test :

@Test
public void testDottedQuad() {
new DottedQuad("127.0.0.1");
}

Output :

(([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])\.)

{3}([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])
Exception in thread "main" java.lang.IllegalArgumentException: Supplied value "127.0.0.1" does not match required pattern "^^(([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])\.){3}

([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])\$$"
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:145)
at org.opendaylight.yang.gen.v1.http.openconfig.net.yang.types.yang.rev170403.DottedQuad.<init>(DottedQuad.java:52)

Comment by Robert Varga [ 20/Jul/17 ]

https://tools.ietf.org/html/rfc6020#section-9.4.6:

The "pattern" statement, which is an optional substatement to the
"type" statement, takes as an argument a regular expression string,
as defined in [XSD-TYPES].

https://www.w3.org/TR/2004/REC-xmlschema-2-20041028/#regexs:

Note: Unlike some popular regular expression languages (including those defined by Perl and standard Unix utilities), the regular expression language defined here implicitly anchors all regular expressions at the head and tail, as the most common use of regular expressions in ·pattern· is to match entire literals.

That means that the models are wrong and they should not contain anchors.

Generated at Wed Feb 07 20:53:41 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.