[YANGTOOLS-587] Parser: XSD regular expressions are interpreted as Java regexes Created: 24/Feb/16 Updated: 10/Apr/22 Resolved: 20/Jul/17 |
|
| Status: | Resolved |
| Project: | yangtools |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Bug | ||
| Reporter: | Bhavesh Kumar | Assignee: | Peter Kajsa |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
Operating System: Linux |
||
| Attachments: |
|
||||||||||||||||||||
| Issue Links: |
|
||||||||||||||||||||
| External issue ID: | 5410 | ||||||||||||||||||||
| Priority: | High | ||||||||||||||||||||
| Description |
|
I am using integration(stable/lithium), have registered a netconf device (device1) successfully. applicationoperation-failedProblem to get data from transaction.ReadFailedException{message=read execution failed, errorList=[RpcError [message=read execution failed, severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=java.lang.IllegalArgumentException: Value $6$AnrKGc0V$B/0/A.pWg4HrrA6YiEJOtFGibQ9Fmm5.4rI/00gEz3QeB7joSxBU3YtbHDm6NSkS1dKTQy3BWhwKKDS8nB5S// does not match regular expression <^^$0$.*|$1$[a-zA-Z0-9./] {1,8}$[a-zA-Z0-9./]{22}|$5$(rounds=\d+$)?[a-zA-Z0-9./]{1,16}$[a-zA-Z0-9./]{43}|$6$(rounds=\d+$)?[a-zA-Z0-9./]{1,16}$[a-zA-Z0-9./]{86}$$>]]} at org.opendaylight.controller.md.sal.common.api.data.ReadFailedException$1.newWithCause(ReadFailedException.java:26) at org.opendaylight.controller.md.sal.common.api.data.ReadFailedException$1.newWithCause(ReadFailedException.java:23) at org.opendaylight.yangtools.util.concurrent.ExceptionMapper.apply(ExceptionMapper.java:80) at org.opendaylight.yangtools.util.concurrent.ExceptionMapper.apply(ExceptionMapper.java:31) at org.opendaylight.yangtools.util.concurrent.MappingCheckedFuture.mapException(MappingCheckedFuture.java:60) at org.opendaylight.yangtools.util.concurrent.MappingCheckedFuture.wrapInExecutionException(MappingCheckedFuture.java:64) at org.opendaylight.yangtools.util.concurrent.MappingCheckedFuture.get(MappingCheckedFuture.java:77) at org.opendaylight.controller.sal.restconf.impl.BrokerFacade.readDataViaTransaction(BrokerFacade.java:198) at org.opendaylight.controller.sal.restconf.impl.BrokerFacade.readConfigurationData(BrokerFacade.java:87) at org.opendaylight.controller.sal.restconf.impl.RestconfImpl.readConfigurationData(RestconfImpl.java:658) at org.opendaylight.controller.sal.restconf.impl.StatisticsRestconfServiceWrapper.readConfigurationData(StatisticsRestconfServiceWrapper.java:95) at org.opendaylight.controller.sal.rest.impl.RestconfCompositeWrapper.readConfigurationData(RestconfCompositeWrapper.java:64) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185) at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.ResourceObjectRule.accept(ResourceObjectRule.java:100) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1511) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1442) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1391) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1381) at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:538) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:716) at javax.servlet.http.HttpServlet.service(HttpServlet.java:668) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1496) at org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:247) at org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:210) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467) at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:82) at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:294) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501) at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:69) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086) at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:240) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:429) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:75) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.eclipse.jetty.server.Server.handle(Server.java:370) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494) at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.IllegalArgumentException: Value $6$AnrKGc0V$B/0/A.pWg4HrrA6YiEJOtFGibQ9Fmm5.4rI/00gEz3QeB7joSxBU3YtbHDm6NSkS1dKTQy3BWhwKKDS8nB5S// does not match regular expression <^^$0$.*|$1$[a-zA-Z0-9./]{1,8}$[a-zA-Z0-9./] {22}|$5$(rounds=\d+$)?[a-zA-Z0-9./] {1,16}$[a-zA-Z0-9./]{43}|$6$(rounds=\d+$)?[a-zA-Z0-9./]{1,16}$[a-zA-Z0-9./] {86}$$> at com.google.common.base.Preconditions.checkArgument(Preconditions.java:145) at org.opendaylight.yangtools.yang.data.impl.codec.CompiledPatternContext.validate(CompiledPatternContext.java:31) at org.opendaylight.yangtools.yang.data.impl.codec.StringPatternCheckingCodec.validate(StringPatternCheckingCodec.java:39) at org.opendaylight.yangtools.yang.data.impl.codec.StringStringCodec.deserialize(StringStringCodec.java:36) at org.opendaylight.yangtools.yang.data.impl.codec.StringStringCodec.deserialize(StringStringCodec.java:14) at org.opendaylight.yangtools.yang.data.impl.schema.transform.dom.DomUtils.parseXmlValue(DomUtils.java:58) at org.opendaylight.yangtools.yang.data.impl.schema.transform.dom.DomUtils.parseXmlValue(DomUtils.java:136) at org.opendaylight.yangtools.yang.data.impl.schema.transform.dom.parser.LeafNodeDomParser.parseLeaf(LeafNodeDomParser.java:45) at org.opendaylight.yangtools.yang.data.impl.schema.transform.dom.parser.LeafNodeDomParser.parseLeaf(LeafNodeDomParser.java:22) at org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.LeafNodeBaseParser.parse(LeafNodeBaseParser.java:46) at org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.LeafNodeBaseParser.parse(LeafNodeBaseParser.java:27) at org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.NodeParserDispatcher$BaseNodeParserDispatcher.dispatchChildElement(NodeParserDispatcher.java:50) at org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.BaseDispatcherParser.parse(BaseDispatcherParser.java:177) at org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.ListEntryNodeBaseParser.parse(ListEntryNodeBaseParser.java:61) at org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.ListEntryNodeBaseParser.parse(ListEntryNodeBaseParser.java:28) at org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.ListNodeBaseParser.parse(ListNodeBaseParser.java:46) at org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.ListNodeBaseParser.parse(ListNodeBaseParser.java:26) at org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.NodeParserDispatcher$BaseNodeParserDispatcher.dispatchChildElement(NodeParserDispatcher.java:60) at org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.BaseDispatcherParser.parse(BaseDispatcherParser.java:177) at org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.ContainerNodeBaseParser.parse(ContainerNodeBaseParser.java:47) at org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.ContainerNodeBaseParser.parse(ContainerNodeBaseParser.java:29) at org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.NodeParserDispatcher$BaseNodeParserDispatcher.dispatchChildElement(NodeParserDispatcher.java:48) at org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.BaseDispatcherParser.parse(BaseDispatcherParser.java:177) at org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.ContainerNodeBaseParser.parse(ContainerNodeBaseParser.java:47) at org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.ContainerNodeBaseParser.parse(ContainerNodeBaseParser.java:29) at org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.NodeParserDispatcher$BaseNodeParserDispatcher.dispatchChildElement(NodeParserDispatcher.java:48) at org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.BaseDispatcherParser.parse(BaseDispatcherParser.java:177) at org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.ContainerNodeBaseParser.parse(ContainerNodeBaseParser.java:47) at org.opendaylight.yangtools.yang.data.impl.schema.transform.base.parser.ContainerNodeBaseParser.parse(ContainerNodeBaseParser.java:29) at org.opendaylight.controller.sal.connect.netconf.schema.mapping.NetconfMessageTransformer.toRpcResult(NetconfMessageTransformer.java:271) at org.opendaylight.controller.sal.connect.netconf.schema.mapping.NetconfMessageTransformer.toRpcResult(NetconfMessageTransformer.java:71) at org.opendaylight.controller.sal.connect.netconf.sal.NetconfDeviceRpc$2.apply(NetconfDeviceRpc.java:69) at org.opendaylight.controller.sal.connect.netconf.sal.NetconfDeviceRpc$2.apply(NetconfDeviceRpc.java:65) at com.google.common.util.concurrent.Futures$2.apply(Futures.java:760) at com.google.common.util.concurrent.Futures$ChainingListenableFuture.run(Futures.java:906) at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457) at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:185) at org.opendaylight.controller.sal.connect.netconf.listener.UncancellableFuture.set(UncancellableFuture.java:45) at org.opendaylight.controller.sal.connect.netconf.listener.NetconfDeviceCommunicator.processMessage(NetconfDeviceCommunicator.java:268) at org.opendaylight.controller.sal.connect.netconf.listener.NetconfDeviceCommunicator.onMessage(NetconfDeviceCommunicator.java:213) at org.opendaylight.controller.sal.connect.netconf.listener.NetconfDeviceCommunicator.onMessage(NetconfDeviceCommunicator.java:46) at org.opendaylight.controller.netconf.nettyutil.AbstractNetconfSession.handleMessage(AbstractNetconfSession.java:61) at org.opendaylight.controller.netconf.nettyutil.AbstractNetconfSession.handleMessage(AbstractNetconfSession.java:32) at org.opendaylight.protocol.framework.AbstractProtocolSession.channelRead0(AbstractProtocolSession.java:53) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339) at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:32) at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:329) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) ... 1 more Logs attached. |
| Comments |
| Comment by Bhavesh Kumar [ 24/Feb/16 ] |
|
Attachment karaf.log has been added with description: karaf log of integration while mounting the device |
| Comment by Robert Varga [ 24/Feb/16 ] |
|
Input YANG file contains invalid regular expression, hence the error. Note that YANG follows XSD regular expressions, as defined in https://www.w3.org/TR/2004/REC-xmlschema-2-20041028/#regexs. These do not allow ^/$ anchors (as anchoring is implicit) – which are obviously present. Error on invalid input, downgrading to improvement/normal, targetting to SR4. What yangtools can do is catch the case when an expression starts/ends with an anchor, complain loudly and ignore the anchor points. |
| Comment by Robert Varga [ 24/Feb/16 ] |
|
Also, a YANG snippet defining the leaf would be nice. |
| Comment by Bhavesh Kumar [ 24/Feb/16 ] |
|
Thanks for your support Robert. |
| Comment by Robert Varga [ 24/Feb/16 ] |
|
I am not sure, but I think one problem with regex matching may be that it does not interpret $ correctly, we'll continue to investigate. |
| Comment by Igor Foltin [ 25/Feb/16 ] |
|
fix pushed: https://git.opendaylight.org/gerrit/#/c/35399/ stable/lithium: https://git.opendaylight.org/gerrit/#/c/35407/ |
| Comment by Robert Varga [ 12/May/16 ] |
|
THe agreed plan of action is to import the XSD RegEx engine from Xerces, under third-party/xsd-regex. Note the import should include only required classes and those need to be imported in the org.opendaylight.yangtools.xsd.regex. These will then need to be modified, such that RegularExpression has a: public java.util.Pattern toPattern() throws PatternSyntaxException; method, which will produce a compiled Pattern. If an expression cannot be converted to a Pattern, PatternSyntaxException should be reported. This pattern will then used in Java to validate inputs. |
| Comment by Robert Varga [ 20/May/16 ] |
|
import of Xerces regex engine: https://git.opendaylight.org/gerrit/39168 |
| Comment by Robert Varga [ 05/Sep/16 ] |
| Comment by Robert Varga [ 05/Sep/16 ] |
|
While using Xerces' RegularExpression works, it has a very undesirable side-effect of expanding regexes – which renders debugging and error reporting a horror. Hence I think it will be more feasible to create a simple XSD RegEx tokenizer, which will emit minimally-modified strings. |
| Comment by Robert Varga [ 21/Nov/16 ] |
|
Alternate approach: https://git.opendaylight.org/gerrit/46990 |
| Comment by Viera Zelcamova [ 17/Feb/17 ] |
|
RCO delivery |
| Comment by Peter Kajsa [ 08/Mar/17 ] |
|
(In reply to Robert Varga from comment #1) I am not sure whether we are really able to do that, because for an example consider the following XSD pattern "x$". Are both chars '' and '$' added mistakenly as anchor points and should therefore be omitted or are they added intentionally in order to match string "^x$" ? |
| Comment by Peter Kajsa [ 08/Mar/17 ] |
|
Adapted XSD regex grammar does not work for ANTLR correctly. I spent some time trying to fix it, but there are more issues in the grammar like ambiguity of rules, order of rules, etc. Therefore, I have proposed the following patch based on iteration over regex chars: https://git.opendaylight.org/gerrit/#/c/52999/ I think it is sufficient for our needs... |
| Comment by Peter Kajsa [ 08/Mar/17 ] |
|
Patch for xsd-regex removal from Yangtools: It needs to be fixed (yangtools-carbon-distribution-check job fails). Resolving in progress ... |
| Comment by Robert Varga [ 11/Mar/17 ] |
|
We should actually backport this to Boron |
| Comment by Dana Kutenicsova [ 19/Jul/17 ] |
|
Reopening as problem persists in Boron-SR3. The comments below suggests, that if a regexp contains anchors, yangtools should ignore them. Instead of that, yangtools is escaping them and adding one more set of anchors, generating a compilable but unusable regexp pattern: Yang (https://github.com/openconfig/public/blob/master/release/models/types/openconfig-yang-types.yang) : typedef dotted-quad { '[0-9]|25[0-5])$'; } } - note all openconfig regexps contain anchors ^ and $ Generated java regexp: public static final List<String> PATTERN_CONSTANTS = ImmutableList.of("^\\^(([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5]) .){3} ([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5]) Test : @Test Output : (([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])\.) {3}([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])Exception in thread "main" java.lang.IllegalArgumentException: Supplied value "127.0.0.1" does not match required pattern "^^(([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])\.){3} ([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])\$$" |
| Comment by Robert Varga [ 20/Jul/17 ] |
|
https://tools.ietf.org/html/rfc6020#section-9.4.6: The "pattern" statement, which is an optional substatement to the https://www.w3.org/TR/2004/REC-xmlschema-2-20041028/#regexs: Note: Unlike some popular regular expression languages (including those defined by Perl and standard Unix utilities), the regular expression language defined here implicitly anchors all regular expressions at the head and tail, as the most common use of regular expressions in ·pattern· is to match entire literals. That means that the models are wrong and they should not contain anchors. |