[CONTROLLER-1360] PUT and POST to yang-ext:mount'd device fails on Li works on He Created: 05/Jun/15 Updated: 09/Jun/15 Resolved: 09/Jun/15 |
|
| Status: | Verified |
| Project: | controller |
| Component/s: | netconf |
| Affects Version/s: | Lithium |
| Fix Version/s: | None |
| Type: | Bug | ||
| Reporter: | Carol Sanders | Assignee: | Tom Pantelis |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
Operating System: All |
||
| External issue ID: | 3604 |
| Priority: | High |
| Description |
|
When I attempt to perform a PUT or POST I receive the following Exception messages: 2015-06-05 15:21:07,255 | TRACE | b]-nio2-thread-3 | AsyncSshHandlerReader | 172 - org.opendaylight.controller.netconf-netty-util - 0.3.0.SNAPSHOT | Reading message on channel: ChannelSubsystem[id=0, recipient=0], message: 2015-06-05 15:21:07,265 | WARN | oupCloseable-6-2 | NetconfDeviceCommunicator | 261 - org.opendaylight.controller.sal-netconf-connector - 1.2.0.SNAPSHOT | RemoteDevice {10.18.160.47}: Error reply from remote device, request: <rpc message-id="m-1" xmlns="urn:ietf:params:xml:ns:netconf:base:1.0"><edit-config> <target> <running/> </target> <config/> </edit-config> </rpc> , response: <rpc-reply xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="m-1"> <rpc-error> <error-type>protocol</error-type> <error-tag>missing-element</error-tag> <error-severity>error</error-severity> <error-path> /rpc/edit-config/config/__other </error-path> <error-info> <bad-element>__other</bad-element> </error-info> </rpc-error> </rpc-reply> NetconfDocumentedException{message=, errorType=protocol, errorTag=missing_element, errorSeverity=error, errorInfo={bad-element=__other}} at org.opendaylight.controller.netconf.api.NetconfDocumentedException.fromXMLDocument(NetconfDocumentedException.java:243) at org.opendaylight.controller.sal.connect.netconf.util.NetconfMessageTransformUtil.checkSuccessReply(NetconfMessageTransformUtil.java:199)[261:org.opendaylight.controller.sal-netconf-connector:1.2.0.SNAPSHOT] at org.opendaylight.controller.sal.connect.netconf.listener.NetconfDeviceCommunicator.processMessage(NetconfDeviceCommunicator.java:258)[261:org.opendaylight.controller.sal-netconf-connector:1.2.0.SNAPSHOT] at org.opendaylight.controller.sal.connect.netconf.listener.NetconfDeviceCommunicator.onMessage(NetconfDeviceCommunicator.java:213)[261:org.opendaylight.controller.sal-netconf-connector:1.2.0.SNAPSHOT] at org.opendaylight.controller.sal.connect.netconf.listener.NetconfDeviceCommunicator.onMessage(NetconfDeviceCommunicator.java:46)[261:org.opendaylight.controller.sal-netconf-connector:1.2.0.SNAPSHOT] at org.opendaylight.controller.netconf.nettyutil.AbstractNetconfSession.handleMessage(AbstractNetconfSession.java:61)[172:org.opendaylight.controller.netconf-netty-util:0.3.0.SNAPSHOT] at org.opendaylight.controller.netconf.nettyutil.AbstractNetconfSession.handleMessage(AbstractNetconfSession.java:32)[172:org.opendaylight.controller.netconf-netty-util:0.3.0.SNAPSHOT] at org.opendaylight.protocol.framework.AbstractProtocolSession.channelRead0(AbstractProtocolSession.java:53)[154:org.opendaylight.controller.protocol-framework:0.6.0.SNAPSHOT] at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)[151:io.netty.transport:4.0.26.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)[151:io.netty.transport:4.0.26.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)[151:io.netty.transport:4.0.26.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242)[177:io.netty.codec:4.0.26.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)[151:io.netty.transport:4.0.26.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)[151:io.netty.transport:4.0.26.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:242)[177:io.netty.codec:4.0.26.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)[151:io.netty.transport:4.0.26.Final] at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:32)[151:io.netty.transport:4.0.26.Final] at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:329)[151:io.netty.transport:4.0.26.Final] at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357)[152:io.netty.common:4.0.26.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)[151:io.netty.transport:4.0.26.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)[152:io.netty.common:4.0.26.Final] at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)[152:io.netty.common:4.0.26.Final] at java.lang.Thread.run(Thread.java:745)[:1.7.0_67] 2015-06-05 15:21:07,267 | TRACE | oupCloseable-6-2 | KeepaliveSalFacade | 261 - org.opendaylight.controller.sal-netconf-connector - 1.2.0.SNAPSHOT | RemoteDevice{10.18.160.47} : Resetting netconf keepalive timer 2015-06-05 15:21:07,267 | WARN | oupCloseable-6-2 | NetconfRpcFutureCallback | 261 - org.opendaylight.controller.sal-netconf-connector - 1.2.0.SNAPSHOT | RemoteDevice{10.18.160.47} : Edit running invoked unsuccessfully: [RpcError [message=, severity=ERROR, errorType=PROTOCOL, tag=missing-element, applicationTag=null, info=<bad-element>__other</bad-element>, cause=null]] NetconfDocumentedException{message=RemoteDevice{10.18.160.47} : Edit running failed: [RpcError [message=, severity=ERROR, errorType=PROTOCOL, tag=missing-element, applicationTag=null, info=<bad-element>__other</bad-element>, cause=null]], errorType=application, errorTag=operation_failed, errorSeverity=warning, errorInfo={}} : Sending message <rpc message-id="m-2" xmlns="urn:ietf:params:xml:ns:netconf:base:1.0"> 2015-06-05 15:21:07,270 | TRACE | oupCloseable-6-2 | NetconfMessageToXMLEncoder | 172 - org.opendaylight.controller.netconf-netty-util - 0.3.0.SNAPSHOT | Sent to encode : <rpc message-id="m-2" xmlns="urn:ietf:params:xml:ns:netconf:base:1.0"> 2015-06-05 15:21:07,270 | TRACE | oupCloseable-6-2 | AsyncSshHandlerWriter | 172 - org.opendaylight.controller.netconf-netty-util - 0.3.0.SNAPSHOT | Writing request on channel: [id: 0xaadde62b], message: ## 2015-06-05 15:21:07,271 | TRACE | b]-nio2-thread-4 | AsyncSshHandlerWriter | 172 - org.opendaylight.controller.netconf-netty-util - 0.3.0.SNAPSHOT | Ssh write request finished on channel: [id: 0xaadde62b] with result: true: and ex:null, message: |
| Comments |
| Comment by Carol Sanders [ 05/Jun/15 ] |
|
Corresponding RESTconf error message: 2015-06-05 13:13:35,544 | INFO | qtp425724088-540 | RestconfImpl | 244 - org.opendaylight.controller.sal-rest-connector - 1.2.0.SNAPSHOT | Error creating data config/network-topology:network-topology/topology/topology-netconf/node/10.18.160.47/yang-ext:mount/ at org.opendaylight.controller.sal.connect.netconf.sal.tx.WriteRunningTx.handleEditException(WriteRunningTx.java:85) at org.opendaylight.controller.sal.connect.netconf.sal.tx.AbstractWriteTx.merge(AbstractWriteTx.java:111) at org.opendaylight.controller.sal.connect.netconf.sal.tx.ReadWriteTx.merge(ReadWriteTx.java:51) at org.opendaylight.controller.sal.restconf.impl.BrokerFacade.postDataViaTransaction(BrokerFacade.java:220)[244:org.opendaylight.controller.sal-rest-connector:1.2.0.SNAPSHOT] at org.opendaylight.controller.sal.restconf.impl.BrokerFacade.commitConfigurationDataPost(BrokerFacade.java:140)[244:org.opendaylight.controller.sal-rest-connector:1.2.0.SNAPSHOT] at org.opendaylight.controller.sal.restconf.impl.RestconfImpl.createConfigurationData(RestconfImpl.java:913)[244:org.opendaylight.controller.sal-rest-connector:1.2.0.SNAPSHOT] at org.opendaylight.controller.sal.restconf.impl.RestconfImpl.createConfigurationData(RestconfImpl.java:860)[244:org.opendaylight.controller.sal-rest-connector:1.2.0.SNAPSHOT] at org.opendaylight.controller.sal.restconf.impl.StatisticsRestconfServiceWrapper.createConfigurationData(StatisticsRestconfServiceWrapper.java:152)[244:org.opendaylight.controller.sal-rest-connector:1.2.0.SNAPSHOT] at org.opendaylight.controller.sal.rest.impl.RestconfCompositeWrapper.createConfigurationData(RestconfCompositeWrapper.java:79)[244:org.opendaylight.controller.sal-rest-connector:1.2.0.SNAPSHOT] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.7.0_67] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)[:1.7.0_67] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.7.0_67] at java.lang.reflect.Method.invoke(Method.java:606)[:1.7.0_67] at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)[218:com.sun.jersey.jersey-server:1.17.0] at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)[218:com.sun.jersey.jersey-server:1.17.0] at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)[218:com.sun.jersey.jersey-server:1.17.0] at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)[218:com.sun.jersey.jersey-server:1.17.0] at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)[218:com.sun.jersey.jersey-server:1.17.0] at com.sun.jersey.server.impl.uri.rules.ResourceObjectRule.accept(ResourceObjectRule.java:100)[218:com.sun.jersey.jersey-server:1.17.0] at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)[218:com.sun.jersey.jersey-server:1.17.0] at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)[218:com.sun.jersey.jersey-server:1.17.0] at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1511)[218:com.sun.jersey.jersey-server:1.17.0] at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1442)[218:com.sun.jersey.jersey-server:1.17.0] at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1391)[218:com.sun.jersey.jersey-server:1.17.0] at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1381)[218:com.sun.jersey.jersey-server:1.17.0] at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)[216:com.sun.jersey.servlet:1.17.0] at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:538)[216:com.sun.jersey.servlet:1.17.0] at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:716)[216:com.sun.jersey.servlet:1.17.0] at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)[65:org.apache.geronimo.specs.geronimo-servlet_3.0_spec:1.0.0] at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)[71:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411] at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1496)[71:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411] at org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:247)[71:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411] at org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:210)[71:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411] at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1467)[71:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411] at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:501)[71:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411] at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:69)[80:org.ops4j.pax.web.pax-web-jetty:3.1.4] at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)[71:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411] at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)[71:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411] at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)[71:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411] at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)[71:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411] at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:240)[80:org.ops4j.pax.web.pax-web-jetty:3.1.4] at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:429)[71:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411] at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)[71:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411] at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)[71:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411] at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)[71:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411] at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:75)[80:org.ops4j.pax.web.pax-web-jetty:3.1.4] at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)[71:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411] at org.eclipse.jetty.server.Server.handle(Server.java:370)[71:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411] at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)[71:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411] at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:982)[71:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411] at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1043)[71:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411] at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865)[71:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411] at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)[71:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411] at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)[71:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411] at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696)[71:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411] at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53)[71:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411] at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)[71:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411] at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)[71:org.eclipse.jetty.aggregate.jetty-all-server:8.1.15.v20140411] at java.lang.Thread.run(Thread.java:745)[:1.7.0_67] Caused by: NetconfDocumentedException{message=RemoteDevice{10.18.160.47} : Edit running failed: [RpcError [message=, severity=ERROR, errorType=PROTOCOL, tag=missing-element, applicationTag=null, info=<bad-element>__other</bad-element>, cause=null]], errorType=application, errorTag=operation_failed, errorSeverity=warning, errorInfo={}} 2015-06-05 13:13:38,916 | TRACE | d-keepalives-445 | NetconfDeviceCommunicator | 261 - org.opendaylight.controller.sal-netconf-connector - 1.2.0.SNAPSHOT | RemoteDevice{controller-config} : Sending message <rpc message-id="m-4" xmlns="urn:ietf:params:xml:ns:netconf:base:1.0"> |
| Comment by Carol Sanders [ 06/Jun/15 ] |
|
To clarify, this bug occurs when attempting to POST or PUT to an externally mounted NETCONF device. |
| Comment by Tony Tkacik [ 08/Jun/15 ] |
|
<error-path> Seems that you are trying to configure something which is not valid from Netconf device perspective, and device rejected edit-config. Seems to be user error and not restconf / netconf error. |
| Comment by Maros Marsalek [ 08/Jun/15 ] |
|
Hi Carol, The error response from device seems reasonable looking at the request: There is no content. So could you show me the entire request (URL + payload + headers) that you tried to POST/PUT ? Full log might also be useful ... if thats an possible. |
| Comment by Balaji Varadaraju [ 08/Jun/15 ] |
|
Hi Maros, Looks like Restconf has issues writing to the Mount point resulting in empty edit-config request to the device in case of Lithium. We tried both POST and PUT operations but in vain. We also tried with both XML and JSON content-type. Both works on Helium but not on Lithium. <xxx> http://localhost:8181/restconf/config/opendaylight-inventory:nodes/node/devicename/yang-ext:mount/ Please note that I have obfuscated certain information . But the structure is complete. Please let me know if you need more information. Key point is why is Restconf request is resulting in empty edit-config request to the device on Lithium but works just fine on Helium? Thanks, |
| Comment by Tom Pantelis [ 08/Jun/15 ] |
|
From the stacktrace, line 220 in BrokerFacade: private CheckedFuture<Void, TransactionCommitFailedException> } else { checkItemDoesNotExists(rWTransaction,datastore, path); ensureParentsByMerge(datastore, path, rWTransaction, schemaContext); rWTransaction.put(datastore, path, payload); } return rWTransaction.submit(); So it appears a yang list entry is being added and the code is purposely trying to merge an empty parent node to ensure the parent container first exists. Apparently the device doesn't like the empty node. |
| Comment by Tom Pantelis [ 08/Jun/15 ] |
|
The yang model is simple - it's just a top-level list and the restconf call is trying to put an entry into the list. We reproduced the issue however now, for some reason, it hits the else clause in postDataViaTransaction, i.e. the payload is a MapEntryNode and not a MapNode. However it still fails in ensureParentsByMerge b/c the YangInstanceIdentifier path contains 2 args, both the same, e.g. urn:org.com:mgmt:my-list/urn:org.com:mgmt:my-list. It seems this path is manufactured by the front-end code to match the internal DOM representation of yang lists where there's a "hidden" MapNode that represents the list container. So the first path arg corresponds to the MapNode and the second corresponds to the actual list entry node (MapEntryNode). The internal DOM datastore forces the client to explicitly create the containing MapNode rather than implicitly/lazily creating it on the first list entry. It seems the code assumes every system behaves the same way which clearly isn't the case. In this case, sending a yang list entry with empty data is rejected by the device (understandably so). Maybe some devices accept/ignore it. I see a couple options here. 1) When writing to a DOMMountPoint, just write the data and elide the ensureParentsByMerge. Let the device error out or implicitly create the parents. 2) When writing to a DOMMountPoint, for ensureParentsByMerge, skip MapNodes, i.e. do not send empty data for a list container. |
| Comment by Tom Pantelis [ 09/Jun/15 ] |
|
3) Ignore empty MapNode in the netconf layer. (In reply to Tom Pantelis from comment #7) |
| Comment by Tony Tkacik [ 09/Jun/15 ] |
|
Actually I would say correct option is to handle this in Netconf
|
| Comment by Maros Marsalek [ 09/Jun/15 ] |
|
I agree with Tony on fixing this in netconf-connector. Heres a fix proposal: Netconf-connector has to ignore data that result in an edit-config with empty payload. So it means ignoring data built only from Mixin normalized nodes. But I dont think we need to recursively check the entire data structure, because there is only one possible case of data being built only from mixins and that is top level list (other mixins have to be contained by a regular/visible node and no other mixin nodes are allowed at top level in yang). With this in mind, the fix is quite simple. |
| Comment by Tom Pantelis [ 09/Jun/15 ] |
|
It can be fixed in the netconf connector as well. However, the ensure parents with empty invisible/mixin nodes mechanism seems to be specific to the internal datastore so the advantage of handling it in restconf is that it isn't exposed to mount points. We probably only have the netconf mount point right now, but if there's others in the future, they may run into a similar issue and have to special case. |