https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/NETCONF-591 netconf <p>Netconf basic CSIT is failing:<br/> <a href="https://jenkins.opendaylight.org/releng/view/netconf/job/netconf-csit-1node-userfeatures-all-neon" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/netconf/job/netconf-csit-1node-userfeatures-all-neon</a></p> <p>Looking at the karaf log we can see:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> Caused by: org.w3c.dom.DOMException: NAMESPACE_ERR: An attempt is made to create or change an object in a way which is incorrect with regard to namespaces. </pre> </div></div> <p>Which means we will need to repair and merge this patch already merged in fluorine:</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/77471" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/77471</a></p> NETCONF-591

Netconf connection failing in master (neon)

Bug High Resolved Done Luis Gomez Luis Gomez Tue, 8 Jan 2019 17:45:42 +0000 Thu, 24 Jan 2019 17:23:07 +0000 Wed, 23 Jan 2019 09:29:38 +0000 Neon Neon 0 3 <p>Neon shading needs to be expanded to make grammar-informed encoding work, which is done by the original patch. The delta in shading is backported to Flourine SR2.</p> <p>Latest master test still fails:</p> <p><a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netconf-csit-1node-userfeatures-all-neon/156/odl_1/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netconf-csit-1node-userfeatures-all-neon/156/odl_1/odl1_karaf.log.gz</a></p> <p>Okay, so partial success, we don't get org.apache.xerces in the way, but it seems we are getting a straight anyxml serialization failure. Last movement around this was <a href="https://jira.opendaylight.org/browse/YANGTOOLS-923" title="Opendaylight netconf parser removes namespaces in anyxml container before sending request to router" class="issue-link" data-issue-key="YANGTOOLS-923"><del>YANGTOOLS-923</del></a>, which was coming from NETCONF. <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=JMorvay" class="user-hover" rel="JMorvay">JMorvay</a> any ideas what may be going on here?</p> <p>I don't quite like woodstox being in this stack trace:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> at com.sun.org.apache.xerces.internal.dom.ElementImpl.setAttributeNS(ElementImpl.java:659) ~[?:?] at com.ctc.wstx.dom.DOMOutputElement.addAttribute(DOMOutputElement.java:210) ~[32:com.fasterxml.woodstox.woodstox-core:5.0.3] at com.ctc.wstx.dom.WstxDOMWrappingWriter.outputAttribute(WstxDOMWrappingWriter.java:561) ~[32:com.fasterxml.woodstox.woodstox-core:5.0.3] at com.ctc.wstx.dom.WstxDOMWrappingWriter.writeAttribute(WstxDOMWrappingWriter.java:251) ~[32:com.fasterxml.woodstox.woodstox-core:5.0.3] at org.opendaylight.yangtools.yang.data.codec.xml.XMLStreamNormalizedNodeStreamWriter$DelegateWriterNoEndDoc.writeNamespace(XMLStreamNormalizedNodeStreamWriter.java:405) ~[509:org.opendaylight.yangtools.yang-data-codec-xml:2.1.7] at com.sun.org.apache.xalan.internal.xsltc.trax.SAX2StAXStreamWriter.startElement(SAX2StAXStreamWriter.java:176) ~[?:?] </pre> </div></div> <p>.. which is coming from com.ctc.wstx.stax.WstxOutputFactory being an advertized XMLOutputFactory. Since it does some transformation, it may be interfering with the documented hack we have in place.</p> <p>So the exception source reads:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>                if (qname.equals("xmlns") &amp;&amp; (namespaceURI == null                     || !namespaceURI.equals(NamespaceContext.XMLNS_URI))                     || (namespaceURI!=null &amp;&amp; namespaceURI.equals(NamespaceContext.XMLNS_URI)                     &amp;&amp; !qname.equals("xmlns"))) {                     String msg =                         DOMMessageFormatter.formatMessage(                                 DOMMessageFormatter.DOM_DOMAIN,                                 "NAMESPACE_ERR",                                 null);                     throw new DOMException(DOMException.NAMESPACE_ERR, msg);                 } </pre> </div></div> <p>which looks awefully close to what we are trying to avoid in the code. We do need the input data here.</p> <p>Btw. the anyxml fix is in yangtools-2.0.15, which is current Fluorine.</p> <p>So my prime suspect is woodstox being pulled into the picture, and it is a jackson dependency. There has been some movement on master w.r.t. swagger, notably <a href="https://jira.opendaylight.org/browse/NETCONF-588" title="Restconf Apidoc can&#39;t show the Model Schema" class="issue-link" data-issue-key="NETCONF-588"><del>NETCONF-588</del></a>. <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=ecelgp" class="user-hover" rel="ecelgp">ecelgp</a> can you try running it without installing apidocs, to see if it helps?</p> <p>The ERROR happens after this REST request:</p> <p>PUT restconf/config/network-topology:network-topology/topology/topology-netconf/node/test-device<br/> &lt;node xmlns="urn:TBD:params:xml:ns:yang:network-topology"&gt;<br/> &lt;node-id&gt;test-device&lt;/node-id&gt;<br/> &lt;host xmlns="urn:opendaylight:netconf-node-topology"&gt;10.30.170.230&lt;/host&gt;<br/> &lt;port xmlns="urn:opendaylight:netconf-node-topology"&gt;2830&lt;/port&gt;<br/> &lt;username xmlns="urn:opendaylight:netconf-node-topology"&gt;admin&lt;/username&gt;<br/> &lt;password xmlns="urn:opendaylight:netconf-node-topology"&gt;admin&lt;/password&gt;<br/> &lt;tcp-only xmlns="urn:opendaylight:netconf-node-topology"&gt;false&lt;/tcp-only&gt;<br/> &lt;keepalive-delay xmlns="urn:opendaylight:netconf-node-topology"&gt;0&lt;/keepalive-delay&gt;<br/> &lt;/node&gt;</p> <p>Sure, will try right away.</p> <p>apidocs may not be related as this job does not install apidocs and still shows the problem: <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netconf-csit-1node-scale-only-neon/98/odl_1/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netconf-csit-1node-scale-only-neon/98/odl_1/odl1_karaf.log.gz</a></p> <p>Regression analysis:</p> <ul class="alternate" type="square"> <li>Dec 22nd AR build works:<br/> <a href="https://jenkins.opendaylight.org/sandbox/job/netconf-csit-1node-userfeatures-all-neon/9/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/sandbox/job/netconf-csit-1node-userfeatures-all-neon/9/</a></li> </ul> <ul class="alternate" type="square"> <li>Dec 23rd AR build fails:<br/> <a href="https://jenkins.opendaylight.org/sandbox/job/netconf-csit-1node-userfeatures-all-neon/8/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/sandbox/job/netconf-csit-1node-userfeatures-all-neon/8/</a></li> </ul> <p>Patches for MRI bump were filed by that time (odlparent-4.0.5, yangtools-2.1.6, mdsal-3.0.3):<br/> <a href="https://git.opendaylight.org/gerrit/#/q/branch:master,525" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/q/branch:master,525</a></p> <p>That puts <a href="https://jira.opendaylight.org/browse/YANGTOOLS-923" title="Opendaylight netconf parser removes namespaces in anyxml container before sending request to router" class="issue-link" data-issue-key="YANGTOOLS-923"><del>YANGTOOLS-923</del></a> into picture, but it does not explain why fluorine works, as it carries the equivalent fix since Dec 22. I don't see anything in odlparent v4.0.2..v4.0.5 which would cause this.</p> <p>That having been said, I think part of the problem is the way yang-data-codec-xml performs the translation to DOM, i.e. the transformer (out of our reach) ends up adding the default namespace after an Element is created, rather than initializing its name with the namespace – and then we are doing a few questionable hacks to deal with the effects of that.</p> <p>I suspect these hacks are what breaks with woodstox – perhaps we can post-process the DOM document to remove these, or get the transformer of out the picture completely by forwarding the input events ourselves... not sure yet.</p> <p>Suspicion confirmed: if I add woodstox-core as a test dependency in yang-data-codec-xml, I am getting 9 failures and 3 errors – with two of those errors being the same thing as what the CSIT is finding.</p> <p> </p> <p><a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netconf-csit-1node-gate-userfeatures-all-neon/5/odl_1/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netconf-csit-1node-gate-userfeatures-all-neon/5/odl_1/odl1_karaf.log.gz</a> shows the factories to be initialized to woodstox ones:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2019-01-15T11:35:45,302 | INFO | features-3-thread-1 | NetconfUtil | 426 - org.opendaylight.netconf.util - 1.6.0.SNAPSHOT | Initializing XML factory initialized to class class com.ctc.wstx.stax.WstxOutputFactory instance com.ctc.wstx.stax.WstxOutputFactory@c8d838a java.lang.Throwable: null at org.opendaylight.netconf.util.NetconfUtil.&lt;clinit&gt;(NetconfUtil.java:43) ~[?:?] at org.opendaylight.netconf.notifications.impl.ops.NotificationsTransformUtil.transform(NotificationsTransformUtil.java:88) ~[?:?] at org.opendaylight.netconf.notifications.impl.ops.NotificationsTransformUtil.transform(NotificationsTransformUtil.java:75) ~[?:?] at org.opendaylight.netconf.notifications.impl.NetconfNotificationManager$BaseNotificationPublisherReg.serializeNotification(NetconfNotificationManager.java:278) ~[?:?] at org.opendaylight.netconf.notifications.impl.NetconfNotificationManager$BaseNotificationPublisherReg.onCapabilityChanged(NetconfNotificationManager.java:284) ~[?:?] at org.opendaylight.netconf.impl.osgi.NetconfCapabilityMonitoringService.onCapabilitiesChanged(NetconfCapabilityMonitoringService.java:225) ~[?:?] at org.opendaylight.netconf.notifications.impl.osgi.Activator$1.registerCapabilityListener(Activator.java:64) ~[?:?] at org.opendaylight.netconf.impl.osgi.AggregatedNetconfOperationServiceFactory.onAddNetconfOperationServiceFactory(AggregatedNetconfOperationServiceFactory.java:56) ~[?:?] at org.opendaylight.netconf.impl.osgi.NetconfOperationServiceFactoryTracker.addingService(NetconfOperationServiceFactoryTracker.java:33) ~[?:?] at org.opendaylight.netconf.impl.osgi.NetconfOperationServiceFactoryTracker.addingService(NetconfOperationServiceFactoryTracker.java:17) ~[?:?] at org.osgi.util.tracker.ServiceTracker$Tracked.customizerAdding(ServiceTracker.java:941) ~[?:?] at org.osgi.util.tracker.ServiceTracker$Tracked.customizerAdding(ServiceTracker.java:870) ~[?:?] at org.osgi.util.tracker.AbstractTracked.trackAdding(AbstractTracked.java:256) ~[?:?] at org.osgi.util.tracker.AbstractTracked.track(AbstractTracked.java:229) ~[?:?] at org.osgi.util.tracker.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:901) ~[?:?] at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:109) ~[?:?] at org.eclipse.osgi.internal.framework.BundleContextImpl.dispatchEvent(BundleContextImpl.java:920) ~[?:?] at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:230) ~[?:?] at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:148) ~[?:?] at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:862) ~[?:?] at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:801) ~[?:?] at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:127) ~[?:?] at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:225) ~[?:?] at org.eclipse.osgi.internal.framework.BundleContextImpl.registerService(BundleContextImpl.java:469) ~[?:?] at org.eclipse.osgi.internal.framework.BundleContextImpl.registerService(BundleContextImpl.java:487) ~[?:?] at org.eclipse.osgi.internal.framework.BundleContextImpl.registerService(BundleContextImpl.java:1004) ~[?:?] at org.opendaylight.netconf.notifications.impl.osgi.Activator.start(Activator.java:92) ~[?:?] at org.eclipse.osgi.internal.framework.BundleContextImpl$3.run(BundleContextImpl.java:779) ~[?:?] at org.eclipse.osgi.internal.framework.BundleContextImpl$3.run(BundleContextImpl.java:1) ~[?:?] at java.security.AccessController.doPrivileged(Native Method) ~[?:?] at org.eclipse.osgi.internal.framework.BundleContextImpl.startActivator(BundleContextImpl.java:772) ~[?:?] at org.eclipse.osgi.internal.framework.BundleContextImpl.start(BundleContextImpl.java:729) ~[?:?] at org.eclipse.osgi.internal.framework.EquinoxBundle.startWorker0(EquinoxBundle.java:933) ~[?:?] at org.eclipse.osgi.internal.framework.EquinoxBundle$EquinoxModule.startWorker(EquinoxBundle.java:309) ~[?:?] at org.eclipse.osgi.container.Module.doStart(Module.java:581) ~[?:?] at org.eclipse.osgi.container.Module.start(Module.java:449) ~[?:?] at org.eclipse.osgi.internal.framework.EquinoxBundle.start(EquinoxBundle.java:383) ~[?:?] at org.eclipse.osgi.internal.framework.EquinoxBundle.start(EquinoxBundle.java:402) ~[?:?] at org.apache.karaf.features.internal.service.BundleInstallSupportImpl.startBundle(BundleInstallSupportImpl.java:161) ~[?:?] at org.apache.karaf.features.internal.service.FeaturesServiceImpl.startBundle(FeaturesServiceImpl.java:1116) ~[?:?] at org.apache.karaf.features.internal.service.Deployer.deploy(Deployer.java:997) ~[?:?] at org.apache.karaf.features.internal.service.FeaturesServiceImpl.doProvision(FeaturesServiceImpl.java:1025) ~[?:?] at org.apache.karaf.features.internal.service.FeaturesServiceImpl.lambda$doProvisionInThread$13(FeaturesServiceImpl.java:964) ~[?:?] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:?] at java.lang.Thread.run(Thread.java:748) [?:?]</pre> </div></div> <p>and</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2019-01-15T11:36:49,135 | INFO | nioEventLoopGroupCloseable-6-3 | CopyConfig | 409 - org.opendaylight.netconf.mdsal-netconf-connector - 1.6.0.SNAPSHOT | Initializing XML factory initialized to class class com.ctc.wstx.stax.WstxOutputFactory instance com.ctc.wstx.stax.WstxOutputFactory@5c7fbd24 java.lang.Throwable: null at org.opendaylight.netconf.mdsal.connector.ops.CopyConfig.&lt;clinit&gt;(CopyConfig.java:67) ~[?:?] at org.opendaylight.netconf.mdsal.connector.OperationProvider.&lt;init&gt;(OperationProvider.java:35) ~[?:?] at org.opendaylight.netconf.mdsal.connector.MdsalNetconfOperationService.&lt;init&gt;(MdsalNetconfOperationService.java:23) ~[?:?] at org.opendaylight.netconf.mdsal.connector.MdsalNetconfOperationServiceFactory.createService(MdsalNetconfOperationServiceFactory.java:73) ~[?:?] at org.opendaylight.netconf.mdsal.connector.MdsalNetconfOperationServiceFactory.createService(MdsalNetconfOperationServiceFactory.java:40) ~[?:?] at org.opendaylight.netconf.impl.osgi.AggregatedNetconfOperationServiceFactory$AggregatedNetconfOperation.&lt;init&gt;(AggregatedNetconfOperationServiceFactory.java:130) ~[?:?] at org.opendaylight.netconf.impl.osgi.AggregatedNetconfOperationServiceFactory.createService(AggregatedNetconfOperationServiceFactory.java:109) ~[?:?] at org.opendaylight.netconf.impl.osgi.AggregatedNetconfOperationServiceFactory$AggregatedNetconfOperation.&lt;init&gt;(AggregatedNetconfOperationServiceFactory.java:130) ~[?:?] at org.opendaylight.netconf.impl.osgi.AggregatedNetconfOperationServiceFactory.createService(AggregatedNetconfOperationServiceFactory.java:109) ~[?:?] at org.opendaylight.netconf.impl.NetconfServerSessionNegotiatorFactory.getOperationServiceForAddress(NetconfServerSessionNegotiatorFactory.java:119) ~[?:?] at org.opendaylight.netconf.impl.NetconfServerSessionNegotiatorFactory.getListener(NetconfServerSessionNegotiatorFactory.java:110) ~[?:?] at org.opendaylight.netconf.impl.NetconfServerSessionNegotiatorFactory.getSessionNegotiator(NetconfServerSessionNegotiatorFactory.java:105) ~[?:?] at org.opendaylight.netconf.impl.NetconfServerDispatcherImpl$ServerChannelInitializer.initializeSessionNegotiator(NetconfServerDispatcherImpl.java:64) ~[?:?] at org.opendaylight.netconf.nettyutil.AbstractChannelInitializer.initialize(AbstractChannelInitializer.java:35) ~[412:org.opendaylight.netconf.netty-util:1.6.0.SNAPSHOT] at org.opendaylight.netconf.nettyutil.AbstractNetconfDispatcher$1.initChannel(AbstractNetconfDispatcher.java:110) [412:org.opendaylight.netconf.netty-util:1.6.0.SNAPSHOT] at io.netty.channel.ChannelInitializer.initChannel(ChannelInitializer.java:115) [61:io.netty.transport:4.1.31.Final] at io.netty.channel.ChannelInitializer.handlerAdded(ChannelInitializer.java:107) [61:io.netty.transport:4.1.31.Final] at io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:637) [61:io.netty.transport:4.1.31.Final] at io.netty.channel.DefaultChannelPipeline.access$000(DefaultChannelPipeline.java:46) [61:io.netty.transport:4.1.31.Final] at io.netty.channel.DefaultChannelPipeline$PendingHandlerAddedTask.execute(DefaultChannelPipeline.java:1487) [61:io.netty.transport:4.1.31.Final] at io.netty.channel.DefaultChannelPipeline.callHandlerAddedForAllHandlers(DefaultChannelPipeline.java:1161) [61:io.netty.transport:4.1.31.Final] at io.netty.channel.DefaultChannelPipeline.invokeHandlerAddedIfNeeded(DefaultChannelPipeline.java:686) [61:io.netty.transport:4.1.31.Final] at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:510) [61:io.netty.transport:4.1.31.Final] at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:423) [61:io.netty.transport:4.1.31.Final] at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:482) [61:io.netty.transport:4.1.31.Final] at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) [58:io.netty.common:4.1.31.Final] at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) [58:io.netty.common:4.1.31.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:466) [61:io.netty.transport:4.1.31.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) [58:io.netty.common:4.1.31.Final] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [58:io.netty.common:4.1.31.Final] at java.lang.Thread.run(Thread.java:748) [?:?] 2019-01-15T11:36:49,144 | INFO | nioEventLoopGroupCloseable-6-3 | AbstractGet | 409 - org.opendaylight.netconf.mdsal-netconf-connector - 1.6.0.SNAPSHOT | Initializing XML factory initialized to class class com.ctc.wstx.stax.WstxOutputFactory instance com.ctc.wstx.stax.WstxOutputFactory@4a1ad9d0 java.lang.Throwable: null at org.opendaylight.netconf.mdsal.connector.ops.get.AbstractGet.&lt;clinit&gt;(AbstractGet.java:55) ~[?:?] at org.opendaylight.netconf.mdsal.connector.OperationProvider.&lt;init&gt;(OperationProvider.java:35) ~[?:?] at org.opendaylight.netconf.mdsal.connector.MdsalNetconfOperationService.&lt;init&gt;(MdsalNetconfOperationService.java:23) ~[?:?] at org.opendaylight.netconf.mdsal.connector.MdsalNetconfOperationServiceFactory.createService(MdsalNetconfOperationServiceFactory.java:73) ~[?:?] at org.opendaylight.netconf.mdsal.connector.MdsalNetconfOperationServiceFactory.createService(MdsalNetconfOperationServiceFactory.java:40) ~[?:?] at org.opendaylight.netconf.impl.osgi.AggregatedNetconfOperationServiceFactory$AggregatedNetconfOperation.&lt;init&gt;(AggregatedNetconfOperationServiceFactory.java:130) ~[?:?] at org.opendaylight.netconf.impl.osgi.AggregatedNetconfOperationServiceFactory.createService(AggregatedNetconfOperationServiceFactory.java:109) ~[?:?] at org.opendaylight.netconf.impl.osgi.AggregatedNetconfOperationServiceFactory$AggregatedNetconfOperation.&lt;init&gt;(AggregatedNetconfOperationServiceFactory.java:130) ~[?:?] at org.opendaylight.netconf.impl.osgi.AggregatedNetconfOperationServiceFactory.createService(AggregatedNetconfOperationServiceFactory.java:109) ~[?:?] at org.opendaylight.netconf.impl.NetconfServerSessionNegotiatorFactory.getOperationServiceForAddress(NetconfServerSessionNegotiatorFactory.java:119) ~[?:?] at org.opendaylight.netconf.impl.NetconfServerSessionNegotiatorFactory.getListener(NetconfServerSessionNegotiatorFactory.java:110) ~[?:?] at org.opendaylight.netconf.impl.NetconfServerSessionNegotiatorFactory.getSessionNegotiator(NetconfServerSessionNegotiatorFactory.java:105) ~[?:?] at org.opendaylight.netconf.impl.NetconfServerDispatcherImpl$ServerChannelInitializer.initializeSessionNegotiator(NetconfServerDispatcherImpl.java:64) ~[?:?] at org.opendaylight.netconf.nettyutil.AbstractChannelInitializer.initialize(AbstractChannelInitializer.java:35) ~[412:org.opendaylight.netconf.netty-util:1.6.0.SNAPSHOT] at org.opendaylight.netconf.nettyutil.AbstractNetconfDispatcher$1.initChannel(AbstractNetconfDispatcher.java:110) [412:org.opendaylight.netconf.netty-util:1.6.0.SNAPSHOT] at io.netty.channel.ChannelInitializer.initChannel(ChannelInitializer.java:115) [61:io.netty.transport:4.1.31.Final] at io.netty.channel.ChannelInitializer.handlerAdded(ChannelInitializer.java:107) [61:io.netty.transport:4.1.31.Final] at io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:637) [61:io.netty.transport:4.1.31.Final] at io.netty.channel.DefaultChannelPipeline.access$000(DefaultChannelPipeline.java:46) [61:io.netty.transport:4.1.31.Final] at io.netty.channel.DefaultChannelPipeline$PendingHandlerAddedTask.execute(DefaultChannelPipeline.java:1487) [61:io.netty.transport:4.1.31.Final] at io.netty.channel.DefaultChannelPipeline.callHandlerAddedForAllHandlers(DefaultChannelPipeline.java:1161) [61:io.netty.transport:4.1.31.Final] at io.netty.channel.DefaultChannelPipeline.invokeHandlerAddedIfNeeded(DefaultChannelPipeline.java:686) [61:io.netty.transport:4.1.31.Final] at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:510) [61:io.netty.transport:4.1.31.Final] at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:423) [61:io.netty.transport:4.1.31.Final] at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:482) [61:io.netty.transport:4.1.31.Final] at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) [58:io.netty.common:4.1.31.Final] at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) [58:io.netty.common:4.1.31.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:466) [61:io.netty.transport:4.1.31.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) [58:io.netty.common:4.1.31.Final] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [58:io.netty.common:4.1.31.Final] at java.lang.Thread.run(Thread.java:748) [?:?] 2019-01-15T11:36:49,155 | INFO | nioEventLoopGroupCloseable-6-3 | RuntimeRpc | 409 - org.opendaylight.netconf.mdsal-netconf-connector - 1.6.0.SNAPSHOT | Initializing XML factory initialized to class class com.ctc.wstx.stax.WstxOutputFactory instance com.ctc.wstx.stax.WstxOutputFactory@30feb6b7 java.lang.Throwable: null at org.opendaylight.netconf.mdsal.connector.ops.RuntimeRpc.&lt;clinit&gt;(RuntimeRpc.java:69) ~[?:?] at org.opendaylight.netconf.mdsal.connector.OperationProvider.&lt;init&gt;(OperationProvider.java:35) ~[?:?] at org.opendaylight.netconf.mdsal.connector.MdsalNetconfOperationService.&lt;init&gt;(MdsalNetconfOperationService.java:23) ~[?:?] at org.opendaylight.netconf.mdsal.connector.MdsalNetconfOperationServiceFactory.createService(MdsalNetconfOperationServiceFactory.java:73) ~[?:?] at org.opendaylight.netconf.mdsal.connector.MdsalNetconfOperationServiceFactory.createService(MdsalNetconfOperationServiceFactory.java:40) ~[?:?] at org.opendaylight.netconf.impl.osgi.AggregatedNetconfOperationServiceFactory$AggregatedNetconfOperation.&lt;init&gt;(AggregatedNetconfOperationServiceFactory.java:130) ~[?:?] at org.opendaylight.netconf.impl.osgi.AggregatedNetconfOperationServiceFactory.createService(AggregatedNetconfOperationServiceFactory.java:109) ~[?:?] at org.opendaylight.netconf.impl.osgi.AggregatedNetconfOperationServiceFactory$AggregatedNetconfOperation.&lt;init&gt;(AggregatedNetconfOperationServiceFactory.java:130) ~[?:?] at org.opendaylight.netconf.impl.osgi.AggregatedNetconfOperationServiceFactory.createService(AggregatedNetconfOperationServiceFactory.java:109) ~[?:?] at org.opendaylight.netconf.impl.NetconfServerSessionNegotiatorFactory.getOperationServiceForAddress(NetconfServerSessionNegotiatorFactory.java:119) ~[?:?] at org.opendaylight.netconf.impl.NetconfServerSessionNegotiatorFactory.getListener(NetconfServerSessionNegotiatorFactory.java:110) ~[?:?] at org.opendaylight.netconf.impl.NetconfServerSessionNegotiatorFactory.getSessionNegotiator(NetconfServerSessionNegotiatorFactory.java:105) ~[?:?] at org.opendaylight.netconf.impl.NetconfServerDispatcherImpl$ServerChannelInitializer.initializeSessionNegotiator(NetconfServerDispatcherImpl.java:64) ~[?:?] at org.opendaylight.netconf.nettyutil.AbstractChannelInitializer.initialize(AbstractChannelInitializer.java:35) ~[412:org.opendaylight.netconf.netty-util:1.6.0.SNAPSHOT] at org.opendaylight.netconf.nettyutil.AbstractNetconfDispatcher$1.initChannel(AbstractNetconfDispatcher.java:110) [412:org.opendaylight.netconf.netty-util:1.6.0.SNAPSHOT] at io.netty.channel.ChannelInitializer.initChannel(ChannelInitializer.java:115) [61:io.netty.transport:4.1.31.Final] at io.netty.channel.ChannelInitializer.handlerAdded(ChannelInitializer.java:107) [61:io.netty.transport:4.1.31.Final] at io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:637) [61:io.netty.transport:4.1.31.Final] at io.netty.channel.DefaultChannelPipeline.access$000(DefaultChannelPipeline.java:46) [61:io.netty.transport:4.1.31.Final] at io.netty.channel.DefaultChannelPipeline$PendingHandlerAddedTask.execute(DefaultChannelPipeline.java:1487) [61:io.netty.transport:4.1.31.Final] at io.netty.channel.DefaultChannelPipeline.callHandlerAddedForAllHandlers(DefaultChannelPipeline.java:1161) [61:io.netty.transport:4.1.31.Final] at io.netty.channel.DefaultChannelPipeline.invokeHandlerAddedIfNeeded(DefaultChannelPipeline.java:686) [61:io.netty.transport:4.1.31.Final] at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:510) [61:io.netty.transport:4.1.31.Final] at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:423) [61:io.netty.transport:4.1.31.Final] at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:482) [61:io.netty.transport:4.1.31.Final] at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) [58:io.netty.common:4.1.31.Final] at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) [58:io.netty.common:4.1.31.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:466) [61:io.netty.transport:4.1.31.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) [58:io.netty.common:4.1.31.Final] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [58:io.netty.common:4.1.31.Final] at java.lang.Thread.run(Thread.java:748) [?:?]</pre> </div></div> <p>Now we just need to figure out how are we arriving at this implementation.</p> <p>Okay, so this is something coming from Karaf, again. <a href="https://issues.apache.org/jira/browse/KARAF-5897" class="external-link" target="_blank" rel="nofollow noopener">https://issues.apache.org/jira/browse/KARAF-5897</a> integrated in 4.2.2 means that the woodstox factory is now leaked across unrelated bundle trees <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/sad.png" height="16" width="16" align="absmiddle" alt="" border="0"/> Let's see how we can fix this....</p> <p>So woodstox does not seem to be at fault, the error seems to lie in JDK's SAX2StAXStreamWriter assumptions about what XMLStreamWriter does, specifically how its startElement() side-steps emits namespaces and uses qName where localName is expected.</p> <p>Since xalan-2.7.2 does not support StAX (it supports on JAXP-1.3), and <a href="http://apache-xml-project.6118.n7.nabble.com/StAXSources-with-Xalan-td21126.html" class="external-link" target="_blank" rel="nofollow noopener">http://apache-xml-project.6118.n7.nabble.com/StAXSources-with-Xalan-td21126.html</a> indicates that the JDK fork is buggy (as seems to be the case above) and that there is now work being done on having xalan-J support StAX.</p> <p>So it's time to ditch transformers at least in the DOM-&gt;StAX path, which is what is biting us here.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=ecelgp" class="user-hover" rel="ecelgp">ecelgp</a> the CSIT seems to be back up, modulo the change in empty data response. Can you update it, please?</p> Blocks YANGTOOLS-926 Relates CONTROLLER-1867 Development Rank 0|i03lx3: