https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/NETCONF-397 netconf <p>The <del>only</del> job passes, and <span class="error">&#91;0&#93;</span> the symptoms are different from <a href="https://jira.opendaylight.org/browse/CONTROLLER-1613" title="Clustering: Member fails to re-start sometimes in csit -all- jobs" class="issue-link" data-issue-key="CONTROLLER-1613"><del>CONTROLLER-1613</del></a>.<br/> Probably some other project is interfering.</p> <p>Connection to testtool is closed, this is seen in karaf.log:<br/> 2017-04-10 02:07:08,435 | WARN | monPool-worker-0 | NetconfDeviceCommunicator | 284 - org.opendaylight.netconf.sal-netconf-connector - 1.4.4.SNAPSHOT | RemoteDevice</p> {test-device} <p>: Session is disconnected, failing RPC request &lt;rpc message-id="m-2874" xmlns="urn:ietf:params:xml:ns:netconf:base:1.0"&gt;<br/> &lt;get-schema xmlns="urn:ietf:params:xml:ns:yang:ietf-netconf-monitoring"&gt;<br/> &lt;identifier&gt;ietf-yang-types&lt;/identifier&gt;<br/> &lt;version&gt;2013-07-15&lt;/version&gt;<br/> &lt;format&gt;yang&lt;/format&gt;<br/> &lt;/get-schema&gt;<br/> &lt;/rpc&gt;</p> <p>The request itself look OK, investigation is under way.</p> <p><span class="error">&#91;0&#93;</span> <a href="https://jenkins.opendaylight.org/releng/view/netconf/job/netconf-csit-3node-clustering-all-boron" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/netconf/job/netconf-csit-3node-clustering-all-boron</a></p> <p>Operating System: All<br/> Platform: All</p> NETCONF-397

Boron netconf cluster -all- job failures

Bug Verified Done Andrej Mak Vratko Polak Mon, 10 Apr 2017 13:42:17 +0000 Fri, 15 Mar 2019 22:22:40 +0000 Sun, 7 May 2017 10:46:50 +0000 netconf 0 2 <p>This Bug does not happen if USC and CAPWAP features are removed from odl-integration-compatible-with-all. More details to follow.</p> <p>It is sufficient to remove CAPWAP. <a href="https://jenkins.opendaylight.org/sandbox/job/integration-multipatch-test-boron/6/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/sandbox/job/integration-multipatch-test-boron/6/</a></p> <p>It seems, that for some reason, slave mount points are registered about 30 s later than master mount point. Test checks "netconf-node-topology:connection-status" == "connected" condition. This attribute is set by master, when connection to device is established and master node mount point registered. However, slave nodes begin schema resolution and mount point registration only after setting "connected" by master.</p> <p><a href="https://logs.opendaylight.org/releng/jenkins092/netconf-csit-3node-clustering-all-boron/477/archives/" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/netconf-csit-3node-clustering-all-boron/477/archives/</a></p> <p>Node 1 - master:<br/> 2017-04-13 01:15:32,374 | INFO | ssing-executor-8 | MasterSalFacade | 509 - org.opendaylight.netconf.topology-singleton - 1.1.4.SNAPSHOT | RemoteDevice</p> {netconf-test-device}: Creating master data broker for device<br/> 2017-04-13 01:15:32,375 | INFO | ssing-executor-8 | NetconfDevice | 284 - org.opendaylight.netconf.sal-netconf-connector - 1.4.4.SNAPSHOT | RemoteDevice{netconf-test-device} <p>: Netconf connector initialized successfully</p> <p>Node 2 slave:<br/> 2017-04-13 01:16:03,501 | INFO | lt-dispatcher-18 | NetconfNodeActor | 509 - org.opendaylight.netconf.topology-singleton - 1.1.4.SNAPSHOT | RemoteDevice</p> {netconf-test-device}: Schema context resolved: [ModuleEffectiveStatementImpl<span class="error">&#91;name=ietf-inet-types, namespace=urn:ietf:params:xml:ns:yang:ietf-inet-types, revision=Fri Sep 24 00:00:00 UTC 2010, prefix=inet, yangVersion=1&#93;</span>, ModuleEffectiveStatementImpl<span class="error">&#91;name=yang-ext, namespace=urn:opendaylight:yang:extension:yang-ext, revision=Tue Jul 09 00:00:00 UTC 2013, prefix=ext, yangVersion=1&#93;</span>, ModuleEffectiveStatementImpl<span class="error">&#91;name=people, namespace=urn:opendaylight:params:xml:ns:yang:controller:config:sal-clustering-it:people, revision=Mon Aug 18 00:00:00 UTC 2014, prefix=people, yangVersion=1&#93;</span>, ModuleEffectiveStatementImpl<span class="error">&#91;name=test, namespace=urn:opendaylight:test:netconf:crud, revision=Sat Oct 18 00:00:00 UTC 2014, prefix=tt, yangVersion=1&#93;</span>, ModuleEffectiveStatementImpl<span class="error">&#91;name=ietf-yang-types, namespace=urn:ietf:params:xml:ns:yang:ietf-yang-types, revision=Mon Jul 15 00:00:00 UTC 2013, prefix=yang, yangVersion=1&#93;</span>, ModuleEffectiveStatementImpl<span class="error">&#91;name=car, namespace=urn:opendaylight:params:xml:ns:yang:controller:config:sal-clustering-it:car, revision=Mon Aug 18 00:00:00 UTC 2014, prefix=car, yangVersion=1&#93;</span>, ModuleEffectiveStatementImpl<span class="error">&#91;name=car-people, namespace=urn:opendaylight:params:xml:ns:yang:controller:config:sal-clustering-it:car-people, revision=Mon Aug 18 00:00:00 UTC 2014, prefix=car, yangVersion=1&#93;</span>, ModuleEffectiveStatementImpl<span class="error">&#91;name=car-purchase, namespace=urn:opendaylight:params:xml:ns:yang:controller:config:sal-clustering-it:car-purchase, revision=Mon Aug 18 00:00:00 UTC 2014, prefix=cp, yangVersion=1&#93;</span>, ModuleEffectiveStatementImpl<span class="error">&#91;name=ncmount-l2fib, namespace=org:opendaylight:coretutorials:ncmount:example:l2fib, revision=Mon Mar 07 00:00:00 UTC 2016, prefix=l2fib, yangVersion=1&#93;</span>, ModuleEffectiveStatementImpl<span class="error">&#91;name=ietf-inet-types, namespace=urn:ietf:params:xml:ns:yang:ietf-inet-types, revision=Mon Jul 15 00:00:00 UTC 2013, prefix=inet, yangVersion=1&#93;</span>, ModuleEffectiveStatementImpl<span class="error">&#91;name=ietf-netconf-monitoring, namespace=urn:ietf:params:xml:ns:yang:ietf-netconf-monitoring, revision=Mon Oct 04 00:00:00 UTC 2010, prefix=ncm, yangVersion=1&#93;</span>, ModuleEffectiveStatementImpl<span class="error">&#91;name=ietf-netconf-monitoring-extension, namespace=urn:ietf:params:xml:ns:yang:ietf-netconf-monitoring-extension, revision=Tue Dec 10 00:00:00 UTC 2013, prefix=ncme, yangVersion=1&#93;</span>]<br/> 2017-04-13 01:16:03,502 | INFO | lt-dispatcher-18 | SlaveSalFacade | 509 - org.opendaylight.netconf.topology-singleton - 1.1.4.SNAPSHOT | RemoteDevice{netconf-test-device} <p>: Slave mount point registered.</p> <p>Node 3 slave:<br/> 2017-04-13 01:16:03,504 | INFO | lt-dispatcher-18 | NetconfNodeActor | 509 - org.opendaylight.netconf.topology-singleton - 1.1.4.SNAPSHOT | RemoteDevice</p> {netconf-test-device}: Schema context resolved: [ModuleEffectiveStatementImpl<span class="error">&#91;name=ietf-inet-types, namespace=urn:ietf:params:xml:ns:yang:ietf-inet-types, revision=Fri Sep 24 00:00:00 UTC 2010, prefix=inet, yangVersion=1&#93;</span>, ModuleEffectiveStatementImpl<span class="error">&#91;name=yang-ext, namespace=urn:opendaylight:yang:extension:yang-ext, revision=Tue Jul 09 00:00:00 UTC 2013, prefix=ext, yangVersion=1&#93;</span>, ModuleEffectiveStatementImpl<span class="error">&#91;name=people, namespace=urn:opendaylight:params:xml:ns:yang:controller:config:sal-clustering-it:people, revision=Mon Aug 18 00:00:00 UTC 2014, prefix=people, yangVersion=1&#93;</span>, ModuleEffectiveStatementImpl<span class="error">&#91;name=test, namespace=urn:opendaylight:test:netconf:crud, revision=Sat Oct 18 00:00:00 UTC 2014, prefix=tt, yangVersion=1&#93;</span>, ModuleEffectiveStatementImpl<span class="error">&#91;name=ietf-yang-types, namespace=urn:ietf:params:xml:ns:yang:ietf-yang-types, revision=Mon Jul 15 00:00:00 UTC 2013, prefix=yang, yangVersion=1&#93;</span>, ModuleEffectiveStatementImpl<span class="error">&#91;name=car, namespace=urn:opendaylight:params:xml:ns:yang:controller:config:sal-clustering-it:car, revision=Mon Aug 18 00:00:00 UTC 2014, prefix=car, yangVersion=1&#93;</span>, ModuleEffectiveStatementImpl<span class="error">&#91;name=car-people, namespace=urn:opendaylight:params:xml:ns:yang:controller:config:sal-clustering-it:car-people, revision=Mon Aug 18 00:00:00 UTC 2014, prefix=car, yangVersion=1&#93;</span>, ModuleEffectiveStatementImpl<span class="error">&#91;name=car-purchase, namespace=urn:opendaylight:params:xml:ns:yang:controller:config:sal-clustering-it:car-purchase, revision=Mon Aug 18 00:00:00 UTC 2014, prefix=cp, yangVersion=1&#93;</span>, ModuleEffectiveStatementImpl<span class="error">&#91;name=ncmount-l2fib, namespace=org:opendaylight:coretutorials:ncmount:example:l2fib, revision=Mon Mar 07 00:00:00 UTC 2016, prefix=l2fib, yangVersion=1&#93;</span>, ModuleEffectiveStatementImpl<span class="error">&#91;name=ietf-inet-types, namespace=urn:ietf:params:xml:ns:yang:ietf-inet-types, revision=Mon Jul 15 00:00:00 UTC 2013, prefix=inet, yangVersion=1&#93;</span>, ModuleEffectiveStatementImpl<span class="error">&#91;name=ietf-netconf-monitoring, namespace=urn:ietf:params:xml:ns:yang:ietf-netconf-monitoring, revision=Mon Oct 04 00:00:00 UTC 2010, prefix=ncm, yangVersion=1&#93;</span>, ModuleEffectiveStatementImpl<span class="error">&#91;name=ietf-netconf-monitoring-extension, namespace=urn:ietf:params:xml:ns:yang:ietf-netconf-monitoring-extension, revision=Tue Dec 10 00:00:00 UTC 2013, prefix=ncme, yangVersion=1&#93;</span>]<br/> 2017-04-13 01:16:03,505 | INFO | lt-dispatcher-18 | SlaveSalFacade | 509 - org.opendaylight.netconf.topology-singleton - 1.1.4.SNAPSHOT | RemoteDevice{netconf-test-device} <p>: Slave mount point registered.</p> <p>CAPWAP features are no longer in odl-integration-compatible-with-all, but the job <span class="error">&#91;0&#93;</span> still contains features.</p> <p>&gt; slave mount points are registered about 30 s later than master mount point.</p> <p>Confirmed.<br/> This happens in current Boron snapshot builds (without Capwap, with or without USC).</p> <p>The interesting thing is this only happens in "CRUD" suite, which is executed second. It does not happen in (basically the same) Bug8086 suite (executed third), and it does not happen in somewhat similar Netconfready suite (executed first).</p> <p>Still no idea why slaves are late just in the second suite of <del>all</del> job.</p> <p>The only suspicious thing visible in karaf.log at INFO level is the following (happens on device master):</p> <p>2017-04-17 16:26:25,348 | INFO | ssing-executor-3 | NetconfDevice | 262 - org.opendaylight.netconf.sal-netconf-connector - 1.4.4.Boron-SR4 | RemoteDevice</p> {netconf-test-device} <p>: Netconf connector initialized successfully<br/> 2017-04-17 16:26:25,454 | INFO | on-dispatcher-62 | SfcProviderSffEntryDataListener | 452 - org.opendaylight.sfc.provider - 0.3.4.Boron-SR4 | org.opendaylight.sfc.provider.SfcProviderSffEntryDataListener.onDataChanged(SfcProviderSffEntryDataListener.java:94): Create SFF [SffName <span class="error">&#91;_value=netconf-test-device&#93;</span>]<br/> 2017-04-17 16:26:25,457 | ERROR | on-dispatcher-66 | DataChangeListener | 191 - org.opendaylight.controller.sal-distributed-datastore - 1.4.4.Boron-SR4 | Error notifying listener org.opendaylight.sfc.sfc_lisp.provider.listener.SfcLispSffDataListener<br/> java.lang.NullPointerException<br/> at org.opendaylight.sfc.sfc_lisp.provider.LispUpdater.containsLispAddress(LispUpdater.java:125)<span class="error">&#91;459:org.opendaylight.sfc.lisp:0.3.4.Boron-SR4&#93;</span><br/> at org.opendaylight.sfc.sfc_lisp.provider.listener.SfcLispSffDataListener.onDataChanged(SfcLispSffDataListener.java:64)<span class="error">&#91;459:org.opendaylight.sfc.lisp:0.3.4.Boron-SR4&#93;</span><br/> at org.opendaylight.controller.md.sal.binding.impl.AbstractForwardedDataBroker$TranslatingDataChangeInvoker.onDataChanged(AbstractForwardedDataBroker.java:143)<span class="error">&#91;161:org.opendaylight.controller.sal-binding-broker-impl:1.4.4.Boron-SR4&#93;</span><br/> at org.opendaylight.controller.cluster.datastore.DataChangeListener.dataChanged(DataChangeListener.java:73)<span class="error">&#91;191:org.opendaylight.controller.sal-distributed-datastore:1.4.4.Boron-SR4&#93;</span><br/> at org.opendaylight.controller.cluster.datastore.DataChangeListener.handleReceive(DataChangeListener.java:45)<span class="error">&#91;191:org.opendaylight.controller.sal-distributed-datastore:1.4.4.Boron-SR4&#93;</span><br/> at org.opendaylight.controller.cluster.common.actor.AbstractUntypedActor.onReceive(AbstractUntypedActor.java:26)<span class="error">&#91;185:org.opendaylight.controller.sal-clustering-commons:1.4.4.Boron-SR4&#93;</span><br/> at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:165)<span class="error">&#91;173:com.typesafe.akka.actor:2.4.7&#93;</span><br/> at akka.actor.Actor$class.aroundReceive(Actor.scala:484)<span class="error">&#91;173:com.typesafe.akka.actor:2.4.7&#93;</span><br/> at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:95)<span class="error">&#91;173:com.typesafe.akka.actor:2.4.7&#93;</span><br/> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)<span class="error">&#91;173:com.typesafe.akka.actor:2.4.7&#93;</span><br/> at akka.actor.ActorCell.invoke(ActorCell.scala:495)<span class="error">&#91;173:com.typesafe.akka.actor:2.4.7&#93;</span><br/> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)<span class="error">&#91;173:com.typesafe.akka.actor:2.4.7&#93;</span><br/> at akka.dispatch.Mailbox.run(Mailbox.scala:224)<span class="error">&#91;173:com.typesafe.akka.actor:2.4.7&#93;</span><br/> at akka.dispatch.Mailbox.exec(Mailbox.scala:234)<span class="error">&#91;173:com.typesafe.akka.actor:2.4.7&#93;</span><br/> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)<span class="error">&#91;169:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8&#93;</span><br/> at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)<span class="error">&#91;169:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8&#93;</span><br/> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)<span class="error">&#91;169:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8&#93;</span><br/> at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)<span class="error">&#91;169:org.scala-lang.scala-library:2.11.8.v20160304-115712-1706a37eb8&#93;</span><br/> 2017-04-17 16:26:25,464 | INFO | on-dispatcher-68 | SfcSffToOvsMappingAPI | 454 - org.opendaylight.sfc.ovs - 0.3.4.Boron-SR4 | Cannot build OvsdbBridgeAugmentation. Missing OVS Bridge augmentation on SFF SffName <span class="error">&#91;_value=netconf-test-device&#93;</span></p> <p>From sandbox run with enhanced logging it seems, that those 30 s on slave node are spent to download yang sources from master during schema resolution. I will add additional logging to find exact cause of this delay. I've attached logs from CRUD suite with enhanced logging.</p> <p>Gerrit debug logs patch:<br/> <a href="https://git.opendaylight.org/gerrit/#/c/55792/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/55792/</a></p> <p>Attachment CRUD-debug-logs.zip has been added with description: CRUD-debug-logs</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/55909/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/55909/</a></p> <p>Carbon:<br/> <a href="https://git.opendaylight.org/gerrit/#/c/55981" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/55981</a><br/> Boron:<br/> <a href="https://git.opendaylight.org/gerrit/#/c/55998" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/55998</a></p> <p>The job is passing now <span class="error">&#91;2&#93;</span>.</p> <p><span class="error">&#91;2&#93;</span> <a href="https://jenkins.opendaylight.org/releng/view/netconf/job/netconf-csit-3node-clustering-all-boron/504/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/netconf/job/netconf-csit-3node-clustering-all-boron/504/</a></p> Development External issue ID 8197 External issue URL Issue Type Rank 0|i01yaf: