[NETCONF-610] Custom scheme-cache-directory yang models are not replicated among cluster members

Mon, 16 Mar 2020 09:52:57 +0000

When a cluster member w/a custom netconf mount goes down, either A) already pre-share to ALL members so it's "hitless" or B) 100% guarantee that the remount to a new member will always create the directory correctly. Need test tooling & cases around this too.

how to replicate:

0. clean up data/ & journal/ & snapshots/ & cache/ in the /karaf/target/assembly/ folder

1. run cluster - with features: odl-jolokia odl-netconf-clustered-topology odl-restconf-all on each node

2. run the testool

3. try to mount the testool with schema-cache-directory (example):
<node xmlns="urn:TBD:params:xml:ns:yang:network-topology">
<node-id>cluster-node</node-id>
<host xmlns="urn:opendaylight:netconf-node-topology">10.0.2.12</host>
<password xmlns="urn:opendaylight:netconf-node-topology">admin</password>
<username xmlns="urn:opendaylight:netconf-node-topology">admin</username>
<schema-cache-directory xmlns="urn:opendaylight:netconf-node-topology">/home/ary/yangs</schema-cache-directory>
<port xmlns="urn:opendaylight:netconf-node-topology">17830</port>
<tcp-only xmlns="urn:opendaylight:netconf-node-topology">false</tcp-only>
<schemaless xmlns="urn:opendaylight:netconf-node-topology">false</schemaless>
<max-connection-attempts xmlns="urn:opendaylight:netconf-node-topology">0</max-connection-attempts>
<connection-timeout-millis xmlns="urn:opendaylight:netconf-node-topology">20000</connection-timeout-millis>
<default-request-timeout-millis xmlns="urn:opendaylight:netconf-node-topology">60000</default-request-timeout-millis>
<sleep-factor xmlns="urn:opendaylight:netconf-node-topology">1.1</sleep-factor>
<between-attempts-timeout-millis xmlns="urn:opendaylight:netconf-node-topology">2000</between-attempts-timeout-millis>
<reconnect-on-changed-schema xmlns="urn:opendaylight:netconf-node-topology">false</reconnect-on-changed-schema>
<keepalive-delay xmlns="urn:opendaylight:netconf-node-topology">600</keepalive-delay>
<concurrent-rpc-limit xmlns="urn:opendaylight:netconf-node-topology">0</concurrent-rpc-limit>
<actor-response-wait-time xmlns="urn:opendaylight:netconf-node-topology">600</actor-response-wait-time>
</node>

04:33:12.098 ERROR [opendaylight-cluster-data-akka.actor.default-dispatcher-58] Got more than one input Future failure. Logging failures after the first
org.opendaylight.yangtools.yang.model.repo.api.MissingSchemaSourceException: All available providers exhausted
at org.opendaylight.yangtools.yang.model.repo.util.AbstractSchemaRepository.lambda$fetchSource$0(AbstractSchemaRepository.java:79) ~[330:org.opendaylight.yangtools.yang-model-util:3.0.1]
at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.doFallback(AbstractCatchingFuture.java:175) ~[84:com.google.guava:27.1.0.jre]
at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.doFallback(AbstractCatchingFuture.java:162) ~[84:com.google.guava:27.1.0.jre]
at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:107) ~[84:com.google.guava:27.1.0.jre]
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) ~[84:com.google.guava:27.1.0.jre]
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1138) ~[84:com.google.guava:27.1.0.jre]
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:958) ~[84:com.google.guava:27.1.0.jre]
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:748) ~[84:com.google.guava:27.1.0.jre]
at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:100) ~[84:com.google.guava:27.1.0.jre]
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) ~[84:com.google.guava:27.1.0.jre]
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1138) ~[84:com.google.guava:27.1.0.jre]
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:958) ~[84:com.google.guava:27.1.0.jre]
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:748) ~[84:com.google.guava:27.1.0.jre]
at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:109) ~[84:com.google.guava:27.1.0.jre]
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) ~[84:com.google.guava:27.1.0.jre]
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1138) ~[84:com.google.guava:27.1.0.jre]
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:958) ~[84:com.google.guava:27.1.0.jre]
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:748) ~[84:com.google.guava:27.1.0.jre]
at com.google.common.util.concurrent.SettableFuture.setException(SettableFuture.java:54) ~[84:com.google.guava:27.1.0.jre]
at org.opendaylight.controller.cluster.schema.provider.impl.RemoteSchemaProvider$2.onComplete(RemoteSchemaProvider.java:66) ~[226:org.opendaylight.controller.sal-clustering-commons:1.10.0.SNAPSHOT]
at org.opendaylight.controller.cluster.schema.provider.impl.RemoteSchemaProvider$2.onComplete(RemoteSchemaProvider.java:58) ~[226:org.opendaylight.controller.sal-clustering-commons:1.10.0.SNAPSHOT]
at akka.dispatch.OnComplete.internal(Future.scala:263) ~[90:com.typesafe.akka.actor:2.5.21]
at akka.dispatch.OnComplete.internal(Future.scala:261) ~[90:com.typesafe.akka.actor:2.5.21]
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:191) ~[90:com.typesafe.akka.actor:2.5.21]
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:188) ~[90:com.typesafe.akka.actor:2.5.21]
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64) ~[350:org.scala-lang.scala-library:2.12.8.v20181128-140630-VFINAL-38cd84d]
at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55) ~[90:com.typesafe.akka.actor:2.5.21]
at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:91) ~[90:com.typesafe.akka.actor:2.5.21]
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) [350:org.scala-lang.scala-library:2.12.8.v20181128-140630-VFINAL-38cd84d]
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:85) [350:org.scala-lang.scala-library:2.12.8.v20181128-140630-VFINAL-38cd84d]
at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:91) [90:com.typesafe.akka.actor:2.5.21]
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40) [90:com.typesafe.akka.actor:2.5.21]
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44) [90:com.typesafe.akka.actor:2.5.21]
at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [90:com.typesafe.akka.actor:2.5.21]
at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [90:com.typesafe.akka.actor:2.5.21]
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [90:com.typesafe.akka.actor:2.5.21]
at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [90:com.typesafe.akka.actor:2.5.21]
Caused by: org.opendaylight.yangtools.yang.model.repo.api.MissingSchemaSourceException: All available providers exhausted
at org.opendaylight.yangtools.yang.model.repo.util.AbstractSchemaRepository.lambda$fetchSource$0(AbstractSchemaRepository.java:79) ~[330:org.opendaylight.yangtools.yang-model-util:3.0.1]
at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.doFallback(AbstractCatchingFuture.java:175) ~[84:com.google.guava:27.1.0.jre]
at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.doFallback(AbstractCatchingFuture.java:162) ~[84:com.google.guava:27.1.0.jre]
at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:107) ~[84:com.google.guava:27.1.0.jre]
... 23 more
Caused by: org.opendaylight.yangtools.yang.model.repo.api.MissingSchemaSourceException: No providers registered for sourceRevisionSourceIdentifier [name=ietf-netconf-monitoring@2010-10-04]
at org.opendaylight.yangtools.yang.model.repo.util.AbstractSchemaRepository.getSchemaSource(AbstractSchemaRepository.java:92) ~[330:org.opendaylight.yangtools.yang-model-util:3.0.1]
at org.opendaylight.netconf.topology.singleton.impl.actors.NetconfNodeActor.sendYangTextSchemaSourceProxy(NetconfNodeActor.java:208) ~[?:?]
at org.opendaylight.netconf.topology.singleton.impl.actors.NetconfNodeActor.handleReceive(NetconfNodeActor.java:146) ~[?:?]
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) ~[90:com.typesafe.akka.actor:2.5.21]
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) ~[90:com.typesafe.akka.actor:2.5.21]
at scala.PartialFunction.applyOrElse(PartialFunction.scala:127) ~[350:org.scala-lang.scala-library:2.12.8.v20181128-140630-VFINAL-38cd84d]
at scala.PartialFunction.applyOrElse$(PartialFunction.scala:126) ~[350:org.scala-lang.scala-library:2.12.8.v20181128-140630-VFINAL-38cd84d]
at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) ~[90:com.typesafe.akka.actor:2.5.21]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:175) ~[350:org.scala-lang.scala-library:2.12.8.v20181128-140630-VFINAL-38cd84d]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:176) ~[350:org.scala-lang.scala-library:2.12.8.v20181128-140630-VFINAL-38cd84d]
at akka.actor.Actor.aroundReceive(Actor.scala:517) ~[90:com.typesafe.akka.actor:2.5.21]
at akka.actor.Actor.aroundReceive$(Actor.scala:515) ~[90:com.typesafe.akka.actor:2.5.21]
at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) ~[90:com.typesafe.akka.actor:2.5.21]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) ~[90:com.typesafe.akka.actor:2.5.21]
at akka.actor.ActorCell.invoke(ActorCell.scala:561) ~[90:com.typesafe.akka.actor:2.5.21]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) ~[90:com.typesafe.akka.actor:2.5.21]
at akka.dispatch.Mailbox.run(Mailbox.scala:225) ~[90:com.typesafe.akka.actor:2.5.21]
at akka.dispatch.Mailbox.exec(Mailbox.scala:235) ~[90:com.typesafe.akka.actor:2.5.21]
... 4 more

OpenDaylight JIRA

[NETCONF-610] Custom scheme-cache-directory yang models are not replicated among cluster members