Netconf crash configuring device on clustered setup: "All available providers exhausted"

Tested on stable/beryllium running on a three node cluster.

After trying to configure a Netconf device on clustered setup, the Netconf crashes with the following errors:

2016-01-24 02:44:03,837 | WARN | ult-dispatcher-2 | ClusteredNetconfDevice | 242 - org.opendaylight.netconf.topology - 1.0.0.SNAPSHOT | RemoteDevice

{netconf-test-device}: Unexpected error resolving device sources: org.opendaylight.yangtools.yang.model.repo.api.MissingSchemaSourceException: All available providers exhausted
2016-01-24 02:44:03,839 | ERROR | lt-dispatcher-27 | Futures$CombinedFuture | 39 - com.google.guava - 18.0.0 | input future failed.
org.opendaylight.yangtools.yang.model.repo.api.MissingSchemaSourceException: All available providers exhausted
at org.opendaylight.yangtools.yang.model.repo.util.AbstractSchemaRepository$1.create(AbstractSchemaRepository.java:83)
at com.google.common.util.concurrent.Futures$FallbackFuture$1.onFailure(Futures.java:471)
at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
at com.google.common.util.concurrent.Futures$ChainingListenableFuture.run(Futures.java:902)
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
at com.google.common.util.concurrent.Futures$FallbackFuture$1.onFailure(Futures.java:492)
at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
at com.google.common.util.concurrent.SettableFuture.setException(SettableFuture.java:68)
at org.opendaylight.controller.cluster.schema.provider.impl.RemoteSchemaProvider$2.onComplete(RemoteSchemaProvider.java:65)
at org.opendaylight.controller.cluster.schema.provider.impl.RemoteSchemaProvider$2.onComplete(RemoteSchemaProvider.java:58)
at akka.dispatch.OnComplete.internal(Future.scala:247)
at akka.dispatch.OnComplete.internal(Future.scala:245)
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:175)
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:172)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: org.opendaylight.yangtools.yang.model.repo.api.MissingSchemaSourceException: All available providers exhausted
at org.opendaylight.yangtools.yang.model.repo.util.AbstractSchemaRepository$1.create(AbstractSchemaRepository.java:83)
at com.google.common.util.concurrent.Futures$FallbackFuture$1.onFailure(Futures.java:471)
... 25 more
Caused by: org.opendaylight.yangtools.yang.model.repo.api.MissingSchemaSourceException: No providers registered for sourceSourceIdentifier [name=ietf-netconf-monitoring@2010-10-04]
at org.opendaylight.yangtools.yang.model.repo.util.AbstractSchemaRepository.getSchemaSource(AbstractSchemaRepository.java:92)
at org.opendaylight.controller.cluster.schema.provider.impl.RemoteYangTextSourceProviderImpl.getYangTextSchemaSource(RemoteYangTextSourceProviderImpl.java:51)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at akka.actor.TypedActor$MethodCall.apply(TypedActor.scala:147)
at akka.actor.TypedActor$TypedActor$$anonfun$receive$1$$anonfun$applyOrElse$2.apply(TypedActor.scala:311)
at akka.actor.TypedActor$TypedActor.withContext(TypedActor.scala:299)
at akka.actor.TypedActor$TypedActor$$anonfun$receive$1.applyOrElse(TypedActor.scala:306)
at akka.actor.Actor$class.aroundReceive(Actor.scala:467)
at akka.actor.TypedActor$TypedActor.aroundReceive(TypedActor.scala:246)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
at akka.dispatch.Mailbox.run(Mailbox.scala:220)
... 5 more
2016-01-24 02:44:03,837 | ERROR | ult-dispatcher-2 | NetconfDevice | 240 - org.opendaylight.netconf.sal-netconf-connector - 1.3.0.SNAPSHOT | RemoteDevice{netconf-test-device}

: Initialization in sal failed, disconnecting from device
org.opendaylight.yangtools.yang.model.repo.api.MissingSchemaSourceException: All available providers exhausted
at org.opendaylight.yangtools.yang.model.repo.util.AbstractSchemaRepository$1.create(AbstractSchemaRepository.java:83)
at com.google.common.util.concurrent.Futures$FallbackFuture$1.onFailure(Futures.java:471)
at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
at com.google.common.util.concurrent.Futures$ChainingListenableFuture.run(Futures.java:902)
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
at com.google.common.util.concurrent.Futures$FallbackFuture$1.onFailure(Futures.java:492)
at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
at com.google.common.util.concurrent.SettableFuture.setException(SettableFuture.java:68)
at org.opendaylight.controller.cluster.schema.provider.impl.RemoteSchemaProvider$2.onComplete(RemoteSchemaProvider.java:65)
at org.opendaylight.controller.cluster.schema.provider.impl.RemoteSchemaProvider$2.onComplete(RemoteSchemaProvider.java:58)
at akka.dispatch.OnComplete.internal(Future.scala:247)
at akka.dispatch.OnComplete.internal(Future.scala:245)
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:175)
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:172)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: org.opendaylight.yangtools.yang.model.repo.api.MissingSchemaSourceException: All available providers exhausted
at org.opendaylight.yangtools.yang.model.repo.util.AbstractSchemaRepository$1.create(AbstractSchemaRepository.java:83)
at com.google.common.util.concurrent.Futures$FallbackFuture$1.onFailure(Futures.java:471)
... 25 more
Caused by: org.opendaylight.yangtools.yang.model.repo.api.MissingSchemaSourceException: No providers registered for sourceSourceIdentifier [name=ietf-netconf-monitoring-extension@2013-12-10]
at org.opendaylight.yangtools.yang.model.repo.util.AbstractSchemaRepository.getSchemaSource(AbstractSchemaRepository.java:92)
at org.opendaylight.controller.cluster.schema.provider.impl.RemoteYangTextSourceProviderImpl.getYangTextSchemaSource(RemoteYangTextSourceProviderImpl.java:51)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at akka.actor.TypedActor$MethodCall.apply(TypedActor.scala:147)
at akka.actor.TypedActor$TypedActor$$anonfun$receive$1$$anonfun$applyOrElse$2.apply(TypedActor.scala:311)
at akka.actor.TypedActor$TypedActor.withContext(TypedActor.scala:299)
at akka.actor.TypedActor$TypedActor$$anonfun$receive$1.applyOrElse(TypedActor.scala:306)
at akka.actor.Actor$class.aroundReceive(Actor.scala:467)
at akka.actor.TypedActor$TypedActor.aroundReceive(TypedActor.scala:246)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
at akka.dispatch.Mailbox.run(Mailbox.scala:220)
... 5 more
2016-01-24 02:44:03,844 | WARN | ult-dispatcher-2 | NetconfDeviceCommunicator | 240 - org.opendaylight.netconf.sal-netconf-connector - 1.3.0.SNAPSHOT | RemoteDevice

{netconf-test-device}

: Session terminated Session closed

The above log piece was taken from https://jenkins.opendaylight.org/releng/view/netconf/job/netconf-csit-3node-clustering-only-beryllium/17/, from odl1_karaf.log. Look at logs around the test case Wait_For_Device_To_Become_Visible_For_Checker

I don't know yet if the crash is caused by configuring the device or by trying to see the device on the other node.