[NETCONF-131] Netconf crash configuring device on clustered setup: "All available providers exhausted" Created: 25/Jan/16  Updated: 15/Mar/19  Resolved: 09/Feb/16

Status: Resolved
Project: netconf
Component/s: netconf
Affects Version/s: None
Fix Version/s: None

Type: Bug
Reporter: Jozef Behran Assignee: Tomas Cere
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: All
Platform: All


Attachments: File 5089-debug.tar.xz    
External issue ID: 5089

 Description   

Tested on stable/beryllium running on a three node cluster.

After trying to configure a Netconf device on clustered setup, the Netconf crashes with the following errors:

2016-01-24 02:44:03,837 | WARN | ult-dispatcher-2 | ClusteredNetconfDevice | 242 - org.opendaylight.netconf.topology - 1.0.0.SNAPSHOT | RemoteDevice

{netconf-test-device}: Unexpected error resolving device sources: org.opendaylight.yangtools.yang.model.repo.api.MissingSchemaSourceException: All available providers exhausted
2016-01-24 02:44:03,839 | ERROR | lt-dispatcher-27 | Futures$CombinedFuture | 39 - com.google.guava - 18.0.0 | input future failed.
org.opendaylight.yangtools.yang.model.repo.api.MissingSchemaSourceException: All available providers exhausted
at org.opendaylight.yangtools.yang.model.repo.util.AbstractSchemaRepository$1.create(AbstractSchemaRepository.java:83)
at com.google.common.util.concurrent.Futures$FallbackFuture$1.onFailure(Futures.java:471)
at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
at com.google.common.util.concurrent.Futures$ChainingListenableFuture.run(Futures.java:902)
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
at com.google.common.util.concurrent.Futures$FallbackFuture$1.onFailure(Futures.java:492)
at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
at com.google.common.util.concurrent.SettableFuture.setException(SettableFuture.java:68)
at org.opendaylight.controller.cluster.schema.provider.impl.RemoteSchemaProvider$2.onComplete(RemoteSchemaProvider.java:65)
at org.opendaylight.controller.cluster.schema.provider.impl.RemoteSchemaProvider$2.onComplete(RemoteSchemaProvider.java:58)
at akka.dispatch.OnComplete.internal(Future.scala:247)
at akka.dispatch.OnComplete.internal(Future.scala:245)
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:175)
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:172)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: org.opendaylight.yangtools.yang.model.repo.api.MissingSchemaSourceException: All available providers exhausted
at org.opendaylight.yangtools.yang.model.repo.util.AbstractSchemaRepository$1.create(AbstractSchemaRepository.java:83)
at com.google.common.util.concurrent.Futures$FallbackFuture$1.onFailure(Futures.java:471)
... 25 more
Caused by: org.opendaylight.yangtools.yang.model.repo.api.MissingSchemaSourceException: No providers registered for sourceSourceIdentifier [name=ietf-netconf-monitoring@2010-10-04]
at org.opendaylight.yangtools.yang.model.repo.util.AbstractSchemaRepository.getSchemaSource(AbstractSchemaRepository.java:92)
at org.opendaylight.controller.cluster.schema.provider.impl.RemoteYangTextSourceProviderImpl.getYangTextSchemaSource(RemoteYangTextSourceProviderImpl.java:51)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at akka.actor.TypedActor$MethodCall.apply(TypedActor.scala:147)
at akka.actor.TypedActor$TypedActor$$anonfun$receive$1$$anonfun$applyOrElse$2.apply(TypedActor.scala:311)
at akka.actor.TypedActor$TypedActor.withContext(TypedActor.scala:299)
at akka.actor.TypedActor$TypedActor$$anonfun$receive$1.applyOrElse(TypedActor.scala:306)
at akka.actor.Actor$class.aroundReceive(Actor.scala:467)
at akka.actor.TypedActor$TypedActor.aroundReceive(TypedActor.scala:246)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
at akka.dispatch.Mailbox.run(Mailbox.scala:220)
... 5 more
2016-01-24 02:44:03,837 | ERROR | ult-dispatcher-2 | NetconfDevice | 240 - org.opendaylight.netconf.sal-netconf-connector - 1.3.0.SNAPSHOT | RemoteDevice{netconf-test-device}

: Initialization in sal failed, disconnecting from device
org.opendaylight.yangtools.yang.model.repo.api.MissingSchemaSourceException: All available providers exhausted
at org.opendaylight.yangtools.yang.model.repo.util.AbstractSchemaRepository$1.create(AbstractSchemaRepository.java:83)
at com.google.common.util.concurrent.Futures$FallbackFuture$1.onFailure(Futures.java:471)
at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
at com.google.common.util.concurrent.Futures$ChainingListenableFuture.run(Futures.java:902)
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
at com.google.common.util.concurrent.Futures$FallbackFuture$1.onFailure(Futures.java:492)
at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
at com.google.common.util.concurrent.SettableFuture.setException(SettableFuture.java:68)
at org.opendaylight.controller.cluster.schema.provider.impl.RemoteSchemaProvider$2.onComplete(RemoteSchemaProvider.java:65)
at org.opendaylight.controller.cluster.schema.provider.impl.RemoteSchemaProvider$2.onComplete(RemoteSchemaProvider.java:58)
at akka.dispatch.OnComplete.internal(Future.scala:247)
at akka.dispatch.OnComplete.internal(Future.scala:245)
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:175)
at akka.dispatch.japi$CallbackBridge.apply(Future.scala:172)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: org.opendaylight.yangtools.yang.model.repo.api.MissingSchemaSourceException: All available providers exhausted
at org.opendaylight.yangtools.yang.model.repo.util.AbstractSchemaRepository$1.create(AbstractSchemaRepository.java:83)
at com.google.common.util.concurrent.Futures$FallbackFuture$1.onFailure(Futures.java:471)
... 25 more
Caused by: org.opendaylight.yangtools.yang.model.repo.api.MissingSchemaSourceException: No providers registered for sourceSourceIdentifier [name=ietf-netconf-monitoring-extension@2013-12-10]
at org.opendaylight.yangtools.yang.model.repo.util.AbstractSchemaRepository.getSchemaSource(AbstractSchemaRepository.java:92)
at org.opendaylight.controller.cluster.schema.provider.impl.RemoteYangTextSourceProviderImpl.getYangTextSchemaSource(RemoteYangTextSourceProviderImpl.java:51)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at akka.actor.TypedActor$MethodCall.apply(TypedActor.scala:147)
at akka.actor.TypedActor$TypedActor$$anonfun$receive$1$$anonfun$applyOrElse$2.apply(TypedActor.scala:311)
at akka.actor.TypedActor$TypedActor.withContext(TypedActor.scala:299)
at akka.actor.TypedActor$TypedActor$$anonfun$receive$1.applyOrElse(TypedActor.scala:306)
at akka.actor.Actor$class.aroundReceive(Actor.scala:467)
at akka.actor.TypedActor$TypedActor.aroundReceive(TypedActor.scala:246)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
at akka.dispatch.Mailbox.run(Mailbox.scala:220)
... 5 more
2016-01-24 02:44:03,844 | WARN | ult-dispatcher-2 | NetconfDeviceCommunicator | 240 - org.opendaylight.netconf.sal-netconf-connector - 1.3.0.SNAPSHOT | RemoteDevice

{netconf-test-device}

: Session terminated Session closed

The above log piece was taken from https://jenkins.opendaylight.org/releng/view/netconf/job/netconf-csit-3node-clustering-only-beryllium/17/, from odl1_karaf.log. Look at logs around the test case Wait_For_Device_To_Become_Visible_For_Checker

I don't know yet if the crash is caused by configuring the device or by trying to see the device on the other node.



 Comments   
Comment by Jozef Behran [ 26/Jan/16 ]

Possible causes:

1. Test-tool changed and now sends models that don't work. Retest with an older
test-tool.
2. Something in the parser broke down. Switch on debug logging and rerun the test.

Comment by Jozef Behran [ 28/Jan/16 ]

Attachment 5089-debug.tar.xz has been added with description: Debug logs from a test run that exhibited the bug.

Comment by Tomas Cere [ 08/Feb/16 ]

https://git.opendaylight.org/gerrit/#/c/34259/

Generated at Wed Feb 07 20:14:15 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.