[CONTROLLER-701] When connecting netconf connector to several 1000s of devices, datastore starts refusing commits

Tue, 25 Jul 2023 08:24:06 +0000

2014-08-18 16:52:50 CEST [com.google.common.util.concurrent.Futures$ImmediateFuture] SEVERE com.google.common.util.concurrent.Futures$ImmediateFuture addListener RuntimeException while executing runnable com.google.common.util.concurrent.Futures$4@15506352 with executor com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService@50a8a76c
java.lang.IllegalStateException: RemoteDevice

{22841-sim-device}

Transaction(update) not committed correctly
at org.opendaylight.controller.sal.connect.netconf.sal.NetconfDeviceDatastoreAdapter$1.onFailure(NetconfDeviceDatastoreAdapter.java:117)
at com.google.common.util.concurrent.Futures$4.run(Futures.java:1140)
at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:293)
at com.google.common.util.concurrent.Futures$ImmediateFuture.addListener(Futures.java:99)
at com.google.common.util.concurrent.Futures.addCallback(Futures.java:1152)
at com.google.common.util.concurrent.Futures.addCallback(Futures.java:1088)
at org.opendaylight.controller.sal.connect.netconf.sal.NetconfDeviceDatastoreAdapter.commitTransaction(NetconfDeviceDatastoreAdapter.java:108)
at org.opendaylight.controller.sal.connect.netconf.sal.NetconfDeviceDatastoreAdapter.updateDeviceState(NetconfDeviceDatastoreAdapter.java:64)
at org.opendaylight.controller.sal.connect.netconf.sal.NetconfDeviceSalFacade.onDeviceConnected(NetconfDeviceSalFacade.java:94)
at org.opendaylight.controller.sal.connect.netconf.sal.NetconfDeviceSalFacade.onDeviceConnected(NetconfDeviceSalFacade.java:38)
at org.opendaylight.controller.sal.connect.netconf.NetconfDevice.handleSalInitializationSuccess(NetconfDevice.java:126)
at org.opendaylight.controller.sal.connect.netconf.NetconfDevice.access$800(NetconfDevice.java:54)
at org.opendaylight.controller.sal.connect.netconf.NetconfDevice$RecursiveSchemaSetup$1.onSuccess(NetconfDevice.java:316)
at org.opendaylight.controller.sal.connect.netconf.NetconfDevice$RecursiveSchemaSetup$1.onSuccess(NetconfDevice.java:311)
at com.google.common.util.concurrent.Futures$4.run(Futures.java:1149)
at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:293)
at com.google.common.util.concurrent.Futures$ImmediateFuture.addListener(Futures.java:99)
at com.google.common.util.concurrent.Futures.addCallback(Futures.java:1152)
at com.google.common.util.concurrent.Futures.addCallback(Futures.java:1088)
at org.opendaylight.controller.sal.connect.netconf.NetconfDevice$RecursiveSchemaSetup.setUpSchema(NetconfDevice.java:340)
at org.opendaylight.controller.sal.connect.netconf.NetconfDevice$RecursiveSchemaSetup.run(NetconfDevice.java:294)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: TransactionCommitFailedException{message=Could not submit the commit task - the commit queue capacity has been exceeded., errorList=[RpcError [message=Could not submit the commit task - the commit queue capacity has been exceeded., severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=java.util.concurrent.RejectedExecutionException: Task org.opendaylight.yangtools.util.concurrent.AsyncNotifyingListenableFutureTask@7b5d9b63 rejected from FastThreadPoolExecutor

{Thread Prefix=WriteTxCommit, Current Thread Pool Size=1, Largest Thread Pool Size=1, Max Thread Pool Size=1, Current Queue Size=5000, Max Queue Size=5000, Active Thread Count=1, Completed Task Count=4214, Total Task Count=9215}]]}
at org.opendaylight.controller.md.sal.dom.broker.impl.DOMDataCommitCoordinatorImpl.submit(DOMDataCommitCoordinatorImpl.java:97)
at org.opendaylight.controller.md.sal.dom.broker.impl.DOMDataBrokerImpl.submit(DOMDataBrokerImpl.java:83)
at org.opendaylight.controller.md.sal.dom.broker.impl.DOMForwardedWriteTransaction.submit(DOMForwardedWriteTransaction.java:137)
at org.opendaylight.controller.md.sal.binding.impl.AbstractWriteTransaction.doSubmit(AbstractWriteTransaction.java:148)
at org.opendaylight.controller.md.sal.binding.impl.BindingDataWriteTransactionImpl.submit(BindingDataWriteTransactionImpl.java:83)
at org.opendaylight.controller.sal.connect.netconf.sal.NetconfDeviceDatastoreAdapter.commitTransaction(NetconfDeviceDatastoreAdapter.java:106)
... 19 more
Caused by: java.util.concurrent.RejectedExecutionException: Task org.opendaylight.yangtools.util.concurrent.AsyncNotifyingListenableFutureTask@7b5d9b63 rejected from FastThreadPoolExecutor{Thread Prefix=WriteTxCommit, Current Thread Pool Size=1, Largest Thread Pool Size=1, Max Thread Pool Size=1, Current Queue Size=5000, Max Queue Size=5000, Active Thread Count=1, Completed Task Count=4214, Total Task Count=9215}

at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
at org.opendaylight.yangtools.util.concurrent.AsyncNotifyingListeningExecutorService.submit(AsyncNotifyingListeningExecutorService.java:123)
at org.opendaylight.yangtools.util.concurrent.DeadlockDetectingListeningExecutorService.submit(DeadlockDetectingListeningExecutorService.java:86)
at org.opendaylight.controller.md.sal.dom.broker.impl.DOMDataCommitCoordinatorImpl.submit(DOMDataCommitCoordinatorImpl.java:93)
... 24 more

OpenDaylight JIRA

[CONTROLLER-701] When connecting netconf connector to several 1000s of devices, datastore starts refusing commits