[CONTROLLER-957] Executor pipeline gets full in scale testing of BGP and PCEP

Fri, 23 Oct 2015 15:12:12 +0000

Scale tests for BGP/PCEP take a long time, because executor gets filled up on requests and gives exception:

014-10-21 12:20:53.506 UTC [nettyThreadgroupModule$NioEventLoopGroupCloseable-7-7] ERROR o.o.c.m.s.d.b.i.DOMDataCommitCoordinatorImpl - The commit executor's queue is full - submit task was rejected.
DeadlockDetectingListeningExecutorService{delegate=FastThreadPoolExecutor{Thread Prefix=WriteTxCommit, Current Thread Pool Size=1, Largest Thread Pool Size=1, Max Thread Pool Size=1, Current Queue Size=5000, Largest Queue Size=5000, Max Queue Size=5000, Active Thread Count=1, Completed Task Count=117528, Total Task Count=122529}}
java.util.concurrent.RejectedExecutionException: Task org.opendaylight.yangtools.util.concurrent.AsyncNotifyingListenableFutureTask$DelegatingAsyncNotifyingListenableFutureTask@63c11a21 rejected from FastThreadPoolExecutor

{Thread Prefix=WriteTxCommit, Current Thread Pool Size=1, Largest Thread Pool Size=1, Max Thread Pool Size=1, Current Queue Size=5000, Largest Queue Size=5000, Max Queue Size=5000, Active Thread Count=1, Completed Task Count=117528, Total Task Count=122529}

at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(Unknown Source) ~[na:1.7.0_67]
at org.opendaylight.yangtools.util.concurrent.CountingRejectedExecutionHandler.rejectedExecution(CountingRejectedExecutionHandler.java:43) ~[bundlefile:na]
at java.util.concurrent.ThreadPoolExecutor.reject(Unknown Source) ~[na:1.7.0_67]
at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source) ~[na:1.7.0_67]
at org.opendaylight.yangtools.util.concurrent.AsyncNotifyingListeningExecutorService.submit(AsyncNotifyingListeningExecutorService.java:123) ~[bundlefile:na]
at org.opendaylight.yangtools.util.concurrent.DeadlockDetectingListeningExecutorService.submit(DeadlockDetectingListeningExecutorService.java:144) ~[bundlefile:na]
at org.opendaylight.controller.md.sal.dom.broker.impl.DOMDataCommitCoordinatorImpl.submit(DOMDataCommitCoordinatorImpl.java:72) ~[bundlefile:na]
at org.opendaylight.controller.md.sal.dom.broker.impl.DOMDataBrokerTransactionChainImpl.submit(DOMDataBrokerTransactionChainImpl.java:94) [bundlefile:na]
at org.opendaylight.controller.md.sal.dom.broker.impl.DOMForwardedWriteTransaction.submit(DOMForwardedWriteTransaction.java:142) [bundlefile:na]
at org.opendaylight.controller.md.sal.binding.impl.AbstractWriteTransaction.doSubmit(AbstractWriteTransaction.java:142) [bundlefile:na]
at org.opendaylight.controller.md.sal.binding.impl.BindingDataWriteTransactionImpl.submit(BindingDataWriteTransactionImpl.java:83) [bundlefile:na]
at org.opendaylight.controller.md.sal.binding.impl.BindingTranslatedTransactionChain$2.submit(BindingTranslatedTransactionChain.java:82) [bundlefile:na]
at org.opendaylight.protocol.bgp.rib.impl.AdjRIBsTransactionImpl.commit(AdjRIBsTransactionImpl.java:50) [bundlefile:na]
at org.opendaylight.protocol.bgp.rib.impl.RIBImpl.updateTables(RIBImpl.java:270) [bundlefile:na]
at org.opendaylight.protocol.bgp.rib.impl.BGPPeer.onMessage(BGPPeer.java:78) [bundlefile:na]
at org.opendaylight.protocol.bgp.rib.impl.BGPPeer.onMessage(BGPPeer.java:44) [bundlefile:na]
at org.opendaylight.protocol.bgp.rib.impl.BGPSessionImpl.handleMessage(BGPSessionImpl.java:199) [bundlefile:na]
at org.opendaylight.protocol.bgp.rib.impl.BGPSessionImpl.handleMessage(BGPSessionImpl.java:50) [bundlefile:na]
at org.opendaylight.protocol.framework.AbstractProtocolSession.channelRead0(AbstractProtocolSession.java:46) [bundlefile:na]
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [bundlefile:4.0.23.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) [bundlefile:4.0.23.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) [bundlefile:4.0.23.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163) [bundlefile:4.0.23.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) [bundlefile:4.0.23.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) [bundlefile:4.0.23.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163) [bundlefile:4.0.23.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) [bundlefile:4.0.23.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) [bundlefile:4.0.23.Final]
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) [bundlefile:4.0.23.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) [bundlefile:4.0.23.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319) [bundlefile:4.0.23.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787) [bundlefile:4.0.23.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130) [bundlefile:4.0.23.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511) [bundlefile:4.0.23.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468) [bundlefile:4.0.23.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382) [bundlefile:4.0.23.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) [bundlefile:4.0.23.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) [bundlefile:4.0.23.Final]
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) [bundlefile:4.0.23.Final]
at java.lang.Thread.run(Unknown Source) [na:1.7.0_67]
2014-10-21 12:20:53.512 UTC [nettyThreadgroupModule$NioEventLoopGroupCloseable-7-7] ERROR o.o.protocol.bgp.rib.impl.RIBImpl - Broken chain in RIB KeyedInstanceIdentifier

{targetType=interface org.opendaylight.yang.gen.v1.urn.opendaylight.params.xml.ns.yang.bgp.rib.rev130925.bgp.rib.Rib, path=[org.opendaylight.yang.gen.v1.urn.opendaylight.params.xml.ns.yang.bgp.rib.rev130925.BgpRib, org.opendaylight.yang.gen.v1.urn.opendaylight.params.xml.ns.yang.bgp.rib.rev130925.bgp.rib.Rib[key=RibKey [_id=Uri [_value=example-bgp-rib]]]]}

transaction DOM-CHAIN-3-62700
org.opendaylight.controller.md.sal.common.api.data.TransactionCommitFailedException: Could not submit the commit task - the commit queue capacity has been exceeded.
at org.opendaylight.controller.md.sal.dom.broker.impl.DOMDataCommitCoordinatorImpl.submit(DOMDataCommitCoordinatorImpl.java:77) ~[na:na]

Increasing the max queue size worked as workaround.

OpenDaylight JIRA

[CONTROLLER-957] Executor pipeline gets full in scale testing of BGP and PCEP