[BGPCEP-200] RejectedExecutionException when pushing over 6500k routes into topology

Sun, 3 Mar 2019 11:50:24 +0000

When sending 6500k or more routes through BGP into Helium SR3 ODL (could not specify in the header of this report), I get RejectedExecutionException error in the logs at a certain point (differs from test run to test run) and there is only between 5300 and 6300 paths stored in the topology. All other routes past this point are rejected with the "Could not submit the commit task - the commit queue capacity has been exceeded" error. I push only basic "next-hop" topology, not linkstate or anything complicated like this into ODL BGP. The exact exception from the logs is shown below.

Similar bug is here: https://bugs.opendaylight.org/show_bug.cgi?id=2255 but that one is from Lithium M2 and this one is in Helium SR3. As these codebases differ greatly, I believe the causes are most likely very different even when the effects are almost the same.

Here is the exception:

DeadlockDetectingListeningExecutorService{delegate=FastThreadPoolExecutor{Thread Prefix=WriteTxCommit, Current Thread Pool Size=1, Largest Thread Pool Size=1, Max Thread Pool Size=1, Current Queue Size=5000, Largest Queue Size=5000, Max Queue Size=5000, Active Thread Count=1, Completed Task Count=4020, Total Task Count=9021}}
java.util.concurrent.RejectedExecutionException: Task org.opendaylight.yangtools.util.concurrent.AsyncNotifyingListenableFutureTask$DelegatingAsyncNotifyingListenableFutureTask@208bd16 rejected from FastThreadPoolExecutor

{Thread Prefix=WriteTxCommit, Current Thread Pool Size=1, Largest Thread Pool Size=1, Max Thread Pool Size=1, Current Queue Size=5000, Largest Queue Size=5000, Max Queue Size=5000, Active Thread Count=1, Completed Task Count=4020, Total Task Count=9021}
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(Unknown Source)[:1.7.0_67]
at org.opendaylight.yangtools.util.concurrent.CountingRejectedExecutionHandler.rejectedExecution(CountingRejectedExecutionHandler.java:43)[78:org.opendaylight.yangtools.util:0.6.5.Helium-SR3]
at java.util.concurrent.ThreadPoolExecutor.reject(Unknown Source)[:1.7.0_67]
at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source)[:1.7.0_67]
at org.opendaylight.yangtools.util.concurrent.AsyncNotifyingListeningExecutorService.submit(AsyncNotifyingListeningExecutorService.java:123)[78:org.opendaylight.yangtools.util:0.6.5.Helium-SR3]
at org.opendaylight.yangtools.util.concurrent.DeadlockDetectingListeningExecutorService.submit(DeadlockDetectingListeningExecutorService.java:144)[78:org.opendaylight.yangtools.util:0.6.5.Helium-SR3]
at org.opendaylight.controller.md.sal.dom.broker.impl.DOMDataCommitCoordinatorImpl.submit(DOMDataCommitCoordinatorImpl.java:72)[165:org.opendaylight.controller.sal-broker-impl:1.1.3.Helium-SR3]
at org.opendaylight.controller.md.sal.dom.broker.impl.DOMDataBrokerTransactionChainImpl.submit(DOMDataBrokerTransactionChainImpl.java:94)[165:org.opendaylight.controller.sal-broker-impl:1.1.3.Helium-SR3]
at org.opendaylight.controller.md.sal.dom.broker.impl.DOMForwardedWriteTransaction.submit(DOMForwardedWriteTransaction.java:142)[165:org.opendaylight.controller.sal-broker-impl:1.1.3.Helium-SR3]
at org.opendaylight.controller.md.sal.binding.impl.AbstractWriteTransaction.doSubmit(AbstractWriteTransaction.java:169)[168:org.opendaylight.controller.sal-binding-broker-impl:1.1.3.Helium-SR3]
at org.opendaylight.controller.md.sal.binding.impl.BindingDataWriteTransactionImpl.submit(BindingDataWriteTransactionImpl.java:54)[168:org.opendaylight.controller.sal-binding-broker-impl:1.1.3.Helium-SR3]
at org.opendaylight.controller.md.sal.binding.impl.BindingTranslatedTransactionChain$2.submit(BindingTranslatedTransactionChain.java:82)[168:org.opendaylight.controller.sal-binding-broker-impl:1.1.3.Helium-SR3]
at org.opendaylight.protocol.bgp.rib.impl.AdjRIBsTransactionImpl.commit(AdjRIBsTransactionImpl.java:50)[178:org.opendaylight.bgpcep.bgp-rib-impl:0.3.4.Helium-SR3]
at org.opendaylight.protocol.bgp.rib.impl.RIBImpl.updateTables(RIBImpl.java:269)[178:org.opendaylight.bgpcep.bgp-rib-impl:0.3.4.Helium-SR3]
at org.opendaylight.protocol.bgp.rib.impl.BGPPeer.onMessage(BGPPeer.java:65)[178:org.opendaylight.bgpcep.bgp-rib-impl:0.3.4.Helium-SR3]
at org.opendaylight.protocol.bgp.rib.impl.BGPPeer.onMessage(BGPPeer.java:36)[178:org.opendaylight.bgpcep.bgp-rib-impl:0.3.4.Helium-SR3]
at org.opendaylight.protocol.bgp.rib.impl.BGPSessionImpl.handleMessage(BGPSessionImpl.java:189)[178:org.opendaylight.bgpcep.bgp-rib-impl:0.3.4.Helium-SR3]
at org.opendaylight.protocol.bgp.rib.impl.BGPSessionImpl.handleMessage(BGPSessionImpl.java:48)[178:org.opendaylight.bgpcep.bgp-rib-impl:0.3.4.Helium-SR3]
at org.opendaylight.protocol.framework.AbstractProtocolSession.channelRead0(AbstractProtocolSession.java:46)[103:org.opendaylight.controller.protocol-framework:0.5.3.Helium-SR3]
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)[95:io.netty.transport:4.0.23.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)[95:io.netty.transport:4.0.23.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)[95:io.netty.transport:4.0.23.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163)[99:io.netty.codec:4.0.23.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)[95:io.netty.transport:4.0.23.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)[95:io.netty.transport:4.0.23.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163)[99:io.netty.codec:4.0.23.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)[95:io.netty.transport:4.0.23.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)[95:io.netty.transport:4.0.23.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)[95:io.netty.transport:4.0.23.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)[95:io.netty.transport:4.0.23.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)[95:io.netty.transport:4.0.23.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)[95:io.netty.transport:4.0.23.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)[95:io.netty.transport:4.0.23.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)[95:io.netty.transport:4.0.23.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)[94:io.netty.common:4.0.23.Final]
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)[94:io.netty.common:4.0.23.Final]
at java.lang.Thread.run(Unknown Source)[:1.7.0_67]

The further writes are rejected with exceptions like this one (showing only one:

2015-03-31 12:03:20,848 | ERROR | oupCloseable-4-3 | RIBImpl | 178 - org.opendaylight.bgpcep.bgp-rib-impl - 0.3.4.Helium-SR3 | Broken chain in RIB KeyedInstanceIdentifier{targetType=interface org.opendaylight.yang.gen.v1.urn.opendaylight.params.xml.ns.yang.bgp.rib.rev130925.bgp.rib.Rib, path=[org.opendaylight.yang.gen.v1.urn.opendaylight.params.xml.ns.yang.bgp.rib.rev130925.BgpRib, org.opendaylight.yang.gen.v1.urn.opendaylight.params.xml.ns.yang.bgp.rib.rev130925.bgp.rib.Rib[key=RibKey [_id=Uri [_value=example-bgp-rib]]]]} transaction DOM-CHAIN-0-6474
TransactionCommitFailedException{message=Could not submit the commit task - the commit queue capacity has been exceeded., errorList=[RpcError [message=Could not submit the commit task - the commit queue capacity has been exceeded., severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=java.util.concurrent.RejectedExecutionException: Task org.opendaylight.yangtools.util.concurrent.AsyncNotifyingListenableFutureTask$DelegatingAsyncNotifyingListenableFutureTask@208bd16 rejected from FastThreadPoolExecutor{Thread Prefix=WriteTxCommit, Current Thread Pool Size=1, Largest Thread Pool Size=1, Max Thread Pool Size=1, Current Queue Size=5000, Largest Queue Size=5000, Max Queue Size=5000, Active Thread Count=1, Completed Task Count=4020, Total Task Count=9021}

]]}
at org.opendaylight.controller.md.sal.dom.broker.impl.DOMDataCommitCoordinatorImpl.submit(DOMDataCommitCoordinatorImpl.java:77)[165:org.opendaylight.controller.sal-broker-impl:1.1.3.Helium-SR3]
at org.opendaylight.controller.md.sal.dom.broker.impl.DOMDataBrokerTransactionChainImpl.submit(DOMDataBrokerTransactionChainImpl.java:94)[165:org.opendaylight.controller.sal-broker-impl:1.1.3.Helium-SR3]
at org.opendaylight.controller.md.sal.dom.broker.impl.DOMForwardedWriteTransaction.submit(DOMForwardedWriteTransaction.java:142)[165:org.opendaylight.controller.sal-broker-impl:1.1.3.Helium-SR3]
at org.opendaylight.controller.md.sal.binding.impl.AbstractWriteTransaction.doSubmit(AbstractWriteTransaction.java:169)[168:org.opendaylight.controller.sal-binding-broker-impl:1.1.3.Helium-SR3]
at org.opendaylight.controller.md.sal.binding.impl.BindingDataWriteTransactionImpl.submit(BindingDataWriteTransactionImpl.java:54)[168:org.opendaylight.controller.sal-binding-broker-impl:1.1.3.Helium-SR3]
at org.opendaylight.controller.md.sal.binding.impl.BindingTranslatedTransactionChain$2.submit(BindingTranslatedTransactionChain.java:82)[168:org.opendaylight.controller.sal-binding-broker-impl:1.1.3.Helium-SR3]
at org.opendaylight.protocol.bgp.rib.impl.AdjRIBsTransactionImpl.commit(AdjRIBsTransactionImpl.java:50)[178:org.opendaylight.bgpcep.bgp-rib-impl:0.3.4.Helium-SR3]
at org.opendaylight.protocol.bgp.rib.impl.RIBImpl.updateTables(RIBImpl.java:269)[178:org.opendaylight.bgpcep.bgp-rib-impl:0.3.4.Helium-SR3]
at org.opendaylight.protocol.bgp.rib.impl.BGPPeer.onMessage(BGPPeer.java:65)[178:org.opendaylight.bgpcep.bgp-rib-impl:0.3.4.Helium-SR3]
at org.opendaylight.protocol.bgp.rib.impl.BGPPeer.onMessage(BGPPeer.java:36)[178:org.opendaylight.bgpcep.bgp-rib-impl:0.3.4.Helium-SR3]
at org.opendaylight.protocol.bgp.rib.impl.BGPSessionImpl.handleMessage(BGPSessionImpl.java:189)[178:org.opendaylight.bgpcep.bgp-rib-impl:0.3.4.Helium-SR3]
at org.opendaylight.protocol.bgp.rib.impl.BGPSessionImpl.handleMessage(BGPSessionImpl.java:48)[178:org.opendaylight.bgpcep.bgp-rib-impl:0.3.4.Helium-SR3]
at org.opendaylight.protocol.framework.AbstractProtocolSession.channelRead0(AbstractProtocolSession.java:46)[103:org.opendaylight.controller.protocol-framework:0.5.3.Helium-SR3]
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)[95:io.netty.transport:4.0.23.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)[95:io.netty.transport:4.0.23.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)[95:io.netty.transport:4.0.23.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163)[99:io.netty.codec:4.0.23.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)[95:io.netty.transport:4.0.23.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)[95:io.netty.transport:4.0.23.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163)[99:io.netty.codec:4.0.23.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)[95:io.netty.transport:4.0.23.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)[95:io.netty.transport:4.0.23.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)[95:io.netty.transport:4.0.23.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)[95:io.netty.transport:4.0.23.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)[95:io.netty.transport:4.0.23.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)[95:io.netty.transport:4.0.23.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)[95:io.netty.transport:4.0.23.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)[95:io.netty.transport:4.0.23.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)[94:io.netty.common:4.0.23.Final]
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)[94:io.netty.common:4.0.23.Final]
at java.lang.Thread.run(Unknown Source)[:1.7.0_67]
Caused by: java.util.concurrent.RejectedExecutionException: Task org.opendaylight.yangtools.util.concurrent.AsyncNotifyingListenableFutureTask$DelegatingAsyncNotifyingListenableFutureTask@208bd16 rejected from FastThreadPoolExecutor

at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(Unknown Source)[:1.7.0_67]
at org.opendaylight.yangtools.util.concurrent.CountingRejectedExecutionHandler.rejectedExecution(CountingRejectedExecutionHandler.java:43)[78:org.opendaylight.yangtools.util:0.6.5.Helium-SR3]
at java.util.concurrent.ThreadPoolExecutor.reject(Unknown Source)[:1.7.0_67]
at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source)[:1.7.0_67]
at org.opendaylight.yangtools.util.concurrent.AsyncNotifyingListeningExecutorService.submit(AsyncNotifyingListeningExecutorService.java:123)[78:org.opendaylight.yangtools.util:0.6.5.Helium-SR3]
at org.opendaylight.yangtools.util.concurrent.DeadlockDetectingListeningExecutorService.submit(DeadlockDetectingListeningExecutorService.java:144)[78:org.opendaylight.yangtools.util:0.6.5.Helium-SR3]
at org.opendaylight.controller.md.sal.dom.broker.impl.DOMDataCommitCoordinatorImpl.submit(DOMDataCommitCoordinatorImpl.java:72)[165:org.opendaylight.controller.sal-broker-impl:1.1.3.Helium-SR3]
... 30 more

OpenDaylight JIRA

[BGPCEP-200] RejectedExecutionException when pushing over 6500k routes into topology