[BGPCEP-200] RejectedExecutionException when pushing over 6500k routes into topology Created: 31/Mar/15  Updated: 03/Mar/19  Resolved: 04/May/15

Status: Resolved
Project: bgpcep
Component/s: BGP
Affects Version/s: Bugzilla Migration
Fix Version/s: Bugzilla Migration

Type: Bug
Reporter: Jozef Behran Assignee: Unassigned
Resolution: Cannot Reproduce Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: Linux
Platform: PC


External issue ID: 2927

 Description   

When sending 6500k or more routes through BGP into Helium SR3 ODL (could not specify in the header of this report), I get RejectedExecutionException error in the logs at a certain point (differs from test run to test run) and there is only between 5300 and 6300 paths stored in the topology. All other routes past this point are rejected with the "Could not submit the commit task - the commit queue capacity has been exceeded" error. I push only basic "next-hop" topology, not linkstate or anything complicated like this into ODL BGP. The exact exception from the logs is shown below.

Similar bug is here: https://bugs.opendaylight.org/show_bug.cgi?id=2255 but that one is from Lithium M2 and this one is in Helium SR3. As these codebases differ greatly, I believe the causes are most likely very different even when the effects are almost the same.

Here is the exception:

DeadlockDetectingListeningExecutorService{delegate=FastThreadPoolExecutor{Thread Prefix=WriteTxCommit, Current Thread Pool Size=1, Largest Thread Pool Size=1, Max Thread Pool Size=1, Current Queue Size=5000, Largest Queue Size=5000, Max Queue Size=5000, Active Thread Count=1, Completed Task Count=4020, Total Task Count=9021}}
java.util.concurrent.RejectedExecutionException: Task org.opendaylight.yangtools.util.concurrent.AsyncNotifyingListenableFutureTask$DelegatingAsyncNotifyingListenableFutureTask@208bd16 rejected from FastThreadPoolExecutor

{Thread Prefix=WriteTxCommit, Current Thread Pool Size=1, Largest Thread Pool Size=1, Max Thread Pool Size=1, Current Queue Size=5000, Largest Queue Size=5000, Max Queue Size=5000, Active Thread Count=1, Completed Task Count=4020, Total Task Count=9021}
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(Unknown Source)[:1.7.0_67]
at org.opendaylight.yangtools.util.concurrent.CountingRejectedExecutionHandler.rejectedExecution(CountingRejectedExecutionHandler.java:43)[78:org.opendaylight.yangtools.util:0.6.5.Helium-SR3]
at java.util.concurrent.ThreadPoolExecutor.reject(Unknown Source)[:1.7.0_67]
at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source)[:1.7.0_67]
at org.opendaylight.yangtools.util.concurrent.AsyncNotifyingListeningExecutorService.submit(AsyncNotifyingListeningExecutorService.java:123)[78:org.opendaylight.yangtools.util:0.6.5.Helium-SR3]
at org.opendaylight.yangtools.util.concurrent.DeadlockDetectingListeningExecutorService.submit(DeadlockDetectingListeningExecutorService.java:144)[78:org.opendaylight.yangtools.util:0.6.5.Helium-SR3]
at org.opendaylight.controller.md.sal.dom.broker.impl.DOMDataCommitCoordinatorImpl.submit(DOMDataCommitCoordinatorImpl.java:72)[165:org.opendaylight.controller.sal-broker-impl:1.1.3.Helium-SR3]
at org.opendaylight.controller.md.sal.dom.broker.impl.DOMDataBrokerTransactionChainImpl.submit(DOMDataBrokerTransactionChainImpl.java:94)[165:org.opendaylight.controller.sal-broker-impl:1.1.3.Helium-SR3]
at org.opendaylight.controller.md.sal.dom.broker.impl.DOMForwardedWriteTransaction.submit(DOMForwardedWriteTransaction.java:142)[165:org.opendaylight.controller.sal-broker-impl:1.1.3.Helium-SR3]
at org.opendaylight.controller.md.sal.binding.impl.AbstractWriteTransaction.doSubmit(AbstractWriteTransaction.java:169)[168:org.opendaylight.controller.sal-binding-broker-impl:1.1.3.Helium-SR3]
at org.opendaylight.controller.md.sal.binding.impl.BindingDataWriteTransactionImpl.submit(BindingDataWriteTransactionImpl.java:54)[168:org.opendaylight.controller.sal-binding-broker-impl:1.1.3.Helium-SR3]
at org.opendaylight.controller.md.sal.binding.impl.BindingTranslatedTransactionChain$2.submit(BindingTranslatedTransactionChain.java:82)[168:org.opendaylight.controller.sal-binding-broker-impl:1.1.3.Helium-SR3]
at org.opendaylight.protocol.bgp.rib.impl.AdjRIBsTransactionImpl.commit(AdjRIBsTransactionImpl.java:50)[178:org.opendaylight.bgpcep.bgp-rib-impl:0.3.4.Helium-SR3]
at org.opendaylight.protocol.bgp.rib.impl.RIBImpl.updateTables(RIBImpl.java:269)[178:org.opendaylight.bgpcep.bgp-rib-impl:0.3.4.Helium-SR3]
at org.opendaylight.protocol.bgp.rib.impl.BGPPeer.onMessage(BGPPeer.java:65)[178:org.opendaylight.bgpcep.bgp-rib-impl:0.3.4.Helium-SR3]
at org.opendaylight.protocol.bgp.rib.impl.BGPPeer.onMessage(BGPPeer.java:36)[178:org.opendaylight.bgpcep.bgp-rib-impl:0.3.4.Helium-SR3]
at org.opendaylight.protocol.bgp.rib.impl.BGPSessionImpl.handleMessage(BGPSessionImpl.java:189)[178:org.opendaylight.bgpcep.bgp-rib-impl:0.3.4.Helium-SR3]
at org.opendaylight.protocol.bgp.rib.impl.BGPSessionImpl.handleMessage(BGPSessionImpl.java:48)[178:org.opendaylight.bgpcep.bgp-rib-impl:0.3.4.Helium-SR3]
at org.opendaylight.protocol.framework.AbstractProtocolSession.channelRead0(AbstractProtocolSession.java:46)[103:org.opendaylight.controller.protocol-framework:0.5.3.Helium-SR3]
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)[95:io.netty.transport:4.0.23.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)[95:io.netty.transport:4.0.23.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)[95:io.netty.transport:4.0.23.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163)[99:io.netty.codec:4.0.23.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)[95:io.netty.transport:4.0.23.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)[95:io.netty.transport:4.0.23.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163)[99:io.netty.codec:4.0.23.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)[95:io.netty.transport:4.0.23.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)[95:io.netty.transport:4.0.23.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)[95:io.netty.transport:4.0.23.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)[95:io.netty.transport:4.0.23.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)[95:io.netty.transport:4.0.23.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)[95:io.netty.transport:4.0.23.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)[95:io.netty.transport:4.0.23.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)[95:io.netty.transport:4.0.23.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)[94:io.netty.common:4.0.23.Final]
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)[94:io.netty.common:4.0.23.Final]
at java.lang.Thread.run(Unknown Source)[:1.7.0_67]

The further writes are rejected with exceptions like this one (showing only one:

2015-03-31 12:03:20,848 | ERROR | oupCloseable-4-3 | RIBImpl | 178 - org.opendaylight.bgpcep.bgp-rib-impl - 0.3.4.Helium-SR3 | Broken chain in RIB KeyedInstanceIdentifier{targetType=interface org.opendaylight.yang.gen.v1.urn.opendaylight.params.xml.ns.yang.bgp.rib.rev130925.bgp.rib.Rib, path=[org.opendaylight.yang.gen.v1.urn.opendaylight.params.xml.ns.yang.bgp.rib.rev130925.BgpRib, org.opendaylight.yang.gen.v1.urn.opendaylight.params.xml.ns.yang.bgp.rib.rev130925.bgp.rib.Rib[key=RibKey [_id=Uri [_value=example-bgp-rib]]]]} transaction DOM-CHAIN-0-6474
TransactionCommitFailedException{message=Could not submit the commit task - the commit queue capacity has been exceeded., errorList=[RpcError [message=Could not submit the commit task - the commit queue capacity has been exceeded., severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=java.util.concurrent.RejectedExecutionException: Task org.opendaylight.yangtools.util.concurrent.AsyncNotifyingListenableFutureTask$DelegatingAsyncNotifyingListenableFutureTask@208bd16 rejected from FastThreadPoolExecutor{Thread Prefix=WriteTxCommit, Current Thread Pool Size=1, Largest Thread Pool Size=1, Max Thread Pool Size=1, Current Queue Size=5000, Largest Queue Size=5000, Max Queue Size=5000, Active Thread Count=1, Completed Task Count=4020, Total Task Count=9021}

]]}
at org.opendaylight.controller.md.sal.dom.broker.impl.DOMDataCommitCoordinatorImpl.submit(DOMDataCommitCoordinatorImpl.java:77)[165:org.opendaylight.controller.sal-broker-impl:1.1.3.Helium-SR3]
at org.opendaylight.controller.md.sal.dom.broker.impl.DOMDataBrokerTransactionChainImpl.submit(DOMDataBrokerTransactionChainImpl.java:94)[165:org.opendaylight.controller.sal-broker-impl:1.1.3.Helium-SR3]
at org.opendaylight.controller.md.sal.dom.broker.impl.DOMForwardedWriteTransaction.submit(DOMForwardedWriteTransaction.java:142)[165:org.opendaylight.controller.sal-broker-impl:1.1.3.Helium-SR3]
at org.opendaylight.controller.md.sal.binding.impl.AbstractWriteTransaction.doSubmit(AbstractWriteTransaction.java:169)[168:org.opendaylight.controller.sal-binding-broker-impl:1.1.3.Helium-SR3]
at org.opendaylight.controller.md.sal.binding.impl.BindingDataWriteTransactionImpl.submit(BindingDataWriteTransactionImpl.java:54)[168:org.opendaylight.controller.sal-binding-broker-impl:1.1.3.Helium-SR3]
at org.opendaylight.controller.md.sal.binding.impl.BindingTranslatedTransactionChain$2.submit(BindingTranslatedTransactionChain.java:82)[168:org.opendaylight.controller.sal-binding-broker-impl:1.1.3.Helium-SR3]
at org.opendaylight.protocol.bgp.rib.impl.AdjRIBsTransactionImpl.commit(AdjRIBsTransactionImpl.java:50)[178:org.opendaylight.bgpcep.bgp-rib-impl:0.3.4.Helium-SR3]
at org.opendaylight.protocol.bgp.rib.impl.RIBImpl.updateTables(RIBImpl.java:269)[178:org.opendaylight.bgpcep.bgp-rib-impl:0.3.4.Helium-SR3]
at org.opendaylight.protocol.bgp.rib.impl.BGPPeer.onMessage(BGPPeer.java:65)[178:org.opendaylight.bgpcep.bgp-rib-impl:0.3.4.Helium-SR3]
at org.opendaylight.protocol.bgp.rib.impl.BGPPeer.onMessage(BGPPeer.java:36)[178:org.opendaylight.bgpcep.bgp-rib-impl:0.3.4.Helium-SR3]
at org.opendaylight.protocol.bgp.rib.impl.BGPSessionImpl.handleMessage(BGPSessionImpl.java:189)[178:org.opendaylight.bgpcep.bgp-rib-impl:0.3.4.Helium-SR3]
at org.opendaylight.protocol.bgp.rib.impl.BGPSessionImpl.handleMessage(BGPSessionImpl.java:48)[178:org.opendaylight.bgpcep.bgp-rib-impl:0.3.4.Helium-SR3]
at org.opendaylight.protocol.framework.AbstractProtocolSession.channelRead0(AbstractProtocolSession.java:46)[103:org.opendaylight.controller.protocol-framework:0.5.3.Helium-SR3]
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)[95:io.netty.transport:4.0.23.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)[95:io.netty.transport:4.0.23.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)[95:io.netty.transport:4.0.23.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163)[99:io.netty.codec:4.0.23.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)[95:io.netty.transport:4.0.23.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)[95:io.netty.transport:4.0.23.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163)[99:io.netty.codec:4.0.23.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)[95:io.netty.transport:4.0.23.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)[95:io.netty.transport:4.0.23.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)[95:io.netty.transport:4.0.23.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)[95:io.netty.transport:4.0.23.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)[95:io.netty.transport:4.0.23.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)[95:io.netty.transport:4.0.23.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)[95:io.netty.transport:4.0.23.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)[95:io.netty.transport:4.0.23.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)[94:io.netty.common:4.0.23.Final]
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)[94:io.netty.common:4.0.23.Final]
at java.lang.Thread.run(Unknown Source)[:1.7.0_67]
Caused by: java.util.concurrent.RejectedExecutionException: Task org.opendaylight.yangtools.util.concurrent.AsyncNotifyingListenableFutureTask$DelegatingAsyncNotifyingListenableFutureTask@208bd16 rejected from FastThreadPoolExecutor

{Thread Prefix=WriteTxCommit, Current Thread Pool Size=1, Largest Thread Pool Size=1, Max Thread Pool Size=1, Current Queue Size=5000, Largest Queue Size=5000, Max Queue Size=5000, Active Thread Count=1, Completed Task Count=4020, Total Task Count=9021}

at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(Unknown Source)[:1.7.0_67]
at org.opendaylight.yangtools.util.concurrent.CountingRejectedExecutionHandler.rejectedExecution(CountingRejectedExecutionHandler.java:43)[78:org.opendaylight.yangtools.util:0.6.5.Helium-SR3]
at java.util.concurrent.ThreadPoolExecutor.reject(Unknown Source)[:1.7.0_67]
at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source)[:1.7.0_67]
at org.opendaylight.yangtools.util.concurrent.AsyncNotifyingListeningExecutorService.submit(AsyncNotifyingListeningExecutorService.java:123)[78:org.opendaylight.yangtools.util:0.6.5.Helium-SR3]
at org.opendaylight.yangtools.util.concurrent.DeadlockDetectingListeningExecutorService.submit(DeadlockDetectingListeningExecutorService.java:144)[78:org.opendaylight.yangtools.util:0.6.5.Helium-SR3]
at org.opendaylight.controller.md.sal.dom.broker.impl.DOMDataCommitCoordinatorImpl.submit(DOMDataCommitCoordinatorImpl.java:72)[165:org.opendaylight.controller.sal-broker-impl:1.1.3.Helium-SR3]
... 30 more



 Comments   
Comment by Dana Kutenicsova [ 04/May/15 ]

Needs to be re-tested with new RIB (I see pointers in the stacktrace to a code that doesn't exist anymore) to ensure this bug still exists or at least to update the stacktrace. Feel free to reopen when retested.

Generated at Wed Feb 07 19:12:19 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.