[BGPCEP-200] RejectedExecutionException when pushing over 6500k routes into topology Created: 31/Mar/15 Updated: 03/Mar/19 Resolved: 04/May/15 |
|
| Status: | Resolved |
| Project: | bgpcep |
| Component/s: | BGP |
| Affects Version/s: | Bugzilla Migration |
| Fix Version/s: | Bugzilla Migration |
| Type: | Bug | ||
| Reporter: | Jozef Behran | Assignee: | Unassigned |
| Resolution: | Cannot Reproduce | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
Operating System: Linux |
||
| External issue ID: | 2927 |
| Description |
|
When sending 6500k or more routes through BGP into Helium SR3 ODL (could not specify in the header of this report), I get RejectedExecutionException error in the logs at a certain point (differs from test run to test run) and there is only between 5300 and 6300 paths stored in the topology. All other routes past this point are rejected with the "Could not submit the commit task - the commit queue capacity has been exceeded" error. I push only basic "next-hop" topology, not linkstate or anything complicated like this into ODL BGP. The exact exception from the logs is shown below. Similar bug is here: https://bugs.opendaylight.org/show_bug.cgi?id=2255 but that one is from Lithium M2 and this one is in Helium SR3. As these codebases differ greatly, I believe the causes are most likely very different even when the effects are almost the same. Here is the exception: DeadlockDetectingListeningExecutorService{delegate=FastThreadPoolExecutor{Thread Prefix=WriteTxCommit, Current Thread Pool Size=1, Largest Thread Pool Size=1, Max Thread Pool Size=1, Current Queue Size=5000, Largest Queue Size=5000, Max Queue Size=5000, Active Thread Count=1, Completed Task Count=4020, Total Task Count=9021}} at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(Unknown Source)[:1.7.0_67] at org.opendaylight.yangtools.util.concurrent.CountingRejectedExecutionHandler.rejectedExecution(CountingRejectedExecutionHandler.java:43)[78:org.opendaylight.yangtools.util:0.6.5.Helium-SR3] at java.util.concurrent.ThreadPoolExecutor.reject(Unknown Source)[:1.7.0_67] at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source)[:1.7.0_67] at org.opendaylight.yangtools.util.concurrent.AsyncNotifyingListeningExecutorService.submit(AsyncNotifyingListeningExecutorService.java:123)[78:org.opendaylight.yangtools.util:0.6.5.Helium-SR3] at org.opendaylight.yangtools.util.concurrent.DeadlockDetectingListeningExecutorService.submit(DeadlockDetectingListeningExecutorService.java:144)[78:org.opendaylight.yangtools.util:0.6.5.Helium-SR3] at org.opendaylight.controller.md.sal.dom.broker.impl.DOMDataCommitCoordinatorImpl.submit(DOMDataCommitCoordinatorImpl.java:72)[165:org.opendaylight.controller.sal-broker-impl:1.1.3.Helium-SR3] at org.opendaylight.controller.md.sal.dom.broker.impl.DOMDataBrokerTransactionChainImpl.submit(DOMDataBrokerTransactionChainImpl.java:94)[165:org.opendaylight.controller.sal-broker-impl:1.1.3.Helium-SR3] at org.opendaylight.controller.md.sal.dom.broker.impl.DOMForwardedWriteTransaction.submit(DOMForwardedWriteTransaction.java:142)[165:org.opendaylight.controller.sal-broker-impl:1.1.3.Helium-SR3] at org.opendaylight.controller.md.sal.binding.impl.AbstractWriteTransaction.doSubmit(AbstractWriteTransaction.java:169)[168:org.opendaylight.controller.sal-binding-broker-impl:1.1.3.Helium-SR3] at org.opendaylight.controller.md.sal.binding.impl.BindingDataWriteTransactionImpl.submit(BindingDataWriteTransactionImpl.java:54)[168:org.opendaylight.controller.sal-binding-broker-impl:1.1.3.Helium-SR3] at org.opendaylight.controller.md.sal.binding.impl.BindingTranslatedTransactionChain$2.submit(BindingTranslatedTransactionChain.java:82)[168:org.opendaylight.controller.sal-binding-broker-impl:1.1.3.Helium-SR3] at org.opendaylight.protocol.bgp.rib.impl.AdjRIBsTransactionImpl.commit(AdjRIBsTransactionImpl.java:50)[178:org.opendaylight.bgpcep.bgp-rib-impl:0.3.4.Helium-SR3] at org.opendaylight.protocol.bgp.rib.impl.RIBImpl.updateTables(RIBImpl.java:269)[178:org.opendaylight.bgpcep.bgp-rib-impl:0.3.4.Helium-SR3] at org.opendaylight.protocol.bgp.rib.impl.BGPPeer.onMessage(BGPPeer.java:65)[178:org.opendaylight.bgpcep.bgp-rib-impl:0.3.4.Helium-SR3] at org.opendaylight.protocol.bgp.rib.impl.BGPPeer.onMessage(BGPPeer.java:36)[178:org.opendaylight.bgpcep.bgp-rib-impl:0.3.4.Helium-SR3] at org.opendaylight.protocol.bgp.rib.impl.BGPSessionImpl.handleMessage(BGPSessionImpl.java:189)[178:org.opendaylight.bgpcep.bgp-rib-impl:0.3.4.Helium-SR3] at org.opendaylight.protocol.bgp.rib.impl.BGPSessionImpl.handleMessage(BGPSessionImpl.java:48)[178:org.opendaylight.bgpcep.bgp-rib-impl:0.3.4.Helium-SR3] at org.opendaylight.protocol.framework.AbstractProtocolSession.channelRead0(AbstractProtocolSession.java:46)[103:org.opendaylight.controller.protocol-framework:0.5.3.Helium-SR3] at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)[95:io.netty.transport:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)[95:io.netty.transport:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)[95:io.netty.transport:4.0.23.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163)[99:io.netty.codec:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)[95:io.netty.transport:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)[95:io.netty.transport:4.0.23.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163)[99:io.netty.codec:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)[95:io.netty.transport:4.0.23.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)[95:io.netty.transport:4.0.23.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)[95:io.netty.transport:4.0.23.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)[95:io.netty.transport:4.0.23.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)[95:io.netty.transport:4.0.23.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)[95:io.netty.transport:4.0.23.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)[95:io.netty.transport:4.0.23.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)[95:io.netty.transport:4.0.23.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)[94:io.netty.common:4.0.23.Final] at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)[94:io.netty.common:4.0.23.Final] at java.lang.Thread.run(Unknown Source)[:1.7.0_67] The further writes are rejected with exceptions like this one (showing only one: 2015-03-31 12:03:20,848 | ERROR | oupCloseable-4-3 | RIBImpl | 178 - org.opendaylight.bgpcep.bgp-rib-impl - 0.3.4.Helium-SR3 | Broken chain in RIB KeyedInstanceIdentifier{targetType=interface org.opendaylight.yang.gen.v1.urn.opendaylight.params.xml.ns.yang.bgp.rib.rev130925.bgp.rib.Rib, path=[org.opendaylight.yang.gen.v1.urn.opendaylight.params.xml.ns.yang.bgp.rib.rev130925.BgpRib, org.opendaylight.yang.gen.v1.urn.opendaylight.params.xml.ns.yang.bgp.rib.rev130925.bgp.rib.Rib[key=RibKey [_id=Uri [_value=example-bgp-rib]]]]} transaction DOM-CHAIN-0-6474 TransactionCommitFailedException{message=Could not submit the commit task - the commit queue capacity has been exceeded., errorList=[RpcError [message=Could not submit the commit task - the commit queue capacity has been exceeded., severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=java.util.concurrent.RejectedExecutionException: Task org.opendaylight.yangtools.util.concurrent.AsyncNotifyingListenableFutureTask$DelegatingAsyncNotifyingListenableFutureTask@208bd16 rejected from FastThreadPoolExecutor{Thread Prefix=WriteTxCommit, Current Thread Pool Size=1, Largest Thread Pool Size=1, Max Thread Pool Size=1, Current Queue Size=5000, Largest Queue Size=5000, Max Queue Size=5000, Active Thread Count=1, Completed Task Count=4020, Total Task Count=9021} ]]} at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(Unknown Source)[:1.7.0_67] |
| Comments |
| Comment by Dana Kutenicsova [ 04/May/15 ] |
|
Needs to be re-tested with new RIB (I see pointers in the stacktrace to a code that doesn't exist anymore) to ensure this bug still exists or at least to update the stacktrace. Feel free to reopen when retested. |