[BGPCEP-328] Immediate reconnects by BGPReconnectPromise

Sun, 3 Mar 2019 11:49:33 +0000

This Beryllium bug typically happens when ODL starts with BGP peer configured (in 41-*.xml, with small holdtimer) and ready. ODL ends up in a loop rapidly reconnecting, which pollutes karaf.log and makes debugging unrelated bugs very hard.
Eventually, connect is successful and system test passes, but karaf.log gets full before that.

BGPProtocolSessionPromise and BGPReconnectPromise should be reworked in such a way that no failure leads to immediate reconnect.

Here is a segment of karaf.log showing start of the loop:

{channel=[id: 0xe2c1bbe5, /10.25.2.9:45622 => /10.25.2.27:179], state=IDLE}

Subsequent iterations are caused like this:

2015-12-08 12:28:01,007 | INFO | oupCloseable-6-5 | AbstractBGPSessionNegotiator | 252 - org.opendaylight.bgpcep.bgp-rib-impl - 0.5.0.SNAPSHOT | Failed to send message Open [<snip>]
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)[:1.7.0_67]
at sun.nio.ch.SocketDispatcher.write(Unknown Source)[:1.7.0_67]
at sun.nio.ch.IOUtil.writeFromNativeBuffer(Unknown Source)[:1.7.0_67]
at sun.nio.ch.IOUtil.write(Unknown Source)[:1.7.0_67]
at sun.nio.ch.SocketChannelImpl.write(Unknown Source)[:1.7.0_67]
at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:287)[124:io.netty.transport:4.0.30.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:761)[124:io.netty.transport:4.0.30.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:311)[124:io.netty.transport:4.0.30.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:729)[124:io.netty.transport:4.0.30.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1127)[124:io.netty.transport:4.0.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:663)[124:io.netty.transport:4.0.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:644)[124:io.netty.transport:4.0.30.Final]
at io.netty.channel.ChannelOutboundHandlerAdapter.flush(ChannelOutboundHandlerAdapter.java:115)[124:io.netty.transport:4.0.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:663)[124:io.netty.transport:4.0.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:693)[124:io.netty.transport:4.0.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:681)[124:io.netty.transport:4.0.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:716)[124:io.netty.transport:4.0.30.Final]
at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:954)[124:io.netty.transport:4.0.30.Final]
at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:244)[124:io.netty.transport:4.0.30.Final]
at org.opendaylight.protocol.bgp.rib.impl.AbstractBGPSessionNegotiator.sendMessage(AbstractBGPSessionNegotiator.java:245)[252:org.opendaylight.bgpcep.bgp-rib-impl:0.5.0.SNAPSHOT]
at org.opendaylight.protocol.bgp.rib.impl.AbstractBGPSessionNegotiator.startNegotiation(AbstractBGPSessionNegotiator.java:111)[252:org.opendaylight.bgpcep.bgp-rib-impl:0.5.0.SNAPSHOT]
at org.opendaylight.protocol.bgp.rib.impl.AbstractBGPSessionNegotiator.channelActive(AbstractBGPSessionNegotiator.java:264)[252:org.opendaylight.bgpcep.bgp-rib-impl:0.5.0.SNAPSHOT]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:183)[124:io.netty.transport:4.0.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:169)[124:io.netty.transport:4.0.30.Final]
at io.netty.channel.ChannelInboundHandlerAdapter.channelActive(ChannelInboundHandlerAdapter.java:64)[124:io.netty.transport:4.0.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:183)[124:io.netty.transport:4.0.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:169)[124:io.netty.transport:4.0.30.Final]
at io.netty.channel.ChannelInboundHandlerAdapter.channelActive(ChannelInboundHandlerAdapter.java:64)[124:io.netty.transport:4.0.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:183)[124:io.netty.transport:4.0.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:169)[124:io.netty.transport:4.0.30.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelActive(DefaultChannelPipeline.java:817)[124:io.netty.transport:4.0.30.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:260)[124:io.netty.transport:4.0.30.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:290)[124:io.netty.transport:4.0.30.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528)[124:io.netty.transport:4.0.30.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)[124:io.netty.transport:4.0.30.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)[124:io.netty.transport:4.0.30.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)[124:io.netty.transport:4.0.30.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:110)[123:io.netty.common:4.0.30.Final]
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)[123:io.netty.common:4.0.30.Final]
at java.lang.Thread.run(Unknown Source)[:1.7.0_67]
2015-12-08 12:28:01,009 | DEBUG | oupCloseable-6-5 | AbstractBGPSessionNegotiator | 252 - org.opendaylight.bgpcep.bgp-rib-impl - 0.5.0.SNAPSHOT | Negotiation on channel [id: 0x2ea5cd45, /10.25.2.9:45623 :> /10.25.2.27:179] failed
java.io.IOException: Connection reset by peer
... (the same stacktrace as above)

OpenDaylight JIRA

[BGPCEP-328] Immediate reconnects by BGPReconnectPromise