[BGPCEP-588] AddPath unit-tests sometimes hangs during execution Created: 21/Oct/16  Updated: 03/Mar/19  Resolved: 08/Nov/16

Status: Resolved
Project: bgpcep
Component/s: BGP
Affects Version/s: Bugzilla Migration
Fix Version/s: Bugzilla Migration

Type: Bug
Reporter: Ajay L Assignee: Ajay L
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: All
Platform: All

Attachments: Text File org.opendaylight.protocol.bgp.rib.impl.AddPathNPathsTest-output.txt    
External issue ID: 7006

 Description   

Seen sporadically both in CI and local environment

https://jenkins.opendaylight.org/releng/job/bgpcep-verify-boron-mvn33-openjdk8/352/console

07:04:46 Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 43.058 sec - in org.opendaylight.protocol.bgp.rib.impl.AddPathBasePathsTest
07:04:47 Running org.opendaylight.protocol.bgp.rib.impl.AddPathNPathsTest
12:49:15 Build timed out (after 360 minutes). Marking the build as failed.
12:49:15 [ssh-agent] Stopped.
12:49:15 Build was aborted

Console output is attached. HOLD_TIMER_EXPIRED message is reported indicating that somehow keepalives are not getting exchanged correctly (e.g. lock-up in sending/receiving messages)

 Comments   
Comment by Ajay L [ 21/Oct/16 ]

Attachment org.opendaylight.protocol.bgp.rib.impl.AddPathNPathsTest-output.txt has been added with description: Fail UT console output

Comment by Ajay L [ 25/Oct/16 ]

Thread dump when the test is in hung state is below. Channel#write() seems to get stuck if peer connection is closed at the same time and this causes EventLoopGroup#shutdownGracefully().awaitUninterruptibly() called from main() to block

Name: main
State: WAITING on io.netty.util.concurrent.DefaultPromise@6f5d2c39
Total blocked: 7 Total waited: 28

Stack trace:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:502)
io.netty.util.concurrent.DefaultPromise.awaitUninterruptibly(DefaultPromise.java:258)
io.netty.util.concurrent.DefaultPromise.awaitUninterruptibly(DefaultPromise.java:34)
org.opendaylight.protocol.bgp.rib.impl.BGPDispatcherImpl.close(BGPDispatcherImpl.java:126)
org.opendaylight.protocol.bgp.rib.impl.AbstractAddPathTest.tearDown(AbstractAddPathTest.java:194)
org.opendaylight.protocol.bgp.rib.impl.AddPathNPathsTest.tearDown(AddPathNPathsTest.java:34)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
org.junit.runners.ParentRunner.run(ParentRunner.java:309)
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)

Name: epollEventLoopGroup-5-2
State: BLOCKED on org.opendaylight.protocol.bgp.rib.impl.BGPSessionImpl@60d922f7 owned by: epollEventLoopGroup-5-5
Total blocked: 3 Total waited: 8

Stack trace:
org.opendaylight.protocol.bgp.rib.impl.BGPSessionImpl.write(BGPSessionImpl.java)
org.opendaylight.protocol.bgp.rib.impl.ChannelOutputLimiter.write(ChannelOutputLimiter.java:54)
org.opendaylight.protocol.bgp.rib.impl.AdjRibOutListener.processRouteChange(AdjRibOutListener.java:128)
org.opendaylight.protocol.bgp.rib.impl.AdjRibOutListener.processSupportedFamilyRoutes(AdjRibOutListener.java:102)
org.opendaylight.protocol.bgp.rib.impl.AdjRibOutListener.onDataTreeChanged(AdjRibOutListener.java:94)
org.opendaylight.controller.md.sal.dom.store.impl.InMemoryDOMStoreTreeChangePublisher$1.invokeListener(InMemoryDOMStoreTreeChangePublisher.java:38)
org.opendaylight.controller.md.sal.dom.store.impl.InMemoryDOMStoreTreeChangePublisher$1.invokeListener(InMemoryDOMStoreTreeChangePublisher.java:32)
org.opendaylight.yangtools.util.concurrent.QueuedNotificationManager.lambda$null$0(QueuedNotificationManager.java:125)
org.opendaylight.yangtools.util.concurrent.QueuedNotificationManager$$Lambda$77/284438421.accept(Unknown Source)
java.lang.Iterable.forEach(Iterable.java:75)
org.opendaylight.yangtools.util.concurrent.QueuedNotificationManager.lambda$new$1(QueuedNotificationManager.java:123)
org.opendaylight.yangtools.util.concurrent.QueuedNotificationManager$$Lambda$29/796240305.invokeListener(Unknown Source)
org.opendaylight.yangtools.util.concurrent.QueuedNotificationManager$NotificationTask.invokeListener(QueuedNotificationManager.java:432)
org.opendaylight.yangtools.util.concurrent.QueuedNotificationManager$NotificationTask.run(QueuedNotificationManager.java:420)
com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:299)
org.opendaylight.yangtools.util.concurrent.QueuedNotificationManager.runTask(QueuedNotificationManager.java:252)
org.opendaylight.yangtools.util.concurrent.QueuedNotificationManager.submitNotifications(QueuedNotificationManager.java:202)
org.opendaylight.yangtools.util.concurrent.QueuedNotificationManager.submitNotification(QueuedNotificationManager.java:167)
org.opendaylight.controller.md.sal.dom.store.impl.InMemoryDOMStoreTreeChangePublisher.notifyListeners(InMemoryDOMStoreTreeChangePublisher.java:55)
org.opendaylight.controller.sal.core.spi.data.AbstractDOMStoreTreeChangePublisher.notifyNode(AbstractDOMStoreTreeChangePublisher.java:118)
org.opendaylight.controller.sal.core.spi.data.AbstractDOMStoreTreeChangePublisher.notifyNode(AbstractDOMStoreTreeChangePublisher.java:125)
org.opendaylight.controller.sal.core.spi.data.AbstractDOMStoreTreeChangePublisher.notifyNode(AbstractDOMStoreTreeChangePublisher.java:125)
org.opendaylight.controller.sal.core.spi.data.AbstractDOMStoreTreeChangePublisher.notifyNode(AbstractDOMStoreTreeChangePublisher.java:125)
org.opendaylight.controller.sal.core.spi.data.AbstractDOMStoreTreeChangePublisher.notifyNode(AbstractDOMStoreTreeChangePublisher.java:125)
org.opendaylight.controller.sal.core.spi.data.AbstractDOMStoreTreeChangePublisher.notifyNode(AbstractDOMStoreTreeChangePublisher.java:125)
org.opendaylight.controller.sal.core.spi.data.AbstractDOMStoreTreeChangePublisher.notifyNode(AbstractDOMStoreTreeChangePublisher.java:125)
org.opendaylight.controller.sal.core.spi.data.AbstractDOMStoreTreeChangePublisher.notifyNode(AbstractDOMStoreTreeChangePublisher.java:125)
org.opendaylight.controller.sal.core.spi.data.AbstractDOMStoreTreeChangePublisher.notifyNode(AbstractDOMStoreTreeChangePublisher.java:125)
org.opendaylight.controller.sal.core.spi.data.AbstractDOMStoreTreeChangePublisher.lookupAndNotify(AbstractDOMStoreTreeChangePublisher.java:106)
org.opendaylight.controller.sal.core.spi.data.AbstractDOMStoreTreeChangePublisher.processCandidateTree(AbstractDOMStoreTreeChangePublisher.java:67)
org.opendaylight.controller.md.sal.dom.store.impl.InMemoryDOMStoreTreeChangePublisher.publishChange(InMemoryDOMStoreTreeChangePublisher.java:80)

  • locked org.opendaylight.controller.md.sal.dom.store.impl.InMemoryDOMStoreTreeChangePublisher@6c64c55
    org.opendaylight.controller.md.sal.dom.store.impl.InMemoryDOMDataStore.commit(InMemoryDOMDataStore.java:249)
  • locked org.opendaylight.controller.md.sal.dom.store.impl.InMemoryDOMDataStore@4f6ce954
    org.opendaylight.controller.md.sal.dom.store.impl.InMemoryDOMStoreThreePhaseCommitCohort.commit(InMemoryDOMStoreThreePhaseCommitCohort.java:105)
    org.opendaylight.controller.md.sal.dom.store.impl.ChainedTransactionCommitImpl.commit(ChainedTransactionCommitImpl.java:26)
    org.opendaylight.controller.md.sal.dom.broker.impl.CommitCoordinationTask.commitAll(CommitCoordinationTask.java:211)
    org.opendaylight.controller.md.sal.dom.broker.impl.CommitCoordinationTask.commitBlocking(CommitCoordinationTask.java:186)
    org.opendaylight.controller.md.sal.dom.broker.impl.CommitCoordinationTask.call(CommitCoordinationTask.java:65)
    org.opendaylight.controller.md.sal.dom.broker.impl.CommitCoordinationTask.call(CommitCoordinationTask.java:29)
    java.util.concurrent.FutureTask.run(FutureTask.java:266)
    com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:299)
    java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
    com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:58)
    org.opendaylight.controller.md.sal.dom.broker.impl.SerializedDOMDataBroker.submit(SerializedDOMDataBroker.java:70)
    org.opendaylight.controller.md.sal.dom.broker.impl.DOMDataBrokerTransactionChainImpl.submit(DOMDataBrokerTransactionChainImpl.java:97)
    org.opendaylight.controller.md.sal.dom.broker.impl.DOMForwardedWriteTransaction.submit(DOMForwardedWriteTransaction.java:144)
    org.opendaylight.protocol.bgp.rib.impl.LocRibWriter.onDataTreeChanged(LocRibWriter.java:143)
    org.opendaylight.controller.md.sal.dom.store.impl.InMemoryDOMStoreTreeChangePublisher$1.invokeListener(InMemoryDOMStoreTreeChangePublisher.java:38)
    org.opendaylight.controller.md.sal.dom.store.impl.InMemoryDOMStoreTreeChangePublisher$1.invokeListener(InMemoryDOMStoreTreeChangePublisher.java:32)
    org.opendaylight.yangtools.util.concurrent.QueuedNotificationManager.lambda$null$0(QueuedNotificationManager.java:125)
    org.opendaylight.yangtools.util.concurrent.QueuedNotificationManager$$Lambda$77/284438421.accept(Unknown Source)
    java.lang.Iterable.forEach(Iterable.java:75)
    org.opendaylight.yangtools.util.concurrent.QueuedNotificationManager.lambda$new$1(QueuedNotificationManager.java:123)
    org.opendaylight.yangtools.util.concurrent.QueuedNotificationManager$$Lambda$29/796240305.invokeListener(Unknown Source)
    org.opendaylight.yangtools.util.concurrent.QueuedNotificationManager$NotificationTask.invokeListener(QueuedNotificationManager.java:432)
    org.opendaylight.yangtools.util.concurrent.QueuedNotificationManager$NotificationTask.run(QueuedNotificationManager.java:420)
    com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:299)
    org.opendaylight.yangtools.util.concurrent.QueuedNotificationManager.runTask(QueuedNotificationManager.java:252)
    org.opendaylight.yangtools.util.concurrent.QueuedNotificationManager.submitNotifications(QueuedNotificationManager.java:202)
    org.opendaylight.yangtools.util.concurrent.QueuedNotificationManager.submitNotification(QueuedNotificationManager.java:167)
    org.opendaylight.controller.md.sal.dom.store.impl.InMemoryDOMStoreTreeChangePublisher.notifyListeners(InMemoryDOMStoreTreeChangePublisher.java:55)
    org.opendaylight.controller.sal.core.spi.data.AbstractDOMStoreTreeChangePublisher.notifyNode(AbstractDOMStoreTreeChangePublisher.java:118)
    org.opendaylight.controller.sal.core.spi.data.AbstractDOMStoreTreeChangePublisher.notifyNode(AbstractDOMStoreTreeChangePublisher.java:125)
    org.opendaylight.controller.sal.core.spi.data.AbstractDOMStoreTreeChangePublisher.notifyNode(AbstractDOMStoreTreeChangePublisher.java:125)
    org.opendaylight.controller.sal.core.spi.data.AbstractDOMStoreTreeChangePublisher.notifyNode(AbstractDOMStoreTreeChangePublisher.java:125)
    org.opendaylight.controller.sal.core.spi.data.AbstractDOMStoreTreeChangePublisher.notifyNode(AbstractDOMStoreTreeChangePublisher.java:129)
    org.opendaylight.controller.sal.core.spi.data.AbstractDOMStoreTreeChangePublisher.notifyNode(AbstractDOMStoreTreeChangePublisher.java:125)
    org.opendaylight.controller.sal.core.spi.data.AbstractDOMStoreTreeChangePublisher.notifyNode(AbstractDOMStoreTreeChangePublisher.java:125)
    org.opendaylight.controller.sal.core.spi.data.AbstractDOMStoreTreeChangePublisher.notifyNode(AbstractDOMStoreTreeChangePublisher.java:125)
    org.opendaylight.controller.sal.core.spi.data.AbstractDOMStoreTreeChangePublisher.notifyNode(AbstractDOMStoreTreeChangePublisher.java:125)
    org.opendaylight.controller.sal.core.spi.data.AbstractDOMStoreTreeChangePublisher.lookupAndNotify(AbstractDOMStoreTreeChangePublisher.java:106)
    org.opendaylight.controller.sal.core.spi.data.AbstractDOMStoreTreeChangePublisher.processCandidateTree(AbstractDOMStoreTreeChangePublisher.java:67)
    org.opendaylight.controller.md.sal.dom.store.impl.InMemoryDOMStoreTreeChangePublisher.publishChange(InMemoryDOMStoreTreeChangePublisher.java:80)
  • locked org.opendaylight.controller.md.sal.dom.store.impl.InMemoryDOMStoreTreeChangePublisher@6c64c55
    org.opendaylight.controller.md.sal.dom.store.impl.InMemoryDOMDataStore.commit(InMemoryDOMDataStore.java:249)
  • locked org.opendaylight.controller.md.sal.dom.store.impl.InMemoryDOMDataStore@4f6ce954
    org.opendaylight.controller.md.sal.dom.store.impl.InMemoryDOMStoreThreePhaseCommitCohort.commit(InMemoryDOMStoreThreePhaseCommitCohort.java:105)
    org.opendaylight.controller.md.sal.dom.store.impl.ChainedTransactionCommitImpl.commit(ChainedTransactionCommitImpl.java:26)
    org.opendaylight.controller.md.sal.dom.broker.impl.CommitCoordinationTask.commitAll(CommitCoordinationTask.java:211)
    org.opendaylight.controller.md.sal.dom.broker.impl.CommitCoordinationTask.commitBlocking(CommitCoordinationTask.java:186)
    org.opendaylight.controller.md.sal.dom.broker.impl.CommitCoordinationTask.call(CommitCoordinationTask.java:65)
    org.opendaylight.controller.md.sal.dom.broker.impl.CommitCoordinationTask.call(CommitCoordinationTask.java:29)
    java.util.concurrent.FutureTask.run(FutureTask.java:266)
    com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:299)
    java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
    com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:58)
    org.opendaylight.controller.md.sal.dom.broker.impl.SerializedDOMDataBroker.submit(SerializedDOMDataBroker.java:70)
    org.opendaylight.controller.md.sal.dom.broker.impl.DOMDataBrokerTransactionChainImpl.submit(DOMDataBrokerTransactionChainImpl.java:97)
    org.opendaylight.controller.md.sal.dom.broker.impl.DOMForwardedWriteTransaction.submit(DOMForwardedWriteTransaction.java:144)
    org.opendaylight.protocol.bgp.rib.impl.AdjRibInWriter.removePeer(AdjRibInWriter.java:234)
    org.opendaylight.protocol.bgp.rib.impl.BGPPeer.cleanup(BGPPeer.java:369)
    org.opendaylight.protocol.bgp.rib.impl.BGPPeer.releaseConnection(BGPPeer.java:413)
  • locked org.opendaylight.protocol.bgp.rib.impl.BGPPeer@25184d46
    org.opendaylight.protocol.bgp.rib.impl.BGPPeer.onSessionDown(BGPPeer.java:381)
    org.opendaylight.protocol.bgp.rib.impl.BGPSessionImpl.endOfInput(BGPSessionImpl.java:283)
  • locked org.opendaylight.protocol.bgp.rib.impl.BGPSessionImpl@5ca3b073
    org.opendaylight.protocol.bgp.rib.impl.BGPSessionImpl.channelInactive(BGPSessionImpl.java:481)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:233)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:219)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:212)
    io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:360)
    io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:325)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:233)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:219)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:212)
    io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:360)
    io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:325)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:233)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:219)
    io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:212)
    io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1275)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:233)
    io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:219)
    io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:872)
    io.netty.channel.AbstractChannel$AbstractUnsafe$7.run(AbstractChannel.java:679)
    io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:358)
    io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:307)
    io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)
    io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:145)
    java.lang.Thread.run(Thread.java:745)
Comment by Claudio David Gasparini [ 03/Nov/16 ]

https://git.opendaylight.org/gerrit/#/c/47885/

Comment by Claudio David Gasparini [ 08/Nov/16 ]

Boron
https://git.opendaylight.org/gerrit/#/c/48107/

Generated at Wed Feb 07 19:13:31 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.