OpenDaylight JIRA

OpenDaylight JIRA https://jira.opendaylight.org This file is an XML representation of an issue en-us

8.20.10

820010

22-06-2022

[BGPCEP-665] InstructionDeployerImplTest hangs periodically https://jira.opendaylight.org/browse/BGPCEP-665 bgpcep <p>In autorelease-carbon job 266 and 275, this test hung for multiple hours (in 275 for at least 13 hours). We need to either fix this test or flag it with @Ignore to keep making progress.</p> <p>Operating System: All<br/> Platform: All</p> BGPCEP-665

InstructionDeployerImplTest hangs periodically

Bug Resolved Done Robert Varga Colin Dixon Fri, 28 Apr 2017 17:30:56 +0000 Sun, 3 Mar 2019 11:50:03 +0000 Wed, 9 Aug 2017 13:32:13 +0000 Bugzilla Migration Bugzilla Migration General 0 4 <p>Also see here:<br/> <a href="https://lists.opendaylight.org/pipermail/release/2017-April/010465.html" class="external-link" target="_blank" rel="nofollow noopener">https://lists.opendaylight.org/pipermail/release/2017-April/010465.html</a></p> <p>Removes the test in stable/carbon:<br/> <a href="https://git.opendaylight.org/gerrit/#/c/56282/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/56282/</a></p> <p>I'm completely open to other solutions.</p> <p>An alternate fix would be to exclude the test from autorelease-carbon:<br/> <a href="https://git.opendaylight.org/gerrit/#/c/56329/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/56329/</a></p> <p>Attachment org.opendaylight.bgpcep.programming.impl.InstructionDeployerImplTest-output.txt.gz has been added with description: surefire output for the test in autorelease-carbon job 285</p> <p>I merged the autorelease version. We'll see if it works.</p> <p>Anil Belur pointed out in the comments of the autorelease patch that it might not work because maven plugin configuration might not pass through when the build starts at the root autorelease pom.xml and then moves into <modules> which it isn't the parent of.</p> <p>I guess we'll likely find out in ~2 hours when either we'll be hung on the BGP test or not.</p> <p>As Anil Belur suspected, the patch to the autorelease Pom file did not actually suppress the test from running as you can see here:<br/> <a href="https://jenkins.opendaylight.org/releng/job/autorelease-release-carbon/286/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/job/autorelease-release-carbon/286/</a></p> <p>I guess we either need to find a way to put that config in the appropriate parent Pom (maybe as part of the autorelease process) or, more easily, get bgpcep to fix/disable the test.</p> <p>As per Thanh's recommendation:</p> <p>I don't think patch 56329 will help. Maven uses inheritance to determine what plugin configuration to use. pom.xml inside of autorelease is not a parent pom of any projects which means the configuration stored there won't get read in by the module in bgpcep.</p> <p>We either have to disable it at the module level or at odlparent which is the top level parent pom of all projects.</p> <p>Therefore, I am reverting the original patch <span class="error">[3.]</span> with <span class="error">[1.]</span> and retrigger autorelease carbon, since we also have the issue resolved with the project test in <span class="error">[2.]</span></p> <p><span class="error">[1.]</span> <a href="https://git.opendaylight.org/gerrit/56360" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/56360</a><br/> <span class="error">[2.]</span> <a href="https://git.opendaylight.org/gerrit/#/c/56282/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/56282/</a><br/> <span class="error">[3.]</span> <a href="https://git.opendaylight.org/gerrit/#/c/56329/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/56329/</a></p> <p>For what it's worth, this also fails periodically in the project's Merge Job:<br/> <a href="https://jenkins.opendaylight.org/releng/view/Merge-Carbon/job/bgpcep-merge-carbon/lastFailedBuild/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/Merge-Carbon/job/bgpcep-merge-carbon/lastFailedBuild/</a></p> <p>Shows that build #190 of of bgpcep-merge-carbon hung on this test.</p> <p><a href="https://jenkins.opendaylight.org/releng/view/Merge-Carbon/job/bgpcep-merge-carbon/190/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/Merge-Carbon/job/bgpcep-merge-carbon/190/</a></p> <p>Downgrading to critical as the test is now @Ignored by this patch:<br/> <a href="https://git.opendaylight.org/gerrit/#/c/56282/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/56282/</a></p> <p>dead lock is caused by AbstractConcurrentDataBrokerTest. <br/> Debugging I see that it hangs when AbstractRegistrationTree#removeRegistration<br/> is trying to take the log.</p> <p>Created a demo class and test to easily reproduce the issue.<br/> <a href="https://git.opendaylight.org/gerrit/#/c/56640/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/56640/</a></p> <p>Attachment dump.txt has been added with description: Thread dump from the hang</p> <p>Democlass locking is a problem here:</p> <p>"main" #1 prio=5 os_prio=0 tid=0x00007fe24000b000 nid=0x279e waiting on condition <span class="error">[0x00007fe248cb5000]</span><br/> java.lang.Thread.State: WAITING (parking)<br/> at sun.misc.Unsafe.park(Native Method)</p> <ul class="alternate" type="square"> <li>parking to wait for <0x00000000d9173978> (a java.util.concurrent.locks.StampedLock)<br/> at java.util.concurrent.locks.StampedLock.acquireWrite(StampedLock.java:1119)<br/> at java.util.concurrent.locks.StampedLock.writeLock(StampedLock.java:354)<br/> at java.util.concurrent.locks.StampedLock$WriteLockView.lock(StampedLock.java:899)<br/> at org.opendaylight.mdsal.dom.spi.AbstractRegistrationTree.removeRegistration(AbstractRegistrationTree.java:82)<br/> at org.opendaylight.controller.sal.core.spi.data.AbstractDOMStoreTreeChangePublisher.access$000(AbstractDOMStoreTreeChangePublisher.java:29)<br/> at org.opendaylight.controller.sal.core.spi.data.AbstractDOMStoreTreeChangePublisher$1.removeRegistration(AbstractDOMStoreTreeChangePublisher.java:78)<br/> at org.opendaylight.yangtools.concepts.AbstractRegistration.close(AbstractRegistration.java:43)<br/> at org.opendaylight.controller.md.sal.binding.impl.BindingDataTreeChangeListenerRegistration.removeRegistration(BindingDataTreeChangeListenerRegistration.java:26)<br/> at org.opendaylight.yangtools.concepts.AbstractRegistration.close(AbstractRegistration.java:43)<br/> at org.opendaylight.bgpcep.programming.impl.DemoClass.close(DemoClass.java:61)</li> <li>locked <0x00000000dc795ea0> (a org.opendaylight.bgpcep.programming.impl.DemoClass)<br/> at org.opendaylight.bgpcep.programming.impl.DemoBugTest.testInstructionDeployer(DemoBugTest.java:20)</li> </ul> <p>(blocked to make modification to listeners)</p> <p>"pool-2-thread-1" #14 prio=5 os_prio=0 tid=0x00007fe241006800 nid=0x27c6 waiting for monitor entry <span class="error">[0x00007fe21d036000]</span><br/> java.lang.Thread.State: BLOCKED (on object monitor)<br/> at org.opendaylight.bgpcep.programming.impl.DemoClass.onDataTreeChanged(DemoClass.java)</p> <ul class="alternate" type="square"> <li>waiting to lock <0x00000000dc795ea0> (a org.opendaylight.bgpcep.programming.impl.DemoClass)<br/> at org.opendaylight.controller.md.sal.binding.impl.BindingDOMDataTreeChangeListenerAdapter.onDataTreeChanged(BindingDOMDataTreeChangeListenerAdapter.java:41)<br/> at org.opendaylight.controller.md.sal.dom.store.impl.InMemoryDOMStoreTreeChangePublisher.lambda$static$0(InMemoryDOMStoreTreeChangePublisher.java:36)<br/> at org.opendaylight.controller.md.sal.dom.store.impl.InMemoryDOMStoreTreeChangePublisher$$Lambda$60/402847244.invokeListener(Unknown Source)<br/> at org.opendaylight.yangtools.util.concurrent.QueuedNotificationManager.lambda$null$0(QueuedNotificationManager.java:126)<br/> at org.opendaylight.yangtools.util.concurrent.QueuedNotificationManager$$Lambda$74/1248860520.accept(Unknown Source)<br/> at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:397)<br/> at org.opendaylight.yangtools.util.concurrent.QueuedNotificationManager.lambda$new$1(QueuedNotificationManager.java:124)<br/> at org.opendaylight.yangtools.util.concurrent.QueuedNotificationManager$$Lambda$59/60254480.invokeListener(Unknown Source)<br/> at org.opendaylight.yangtools.util.concurrent.QueuedNotificationManager$NotificationTask.invokeListener(QueuedNotificationManager.java:434)<br/> at org.opendaylight.yangtools.util.concurrent.QueuedNotificationManager$NotificationTask.run(QueuedNotificationManager.java:421)<br/> at com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:260)<br/> at org.opendaylight.yangtools.util.concurrent.QueuedNotificationManager.runTask(QueuedNotificationManager.java:253)<br/> at org.opendaylight.yangtools.util.concurrent.QueuedNotificationManager.submitNotifications(QueuedNotificationManager.java:203)<br/> at org.opendaylight.yangtools.util.concurrent.QueuedNotificationManager.submitNotification(QueuedNotificationManager.java:168)<br/> at org.opendaylight.controller.md.sal.dom.store.impl.InMemoryDOMStoreTreeChangePublisher.notifyListeners(InMemoryDOMStoreTreeChangePublisher.java:52)<br/> at org.opendaylight.controller.sal.core.spi.data.AbstractDOMStoreTreeChangePublisher.notifyNode(AbstractDOMStoreTreeChangePublisher.java:116)<br/> at org.opendaylight.controller.sal.core.spi.data.AbstractDOMStoreTreeChangePublisher.notifyNode(AbstractDOMStoreTreeChangePublisher.java:123)<br/> at org.opendaylight.controller.sal.core.spi.data.AbstractDOMStoreTreeChangePublisher.lookupAndNotify(AbstractDOMStoreTreeChangePublisher.java:104)<br/> at org.opendaylight.controller.sal.core.spi.data.AbstractDOMStoreTreeChangePublisher.processCandidateTree(AbstractDOMStoreTreeChangePublisher.java:65)<br/> at org.opendaylight.controller.md.sal.dom.store.impl.InMemoryDOMStoreTreeChangePublisher.publishChange(InMemoryDOMStoreTreeChangePublisher.java:77)</li> <li>locked <0x00000000d91738a8> (a org.opendaylight.controller.md.sal.dom.store.impl.InMemoryDOMStoreTreeChangePublisher)<br/> at org.opendaylight.controller.md.sal.dom.store.impl.InMemoryDOMDataStore.commit(InMemoryDOMDataStore.java:245)</li> <li>locked <0x00000000d91736a0> (a org.opendaylight.controller.md.sal.dom.store.impl.InMemoryDOMDataStore)</li> </ul> <p>(blocked to re-enter demo class)</p> <p>Which is a classic AB/BA deadlock. close() should not hold a lock (allowing DTCL to freely progress) when invoking registration.close(). It can take a lock prior to that to setup object state so that the DTCL turns into a no-op.</p> <p>master: <a href="https://git.opendaylight.org/gerrit/#/c/60317/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/60317/</a></p> Blocks CONTROLLER-1629 Development External issue ID 8335 External issue URL Issue Type ODL SR Target Milestone Rank 0|i02dov: