https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-1629 controller <p>The AbstractConcurrentDataBrokerTest, introduced in <a href="https://jira.opendaylight.org/browse/CONTROLLER-1578" title="Multi-threaded AbstractDataBrokerTest/s fail with TransactionCommitFailedException Caused by: java.lang.IllegalStateException: Store tree ... and candidate base ... differ." class="issue-link" data-issue-key="CONTROLLER-1578"><del>CONTROLLER-1578</del></a> to replace the AbstractDataBrokerTest, while working well in a number of tests I've been using it since, seems to cause deadlocks in some rare cases.</p> <p>For example, <a href="https://git.opendaylight.org/gerrit/#/c/54353/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/54353/</a> replaces AbstractDataBrokerTest in Daexim by AbstractConcurrentDataBrokerTest, and causes DataExportImportAppProviderTest to deadlock... <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/sad.png" height="16" width="16" align="absmiddle" alt="" border="0"/> Attached is a thread dump.</p> <p>Tali Ben-Meir seems to have hit a similar issue in a not-yet-published new test she's writing for project Federation, according to private email exchange with a thread dump that looks similar.</p> <p>Operating System: All<br/> Platform: All</p> CONTROLLER-1629

AbstractConcurrentDataBrokerTest causes deadlocks which AbstractDataBrokerTest did not

Bug Resolved Done Unassigned Michael Vorburger Wed, 5 Apr 2017 17:56:36 +0000 Tue, 25 Jul 2023 08:24:28 +0000 Thu, 13 Jul 2017 21:59:07 +0000 mdsal 0 2 <p>Attachment bug8163-jstack-1.txt has been added with description: jstack of deadlock in DataExportImportAppProviderTest when changing from @Deprecated AbstractDataBrokerTest to AbstractConcurrentDataBrokerTest</p> <p>The problem is the use of MoreExecutors$DirectExecutorService in the QueuedNotificationManager and the fact that the DataExportImportAppProvider does a blocking get on the submit Future in onDataTreeChanged. The solution is to not use a direct executor for publishing change notifications.</p> <p>&gt; DataExportImportAppProvider does a blocking get on the submit Future in onDataTreeChanged</p> <p>I may be too tired to see it - but where? DataExportImportAppProvider has a registerDataTreeChangeListener() in init() with an (anonymous / lambda) onDataTreeChanged which goes into ipcHandler() - but I don't see a get() in there? There definitely are many in other places though (c/55022), so I'm probably just not seeing what you're pointing to...</p> <p>&gt; The problem is the use of MoreExecutors$DirectExecutorService in the QueuedNotificationManager (...) The solution is to not use a direct executor for publishing change notifications.</p> <p>In Daexim or in controller? Oh wait, I'm seeing that this DirectExecutorService is created in AbstractDataBrokerTestCustomizer .. this sounds strangely familiar.. <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/wink.png" height="16" width="16" align="absmiddle" alt="" border="0"/> Is this the discussion we've already had in <a href="https://bugs.opendaylight.org/show_bug.cgi?id=7538#c9" class="external-link" target="_blank" rel="nofollow noopener">https://bugs.opendaylight.org/show_bug.cgi?id=7538#c9</a> ? I can try to pick that up from where left that off...</p> <p>PS, just FTR: This WORKED before switching from the AbstractDataBrokerTest to the AbstractConcurrentDataBrokerTest, and more importantly it works in "production" (with the "real" not test DataBroker) .. so it's a problem in controller utilities?</p> <p>(In reply to Michael Vorburger from comment #3)<br/> &gt; I may be too tired to see it - but where? DataExportImportAppProvider has a<br/> &gt; registerDataTreeChangeListener() in init() with an (anonymous / lambda)<br/> &gt; onDataTreeChanged which goes into ipcHandler() - but I don't see a get() in<br/> &gt; there?</p> <p>One code path is ipcHandler -&gt; updateNodeStatus. The code in this class really should be changed to be async and not block on Futures.</p> <p>&gt; Is this the discussion we've already had in<br/> &gt; <a href="https://bugs.opendaylight.org/show_bug.cgi?id=7538#c9" class="external-link" target="_blank" rel="nofollow noopener">https://bugs.opendaylight.org/show_bug.cgi?id=7538#c9</a> ? I can try to pick<br/> &gt; that up from where left that off...</p> <p>Yes. </p> <p>&gt; PS, just FTR: This WORKED before switching from the AbstractDataBrokerTest<br/> &gt; to the AbstractConcurrentDataBrokerTest, and more importantly it works in<br/> &gt; "production" (with the "real" not test DataBroker) .. so it's a problem in<br/> &gt; controller utilities?</p> <p>It works in production b/c there are no direct executors used in production core code. CDS uses akka actors and is all async. In this case the InMemoryDOMDataStore executor in the tests cannot not be a direct executor since the DataExportImportAppProvider code does blocking gets in its DTCL callback.</p> <p>Hi, investigating BUG-8335 It seems that Im hitting the same issue.</p> <p>dead lock is caused by AbstractConcurrentDataBrokerTest. <br/> Debugging I see that it hangs when AbstractRegistrationTree#removeRegistration<br/> is trying to take the log.</p> <p>Created a demo class and test to easily reproduce the issue.<br/> <a href="https://git.opendaylight.org/gerrit/#/c/56640/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/56640/</a></p> <p>moving the bug as confirmed, you can changed it you don't agree</p> <p>Is this fixed now?</p> <p>&gt; Is this fixed now?</p> <p>Nope, this one slipped through the cracks on my side; never got to this, due to other (internal, downstream) prios .. if you/someone would like to have a go at it, please don't let me stop you.. <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/wink.png" height="16" width="16" align="absmiddle" alt="" border="0"/> Otherwise I'll try to get to this in the next weeks, time permitting.</p> <p>(In reply to Michael Vorburger from comment #7)<br/> &gt; &gt; Is this fixed now?<br/> &gt; <br/> &gt; Nope, this one slipped through the cracks on my side; never got to this, due<br/> &gt; to other (internal, downstream) prios .. if you/someone would like to have a<br/> &gt; go at it, please don't let me stop you.. <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/wink.png" height="16" width="16" align="absmiddle" alt="" border="0"/> Otherwise I'll try to get to<br/> &gt; this in the next weeks, time permitting.</p> <p>I commented earlier on the solution based on the original thread dump - do not use a direct executor for publishing change notifications. I can push a patch for that but I suspect some downstream tests may fail intermittently if they're not expecting DTCL notification to be async.</p> <p>Submitted <a href="https://git.opendaylight.org/gerrit/#/c/60022/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/60022/</a> to use a thread pool executor instead of direct executor for DTCL notifications. This fixes the deadlock with the DemoBugTest in <a href="https://git.opendaylight.org/gerrit/#/c/56640/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/56640/</a>. </p> <p>This change may break existing tests if they expect DTCL notifications to be immediate but we can tackle those if/when they occur. If it becomes too problematic we can keep AbstractConcurrentDataBrokerTest using the direct executor and create another version that uses a thread pool although I'd rather not do that.</p> <p>&gt; create another version that uses a thread pool although I'd rather not do that</p> <p>FTR: <a href="https://git.opendaylight.org/gerrit/#/c/60022/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/60022/</a> ultimately ended up having to do this in order not to break backward compatibility, and introduces an option to choose whether or not you want the DataTreeChangeListenerExecutor to be multi threaded or not.</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/60312/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/60312/</a></p> Blocks BGPCEP-665 Development External issue ID 8163 External issue URL Rank 0|i02rr3: