https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-1893 controller <p>A deadlock occurred between Application thread (reading config DS) and AKKA thread inside org.opendaylight.controller.cluster.access.client.AbstractClientConnection. Which seems to completely block all interactions with the Datastore and requires manual restart.</p> <p>Attached is:</p> <ul> <li>DEADLOCK stacktraces from jstack</li> <li>GC logs </li> <li>Snippet from karaf.log which is related to this issue (the rest of the logs did not contain anything of substance, just netconf disconnect and reconnect details)</li> </ul> <p> </p> <p>Initial analysis of jstack: Looks like an ABBA deadlock between 2 instances of AbstractClientSession. Thread A (Uniconfig-task-20) starts with invocation of ReadOnlyTransaction.close() which flows through 2 AbstractClientSession instances trying to acquire lock in each instance. Thread B (opendaylight-cluster-data-akka.actor.default-dispatcher-105) is triggerred by ClientActorBehavior.onReceiveCommand and due to timeout triggers "poison" path in the code, again passing through 2 instances of AbstractClientSession trying to acquire locks in the process (however the order is opposite). More details can be found in the stacktrace or in diagram:</p> <p><span class="image-wrap" style=""><img src="https://jira.opendaylight.org/secure/attachment/15210/15210_image-2019-05-07-16-01-17-465.png" style="border: 0px solid black" /></span></p> <p>I am really not sure why there are 2 instances of AbstractClientSession, nor have I any idea why the timeout/poison was triggerred (according to the log, it was 30 minutes inactive, but overall there is a lot of activity prior to this deadlock).</p> <p> </p> <p>Any idea how this deadlock could be fixed ? Is it an issue inside AbstractClientSession or perhaps some mismanagement on the application side ? I tried to simulate this deadlock in a unit test but so far no luck. </p> <p> </p> <p>ODL env info:</p> <ul> <li>version: Oxygen-SR2 based (But the code for AbstractClientSession is almost identical on master branch as well)</li> <li>deployment: Single node</li> <li>Uptime: approx 15 hours</li> <li>TELL based</li> <li>Xmx10G</li> <li>cores == 12</li> <li>Opendaylight was running netconf southbound + FRINX specific application <ul> <li>Netconf southbound was connected to 1000 devices and frequently reconnecting them due to faulty network (intentinally)</li> <li>FRINX app was listening for the mountpoints, reading config from them and then READing and WRITEing some data into Datastore</li> </ul> </li> </ul> CONTROLLER-1893

AbstractClientConnection deadlock

Bug Medium Resolved Done Robert Varga Maros Marsalek Tue, 7 May 2019 14:20:07 +0000 Thu, 16 May 2019 14:14:37 +0000 Thu, 16 May 2019 14:14:37 +0000 Oxygen SR2 Sodium Fluorine SR3 Neon SR2 clustering 0 2 <p>Regarding:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> karaf.log.7:2019-05-06T15:08:50,640 | ERROR | opendaylight-cluster-data-akka.actor.<span class="code-keyword">default</span>-dispatcher-105 | AbstractClientConnection | 203 - org.opendaylight.controller.cds-access-client - 1.3.2.Oxygen-SR2_4_2_1_rc3-frinxodl-SNAPSHOT | Queue ReconnectingClientConnection{client=ClientIdentifier{frontend=member-1-frontend-datastore-config, generation=0}, cookie=0, backend=ShardBackendInfo{actor=Actor[akka:<span class="code-comment">//opendaylight-cluster-data/user/shardmanager-config/member-1-shard-<span class="code-keyword">default</span>-config#2024575561], sessionId=0, version=BORON, maxMessages=1000, cookie=0, shard=<span class="code-keyword">default</span>, dataTree=present}} has not seen progress in 1964 seconds, failing all requests</span></pre> </div></div> <p> </p> <p>I don't really know what has happened here ... and what would be the consequences had the DEADLOCK not occurred. Any details on this would be also helpful.</p> <p> </p> <p>Thanks,</p> <p>Maros</p> <p> </p> <p>cds-access-client logs from INFO level would be nice, as we need to understand the lifecycle of the two connections. The connection being poisoned seems to be the successor in this case – which would indicate the originating transaction was allocated while it was still not there – which seems to be 1964 seconds by the reconnecting connection's accounting (which itself may inaccurate).</p> <p>If there is real activity going on, the connection should have reconnected and been replaced by a successor – which would have drained its queue, preventing the timer from being anything but a no-op.</p> <p> </p> <p>I would suggest picking up the reestablishment patches that went into Neon, which can help masking the issue by preventing poison from happening.</p> <p>This seems similar to <a href="https://jira.opendaylight.org/browse/CONTROLLER-1745" title="produce-transactions can get stuck when closing itemProducer" class="issue-link" data-issue-key="CONTROLLER-1745"><del>CONTROLLER-1745</del></a> – hence moving poisoning outside the lock should help.</p> <p>I can confirm that <a href="https://git.opendaylight.org/gerrit/#/c/81949/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/81949/</a> works. Verified by a unit test: <a href="https://git.opendaylight.org/gerrit/#/c/81979/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/81979/</a> as well as by running ODL instance.</p> <p>Ad "verified by running ODL instance": I think the same problem occurred again and the poison was triggerred. This time however there was no deadlock. The Datastore responds but with an error:</p> <p>   see attached stacktrace: stacktrace.poison.txt</p> <p>Robert, you mentioned "reestablishment patches" could help here. Could you give me a pointer please ?</p> <p>Also, I don't have "cds-access-client logs from INFO" yet, but I will try to acquire them from the next ODL deployment.</p> Development Rank 0|i03npz: