https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-1620 controller <p>Scale testing on latest Boron code with 3-node cluster as part of an openstack solution. <br/> 200 OVSs are connected, and 1 VM per OVS.<br/> ODL3 crashed because of a StackOverflowError:<br/> 2017-03-22T21:00:38.704+0200: <span class="error">&#91;GC pause (G1 Evacuation Pause) (young) 21G-&gt;8029M(24G), 0.0591195 secs&#93;</span><br/> Uncaught error from thread <span class="error">&#91;opendaylight-cluster-data-akka.actor.default-dispatcher-49&#93;</span> shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem<span class="error">&#91;opendaylight-cluster-data&#93;</span><br/> java.lang.StackOverflowError<br/> at org.opendaylight.yangtools.yang.data.impl.schema.nodes.AbstractImmutableDataContainerNode.getValue(AbstractImmutableDataContainerNode.java:38)<br/> at org.opendaylight.yangtools.yang.data.impl.schema.builder.impl.ImmutableMapEntryNodeBuilder.&lt;init&gt;(ImmutableMapEntryNodeBuilder.java:45)<br/> at org.opendaylight.yangtools.yang.data.impl.schema.builder.impl.ImmutableMapEntryNodeBuilder.create(ImmutableMapEntryNodeBuilder.java:61)<br/> at org.opendaylight.yangtools.yang.data.impl.schema.tree.ListEntryModificationStrategy.createBuilder(ListEntryModificationStrategy.java:64)<br/> at org.opendaylight.yangtools.yang.data.impl.schema.tree.ListEntryModificationStrategy.createBuilder(ListEntryModificationStrategy.java:22)<br/> ....</p> <p>This can be seen in karaf.out</p> <p>Operating System: All<br/> Platform: All</p> CONTROLLER-1620

StackOverflowError while shard is committing

Bug Resolved Done Robert Varga Guy Sela Thu, 23 Mar 2017 09:51:02 +0000 Tue, 25 Jul 2023 08:24:27 +0000 Mon, 15 May 2017 22:44:43 +0000 clustering 0 5 <p>Attachment odl3.tar.gz has been added with description: ODL3 logs</p> <p>I assume we can ignore the NPEs from neutron for now.</p> <p>The interesting part of the stack trace is actually:</p> <p> at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractDataTreeTip.prepare(AbstractDataTreeTip.java:53)<br/> at org.opendaylight.controller.cluster.datastore.ShardDataTree.startPreCommit(ShardDataTree.java:577)<br/> at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.preCommit(SimpleShardDataTreeCohort.java:91)<br/> at org.opendaylight.controller.cluster.datastore.CohortEntry.preCommit(CohortEntry.java:102)<br/> at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.doCommit(ShardCommitCoordinator.java:296)<br/> at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.access$200(ShardCommitCoordinator.java:49)<br/> at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator$2.onSuccess(ShardCommitCoordinator.java:243)<br/> at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator$2.onSuccess(ShardCommitCoordinator.java:237)<br/> at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.successfulCanCommit(SimpleShardDataTreeCohort.java:145)<br/> at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextTransaction(ShardDataTree.java:526)<br/> at org.opendaylight.controller.cluster.datastore.ShardDataTree.finishCommit(ShardDataTree.java:623)<br/> at org.opendaylight.controller.cluster.datastore.ShardDataTree.startCommit(ShardDataTree.java:635)<br/> at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.commit(SimpleShardDataTreeCohort.java:132)<br/> at org.opendaylight.controller.cluster.datastore.CohortEntry.commit(CohortEntry.java:106)<br/> at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.finishCommit(ShardCommitCoordinator.java:316)<br/> at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.access$400(ShardCommitCoordinator.java:49)<br/> at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator$3.onSuccess(ShardCommitCoordinator.java:299)<br/> at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator$3.onSuccess(ShardCommitCoordinator.java:296)<br/> at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.successfulPreCommit(SimpleShardDataTreeCohort.java:169)<br/> at org.opendaylight.controller.cluster.datastore.ShardDataTree.startPreCommit(ShardDataTree.java:591)<br/> at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.preCommit(SimpleShardDataTreeCohort.java:91)<br/> at org.opendaylight.controller.cluster.datastore.CohortEntry.preCommit(CohortEntry.java:102)<br/> at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.doCommit(ShardCommitCoordinator.java:296)<br/> at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.access$200(ShardCommitCoordinator.java:49)<br/> at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator$2.onSuccess(ShardCommitCoordinator.java:243)<br/> at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator$2.onSuccess(ShardCommitCoordinator.java:237)<br/> at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.successfulCanCommit(SimpleShardDataTreeCohort.java:145)<br/> at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextTransaction(ShardDataTree.java:526)<br/> at org.opendaylight.controller.cluster.datastore.ShardDataTree.finishCommit(ShardDataTree.java:623)<br/> at org.opendaylight.controller.cluster.datastore.ShardDataTree.startCommit(ShardDataTree.java:635)<br/> at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.commit(SimpleShardDataTreeCohort.java:132)<br/> at org.opendaylight.controller.cluster.datastore.CohortEntry.commit(CohortEntry.java:106)<br/> at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.finishCommit(ShardCommitCoordinator.java:316)<br/> at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.access$400(ShardCommitCoordinator.java:49)<br/> at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator$3.onSuccess(ShardCommitCoordinator.java:299)<br/> at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator$3.onSuccess(ShardCommitCoordinator.java:296)<br/> at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.successfulPreCommit(SimpleShardDataTreeCohort.java:169)<br/> at org.opendaylight.controller.cluster.datastore.ShardDataTree.startPreCommit(ShardDataTree.java:591)<br/> at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.preCommit(SimpleShardDataTreeCohort.java:91)<br/> at org.opendaylight.controller.cluster.datastore.CohortEntry.preCommit(CohortEntry.java:102)<br/> at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.doCommit(ShardCommitCoordinator.java:296)<br/> at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.access$200(ShardCommitCoordinator.java:49)<br/> at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator$2.onSuccess(ShardCommitCoordinator.java:243)<br/> at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator$2.onSuccess(ShardCommitCoordinator.java:237)<br/> at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.successfulCanCommit(SimpleShardDataTreeCohort.java:145)<br/> at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextTransaction(ShardDataTree.java:526)<br/> at org.opendaylight.controller.cluster.datastore.ShardDataTree.finishCommit(ShardDataTree.java:623)<br/> at org.opendaylight.controller.cluster.datastore.ShardDataTree.startCommit(ShardDataTree.java:635)<br/> at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.commit(SimpleShardDataTreeCohort.java:132)<br/> at org.opendaylight.controller.cluster.datastore.CohortEntry.commit(CohortEntry.java:106)<br/> at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.finishCommit(ShardCommitCoordinator.java:316)<br/> at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.access$400(ShardCommitCoordinator.java:49)<br/> at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator$3.onSuccess(ShardCommitCoordinator.java:299)<br/> at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator$3.onSuccess(ShardCommitCoordinator.java:296)<br/> at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.successfulPreCommit(SimpleShardDataTreeCohort.java:169)<br/> at org.opendaylight.controller.cluster.datastore.ShardDataTree.startPreCommit(ShardDataTree.java:591)<br/> at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.preCommit(SimpleShardDataTreeCohort.java:91)<br/> at org.opendaylight.controller.cluster.datastore.CohortEntry.preCommit(CohortEntry.java:102)<br/> at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.doCommit(ShardCommitCoordinator.java:296)<br/> at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.access$200(ShardCommitCoordinator.java:49)<br/> at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator$2.onSuccess(ShardCommitCoordinator.java:243)<br/> at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator$2.onSuccess(ShardCommitCoordinator.java:237)<br/> at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.successfulCanCommit(SimpleShardDataTreeCohort.java:145)<br/> at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextTransaction(ShardDataTree.java:526)<br/> at org.opendaylight.controller.cluster.datastore.ShardDataTree.finishCommit(ShardDataTree.java:623)<br/> at org.opendaylight.controller.cluster.datastore.ShardDataTree.startCommit(ShardDataTree.java:635)<br/> at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.commit(SimpleShardDataTreeCohort.java:132)<br/> at org.opendaylight.controller.cluster.datastore.CohortEntry.commit(CohortEntry.java:106)</p> <p>Which means we have a deep chain of transactions completing with direct commit protocol, which is exhausting stack to the point where there is not enough head room for codecs to run.</p> <p>If there is no bug there, just a long stack, we can just use: -Xss</p> <p>Well, stack size is a rather huge hammer which will cover the problem but not solve it, as the number of transactions completing does not really have an upper bound. It also has the downside of each thread costing more VSS, clamping the ability to spawn threads.</p> <p>Can you enable TRACE logs for org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator, please?</p> <p>Yeah but reproduction is not trivial (scale setup), so when I'll reach this scenario again I'll try to add the log.</p> <p><a href="https://git.opendaylight.org/gerrit/53738" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/53738</a> helps a tiny bit by eliminating synthetic accessors seen in the stack trace.</p> <p>So the code structure seems to indicate that we are dealing with operational data store, tons of small transactions which get unblocked after some sort of an up-event (data store being ready or similar).</p> <p>What we have here is an on-stack chain of futures completing immediately and invoking callbacks. I suspect we need to keep some sort of a counter as to how many transactions we have batched and check it between:</p> <p> at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextTransaction(ShardDataTree.java:526)<br/> at org.opendaylight.controller.cluster.datastore.ShardDataTree.finishCommit(ShardDataTree.java:623)</p> <p>I.e. at some point we want processNextTransaction() to decide it has committed enough transactions and it is time to sleep for a bit &#8211; perhaps by enqueueing a message to re-trigger itself. Then it will process next batch of transactions with an unwound stack.</p> <p>It's interesting that it's taking the short-cut to bypass replication in startCommit:</p> <p> if (shard.canSkipPayload() || candidate.getRootNode()<br/> .getModificationType() == ModificationType.UNMODIFIED) </p> { finishCommit(cohort); return; } <p>which is what causes the recursion. Since this is a 3-node cluster, canSkipPayload should not evaluate to true as it has followers. Therefore it seems it's getting a lot of successive UNMODIFIED transactions. Not sure if that is normal.</p> <p>This occurred in Boron. In Carbon we removed the shortcut for UNMODIFIED transactions - they are now persisted/replicated/applied. However it can still recurse if data isn't persisted nor replicated (i.e. operational data in single-node) plus we added the pipe-lining which I think could also lead to a stack overflow.</p> <p>We need to limit the number of recursive calls in ShardDataTree in the context of processing a single message. Perhaps we can use a thread local var to track the count.</p> <p>(In reply to Robert Varga from comment #6)<br/> &gt; So the code structure seems to indicate that we are dealing with operational<br/> &gt; data store, tons of small transactions which get unblocked after some sort<br/> &gt; of an up-event (data store being ready or similar).<br/> &gt; <br/> &gt; What we have here is an on-stack chain of futures completing immediately and<br/> &gt; invoking callbacks. I suspect we need to keep some sort of a counter as to<br/> &gt; how many transactions we have batched and check it between:<br/> &gt; <br/> &gt; at<br/> &gt; org.opendaylight.controller.cluster.datastore.ShardDataTree.<br/> &gt; processNextTransaction(ShardDataTree.java:526)<br/> &gt; at<br/> &gt; org.opendaylight.controller.cluster.datastore.ShardDataTree.<br/> &gt; finishCommit(ShardDataTree.java:623)<br/> &gt; <br/> &gt; I.e. at some point we want processNextTransaction() to decide it has<br/> &gt; committed enough transactions and it is time to sleep for a bit &#8211; perhaps<br/> &gt; by enqueueing a message to re-trigger itself. Then it will process next<br/> &gt; batch of transactions with an unwound stack.</p> <p>It does not need to be thread-local. We can keep a simple counter in ShardDataTree and reset it from Shard everytime we start processing a message.</p> <p>We ran into the same error today in our scale setup using stable/boron.<br/> We are using resource batch manager from genius repo to batch the transactions.<br/> The volume of write from batch manager across transactions is 4k.<br/> Not sure about how many transactions batch manager created for this 4k as we did not enable logs.</p> <p>Actually two out of three controllers wend down from the 3 node cluster with the same error.</p> <p>carbon: <a href="https://git.opendaylight.org/gerrit/56948" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/56948</a></p> <p>This can actually occur if a prefix-based shard is not replicated and not persisted &#8211; hence this is a blocker for Carbon.</p> <p>Patch <a href="https://git.opendaylight.org/gerrit/#/c/56948/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/56948/</a> was successfully merged. Can we mark this bug as fixed and resolved?</p> Duplicate CONTROLLER-1666 Development External issue ID 8056 External issue URL Issue Type ODL SR Target Milestone Rank 0|i02rp3: