https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-1666 controller <p>Testested scenario: leader isolation , tell-based=true</p> <p>-write-transactions rpc called on each node of the cluster<br/> -leader of the default config shard isolated (iptables)<br/> -remaining nodes elected new leader<br/> -rejoin of isolated node happened approximately 31-33s after isolation<br/> -after another 40s jvm went down</p> <p><a href="https://jenkins.opendaylight.org/sandbox/job/controller-csit-3node-clustering-only-carbon-4th/8/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/sandbox/job/controller-csit-3node-clustering-only-carbon-4th/8/</a></p> <p><a href="https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon-4th/8/archives/odl3_karaf_console.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon-4th/8/archives/odl3_karaf_console.log.gz</a></p> <p>Uncaught error from thread <span class="error">&#91;opendaylight-cluster-data-akka.actor.default-dispatcher-5&#93;</span> shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem<span class="error">&#91;opendaylight-cluster-data&#93;</span><br/> java.lang.StackOverflowError<br/> at com.romix.scala.collection.concurrent.TrieMap$Equiv.equiv(TrieMap.java:994)<br/> at com.romix.scala.collection.concurrent.TrieMap$INode.equal(TrieMap.java:173)<br/> at com.romix.scala.collection.concurrent.TrieMap$INode.rec_remove(TrieMap.java:509)<br/> at com.romix.scala.collection.concurrent.TrieMap$INode.rec_remove(TrieMap.java:499)<br/> at com.romix.scala.collection.concurrent.TrieMap$INode.rec_remove(TrieMap.java:502)<br/> at com.romix.scala.collection.concurrent.TrieMap$INode.rec_remove(TrieMap.java:499)<br/> at com.romix.scala.collection.concurrent.TrieMap$INode.rec_remove(TrieMap.java:502)<br/> at com.romix.scala.collection.concurrent.TrieMap$INode.rec_remove(TrieMap.java:499)<br/> at com.romix.scala.collection.concurrent.TrieMap$INode.rec_remove(TrieMap.java:502)<br/> at com.romix.scala.collection.concurrent.TrieMap.removehc(TrieMap.java:1213)<br/> at com.romix.scala.collection.concurrent.TrieMap.remove(TrieMap.java:1396)<br/> at org.opendaylight.yangtools.util.ReadWriteTrieMap.remove(ReadWriteTrieMap.java:88)<br/> at org.opendaylight.yangtools.yang.data.impl.schema.builder.impl.ImmutableMapNodeBuilder.withoutChild(ImmutableMapNodeBuilder.java:74)<br/> at org.opendaylight.yangtools.yang.data.impl.schema.builder.impl.ImmutableMapNodeBuilder.removeChild(ImmutableMapNodeBuilder.java:109)<br/> at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.mutateChildren(AbstractNodeContainerModificationStrategy.java:146)<br/> at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.applyTouch(AbstractNodeContainerModificationStrategy.java:252)<br/> at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.apply(SchemaAwareApplyOperation.java:200)<br/> at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.mutateChildren(AbstractNodeContainerModificationStrategy.java:139)<br/> at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.applyTouch(AbstractNodeContainerModificationStrategy.java:252)<br/> at org.opendaylight.yangtools.yang.data.impl.schema.tree.ListEntryModificationStrategy.applyTouch(ListEntryModificationStrategy.java:55)<br/> at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.apply(SchemaAwareApplyOperation.java:200)<br/> at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.mutateChildren(AbstractNodeContainerModificationStrategy.java:139)<br/> at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.applyTouch(AbstractNodeContainerModificationStrategy.java:252)<br/> at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.apply(SchemaAwareApplyOperation.java:200)<br/> at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.mutateChildren(AbstractNodeContainerModificationStrategy.java:139)<br/> at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.applyTouch(AbstractNodeContainerModificationStrategy.java:252)<br/> at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.apply(SchemaAwareApplyOperation.java:200)<br/> at org.opendaylight.yangtools.yang.data.impl.schema.tree.StructuralContainerModificationStrategy.apply(StructuralContainerModificationStrategy.java:71)<br/> at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.mutateChildren(AbstractNodeContainerModificationStrategy.java:139)<br/> at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.applyTouch(AbstractNodeContainerModificationStrategy.java:252)<br/> at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.apply(SchemaAwareApplyOperation.java:200)<br/> at org.opendaylight.yangtools.yang.data.impl.schema.tree.RootModificationApplyOperation.apply(RootModificationApplyOperation.java:78)<br/> at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractDataTreeTip.prepare(AbstractDataTreeTip.java:53)<br/> at org.opendaylight.controller.cluster.datastore.ShardDataTree.startPreCommit(ShardDataTree.java:809)<br/> at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.preCommit(SimpleShardDataTreeCohort.java:115)<br/> at org.opendaylight.controller.cluster.datastore.ChainedCommitCohort.preCommit(ChainedCommitCohort.java:63)<br/> at org.opendaylight.controller.cluster.datastore.FrontendReadWriteTransaction.successfulDirectCanCommit(FrontendReadWriteTransaction.java:212)<br/> at org.opendaylight.controller.cluster.datastore.FrontendReadWriteTransaction$5.onSuccess(FrontendReadWriteTransaction.java:200)<br/> at org.opendaylight.controller.cluster.datastore.FrontendReadWriteTransaction$5.onSuccess(FrontendReadWriteTransaction.java:197)<br/> at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.successfulCanCommit(SimpleShardDataTreeCohort.java:183)<br/> at org.opendaylight.controller.cluster.datastore.ShardDataTree.lambda$processNextPendingTransaction$0(ShardDataTree.java:718)<br/> at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPending(ShardDataTree.java:762)<br/> at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPendingTransaction(ShardDataTree.java:707)<br/> at org.opendaylight.controller.cluster.datastore.ShardDataTree.startPreCommit(ShardDataTree.java:828)<br/> at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.preCommit(SimpleShardDataTreeCohort.java:115)<br/> at org.opendaylight.controller.cluster.datastore.ChainedCommitCohort.preCommit(ChainedCommitCohort.java:63)<br/> at org.opendaylight.controller.cluster.datastore.FrontendReadWriteTransaction.successfulDirectCanCommit(FrontendReadWriteTransaction.java:212)<br/> at org.opendaylight.controller.cluster.datastore.FrontendReadWriteTransaction$5.onSuccess(FrontendReadWriteTransaction.java:200)<br/> at org.opendaylight.controller.cluster.datastore.FrontendReadWriteTransaction$5.onSuccess(FrontendReadWriteTransaction.java:197)<br/> at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.successfulCanCommit(SimpleShardDataTreeCohort.java:183)<br/> at org.opendaylight.controller.cluster.datastore.ShardDataTree.lambda$processNextPendingTransaction$0(ShardDataTree.java:718)<br/> at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPending(ShardDataTree.java:762)<br/> at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPendingTransaction(ShardDataTree.java:707)<br/> at org.opendaylight.controller.cluster.datastore.ShardDataTree.startPreCommit(ShardDataTree.java:828)<br/> at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.preCommit(SimpleShardDataTreeCohort.java:115)<br/> at org.opendaylight.controller.cluster.datastore.ChainedCommitCohort.preCommit(ChainedCommitCohort.java:63)<br/> at org.opendaylight.controller.cluster.datastore.FrontendReadWriteTransaction.successfulDirectCanCommit(FrontendReadWriteTransaction.java:212)<br/> at org.opendaylight.controller.cluster.datastore.FrontendReadWriteTransaction$5.onSuccess(FrontendReadWriteTransaction.java:200)<br/> at org.opendaylight.controller.cluster.datastore.FrontendReadWriteTransaction$5.onSuccess(FrontendReadWriteTransaction.java:197)<br/> at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.successfulCanCommit(SimpleShardDataTreeCohort.java:183)<br/> at org.opendaylight.controller.cluster.datastore.ShardDataTree.lambda$processNextPendingTransaction$0(ShardDataTree.java:718)<br/> at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPending(ShardDataTree.java:762)<br/> at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPendingTransaction(ShardDataTree.java:707)<br/> at org.opendaylight.controller.cluster.datastore.ShardDataTree.startPreCommit(ShardDataTree.java:828)<br/> at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.preCommit(SimpleShardDataTreeCohort.java:115)<br/> at org.opendaylight.controller.cluster.datastore.ChainedCommitCohort.preCommit(ChainedCommitCohort.java:63)<br/> at org.opendaylight.controller.cluster.datastore.FrontendReadWriteTransaction.successfulDirectCanCommit(FrontendReadWriteTransaction.java:212)<br/> at org.opendaylight.controller.cluster.datastore.FrontendReadWriteTransaction$5.onSuccess(FrontendReadWriteTransaction.java:200)<br/> at org.opendaylight.controller.cluster.datastore.FrontendReadWriteTransaction$5.onSuccess(FrontendReadWriteTransaction.java:197)<br/> at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.successfulCanCommit(SimpleShardDataTreeCohort.java:183)<br/> at org.opendaylight.controller.cluster.datastore.ShardDataTree.lambda$processNextPendingTransaction$0(ShardDataTree.java:718)<br/> at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPending(ShardDataTree.java:762)<br/> at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPendingTransaction(ShardDataTree.java:707)<br/> at org.opendaylight.controller.cluster.datastore.ShardDataTree.startPreCommit(ShardDataTree.java:828)<br/> at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.preCommit(SimpleShardDataTreeCohort.java:115)<br/> at org.opendaylight.controller.cluster.datastore.ChainedCommitCohort.preCommit(ChainedCommitCohort.java:63)<br/> at org.opendaylight.controller.cluster.datastore.FrontendReadWriteTransaction.successfulDirectCanCommit(FrontendReadWriteTransaction.java:212)<br/> at org.opendaylight.controller.cluster.datastore.FrontendReadWriteTransaction$5.onSuccess(FrontendReadWriteTransaction.java:200)<br/> at org.opendaylight.controller.cluster.datastore.FrontendReadWriteTransaction$5.onSuccess(FrontendReadWriteTransaction.java:197)<br/> at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.successfulCanCommit(SimpleShardDataTreeCohort.java:183)<br/> at org.opendaylight.controller.cluster.datastore.ShardDataTree.lambda$processNextPendingTransaction$0(ShardDataTree.java:718)<br/> at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPending(ShardDataTree.java:762)<br/> at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPendingTransaction(ShardDataTree.java:707)<br/> at org.opendaylight.controller.cluster.datastore.ShardDataTree.startPreCommit(ShardDataTree.java:828)<br/> at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.preCommit(SimpleShardDataTreeCohort.java:115)<br/> at org.opendaylight.controller.cluster.datastore.ChainedCommitCohort.preCommit(ChainedCommitCohort.java:63)<br/> at org.opendaylight.controller.cluster.datastore.FrontendReadWriteTransaction.successfulDirectCanCommit(FrontendReadWriteTransaction.java:212)<br/> at org.opendaylight.controller.cluster.datastore.FrontendReadWriteTransaction$5.onSuccess(FrontendReadWriteTransaction.java:200)<br/> at org.opendaylight.controller.cluster.datastore.FrontendReadWriteTransaction$5.onSuccess(FrontendReadWriteTransaction.java:197)<br/> at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.successfulCanCommit(SimpleShardDataTreeCohort.java:183)<br/> at org.opendaylight.controller.cluster.datastore.ShardDataTree.lambda$processNextPendingTransaction$0(ShardDataTree.java:718)<br/> at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPending(Shar</p> <p>Operating System: All<br/> Platform: All</p> CONTROLLER-1666

C: JVM down, StackOverflowError; OutOfMemoryError while debugging the problem

Bug Resolved Duplicate Unassigned Peter Gubka Thu, 11 May 2017 09:35:39 +0000 Tue, 25 Jul 2023 08:24:35 +0000 Mon, 7 Aug 2017 13:48:22 +0000 clustering 0 2 <p>Reproducing the same test again without debug log level leads to different/another exceptions</p> <p><a href="https://jenkins.opendaylight.org/sandbox/job/controller-csit-3node-clustering-only-carbon-4th/10/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/sandbox/job/controller-csit-3node-clustering-only-carbon-4th/10/</a></p> <p><a href="https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon-4th/10/archives/odl2_karaf_console.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon-4th/10/archives/odl2_karaf_console.log.gz</a></p> <p>Heap dump file created <span class="error">&#91;1392668431 bytes in 11.776 secs&#93;</span><br/> Exception in thread "AsyncAppender-Dispatcher-Thread-5" java.lang.OutOfMemoryError: Java heap space<br/> at java.util.Arrays.copyOf(Arrays.java:3332)<br/> at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)<br/> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)<br/> at java.lang.StringBuffer.append(StringBuffer.java:270)<br/> at org.apache.log4j.helpers.PatternParser$LiteralPatternConverter.format(PatternParser.java:419)<br/> at org.apache.log4j.PatternLayout.format(PatternLayout.java:506)<br/> at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:310)<br/> at org.apache.log4j.RollingFileAppender.subAppend(RollingFileAppender.java:276)<br/> at org.apache.log4j.WriterAppender.append(WriterAppender.java:162)<br/> at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)<br/> at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:59)<br/> at org.apache.log4j.AsyncAppender$Dispatcher.run(AsyncAppender.java:552)<br/> at java.lang.Thread.run(Thread.java:745)<br/> SLF4J: Failed toString() invocation on an object of type <span class="error">&#91;org.opendaylight.controller.cluster.access.client.ReconnectingClientConnection&#93;</span><br/> java.lang.OutOfMemoryError: Java heap space<br/> at java.util.Arrays.copyOf(Arrays.java:3332)<br/> at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)<br/> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)<br/> at java.lang.StringBuilder.append(StringBuilder.java:136)<br/> at org.opendaylight.yangtools.yang.data.api.schema.NormalizedNodes.toStringTree(NormalizedNodes.java:102)<br/> at org.opendaylight.yangtools.yang.data.api.schema.NormalizedNodes.toStringTree(NormalizedNodes.java:108)</p> <p>Running the same test with integration/test patch refs/changes/49/56849/2:</p> <p>Runs with </p> <p><a href="https://jenkins.opendaylight.org/sandbox/job/controller-csit-3node-clustering-only-carbon-4th/11/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/sandbox/job/controller-csit-3node-clustering-only-carbon-4th/11/</a></p> <p>OutOfMemoryError appeared on all 3 nodes<br/> <a href="https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon-4th/11/archives/odl1_karaf_console.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon-4th/11/archives/odl1_karaf_console.log.gz</a><br/> <a href="https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon-4th/11/archives/odl2_karaf_console.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon-4th/11/archives/odl2_karaf_console.log.gz</a><br/> <a href="https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon-4th/11/archives/odl3_karaf_console.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon-4th/11/archives/odl3_karaf_console.log.gz</a></p> <p>The logs are big. Debug was on for org.opendaylight.controller.cluster.databroker.actors<br/> org.opendaylight.controller.cluster.datastore</p> <p>OOM is the same as <a href="https://bugs.opendaylight.org/show_bug.cgi?id=8424" class="external-link" target="_blank" rel="nofollow noopener">https://bugs.opendaylight.org/show_bug.cgi?id=8424</a>.</p> <p>StackOverflowError looks that same as <a href="https://bugs.opendaylight.org/show_bug.cgi?id=8056" class="external-link" target="_blank" rel="nofollow noopener">https://bugs.opendaylight.org/show_bug.cgi?id=8056</a>.</p> <p>(In reply to Peter Gubka from comment #1)<br/> &gt; Reproducing the same test again without debug log level leads to<br/> &gt; different/another exceptions<br/> &gt; <br/> &gt; <a href="https://jenkins.opendaylight.org/sandbox/job/controller-csit-3node-" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/sandbox/job/controller-csit-3node-</a><br/> &gt; clustering-only-carbon-4th/10/<br/> &gt; <br/> &gt; <a href="https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-</a><br/> &gt; clustering-only-carbon-4th/10/archives/odl2_karaf_console.log.gz<br/> &gt; <br/> &gt; Heap dump file created <span class="error">&#91;1392668431 bytes in 11.776 secs&#93;</span><br/> &gt; Exception in thread "AsyncAppender-Dispatcher-Thread-5"<br/> &gt; java.lang.OutOfMemoryError: Java heap space<br/> &gt; at java.util.Arrays.copyOf(Arrays.java:3332)<br/> &gt; at<br/> &gt; java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.<br/> &gt; java:124)<br/> &gt; at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)<br/> &gt; at java.lang.StringBuffer.append(StringBuffer.java:270)<br/> &gt; at<br/> &gt; org.apache.log4j.helpers.PatternParser$LiteralPatternConverter.<br/> &gt; format(PatternParser.java:419)<br/> &gt; at org.apache.log4j.PatternLayout.format(PatternLayout.java:506)<br/> &gt; at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:310)<br/> &gt; at<br/> &gt; org.apache.log4j.RollingFileAppender.subAppend(RollingFileAppender.java:276)<br/> &gt; at org.apache.log4j.WriterAppender.append(WriterAppender.java:162)<br/> &gt; at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)<br/> &gt; at<br/> &gt; org.apache.log4j.helpers.AppenderAttachableImpl.<br/> &gt; appendLoopOnAppenders(AppenderAttachableImpl.java:59)<br/> &gt; at org.apache.log4j.AsyncAppender$Dispatcher.run(AsyncAppender.java:552)<br/> &gt; at java.lang.Thread.run(Thread.java:745)<br/> &gt; SLF4J: Failed toString() invocation on an object of type<br/> &gt; [org.opendaylight.controller.cluster.access.client.<br/> &gt; ReconnectingClientConnection]<br/> &gt; java.lang.OutOfMemoryError: Java heap space<br/> &gt; at java.util.Arrays.copyOf(Arrays.java:3332)<br/> &gt; at<br/> &gt; java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.<br/> &gt; java:124)<br/> &gt; at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)<br/> &gt; at java.lang.StringBuilder.append(StringBuilder.java:136)<br/> &gt; at<br/> &gt; org.opendaylight.yangtools.yang.data.api.schema.NormalizedNodes.<br/> &gt; toStringTree(NormalizedNodes.java:102)<br/> &gt; at<br/> &gt; org.opendaylight.yangtools.yang.data.api.schema.NormalizedNodes.<br/> &gt; toStringTree(NormalizedNodes.java:108)</p> Blocks CONTROLLER-1728 Duplicate CONTROLLER-1620 Development External issue ID 8419 External issue URL Rank 0|i02rzb: