[CONTROLLER-1666] C: JVM down, StackOverflowError; OutOfMemoryError while debugging the problem Created: 11/May/17  Updated: 25/Jul/23  Resolved: 07/Aug/17

Status: Resolved
Project: controller
Component/s: clustering
Affects Version/s: None
Fix Version/s: None

Type: Bug
Reporter: Peter Gubka Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: All
Platform: All

Issue Links:
Blocks
is blocked by CONTROLLER-1728 startup archetype upgrade to karaf 4 Resolved
Duplicate
duplicates CONTROLLER-1620 StackOverflowError while shard is com... Resolved
External issue ID: 8419

 Description   

Testested scenario: leader isolation , tell-based=true

-write-transactions rpc called on each node of the cluster
-leader of the default config shard isolated (iptables)
-remaining nodes elected new leader
-rejoin of isolated node happened approximately 31-33s after isolation
-after another 40s jvm went down

https://jenkins.opendaylight.org/sandbox/job/controller-csit-3node-clustering-only-carbon-4th/8/

https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon-4th/8/archives/odl3_karaf_console.log.gz

Uncaught error from thread [opendaylight-cluster-data-akka.actor.default-dispatcher-5] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[opendaylight-cluster-data]
java.lang.StackOverflowError
at com.romix.scala.collection.concurrent.TrieMap$Equiv.equiv(TrieMap.java:994)
at com.romix.scala.collection.concurrent.TrieMap$INode.equal(TrieMap.java:173)
at com.romix.scala.collection.concurrent.TrieMap$INode.rec_remove(TrieMap.java:509)
at com.romix.scala.collection.concurrent.TrieMap$INode.rec_remove(TrieMap.java:499)
at com.romix.scala.collection.concurrent.TrieMap$INode.rec_remove(TrieMap.java:502)
at com.romix.scala.collection.concurrent.TrieMap$INode.rec_remove(TrieMap.java:499)
at com.romix.scala.collection.concurrent.TrieMap$INode.rec_remove(TrieMap.java:502)
at com.romix.scala.collection.concurrent.TrieMap$INode.rec_remove(TrieMap.java:499)
at com.romix.scala.collection.concurrent.TrieMap$INode.rec_remove(TrieMap.java:502)
at com.romix.scala.collection.concurrent.TrieMap.removehc(TrieMap.java:1213)
at com.romix.scala.collection.concurrent.TrieMap.remove(TrieMap.java:1396)
at org.opendaylight.yangtools.util.ReadWriteTrieMap.remove(ReadWriteTrieMap.java:88)
at org.opendaylight.yangtools.yang.data.impl.schema.builder.impl.ImmutableMapNodeBuilder.withoutChild(ImmutableMapNodeBuilder.java:74)
at org.opendaylight.yangtools.yang.data.impl.schema.builder.impl.ImmutableMapNodeBuilder.removeChild(ImmutableMapNodeBuilder.java:109)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.mutateChildren(AbstractNodeContainerModificationStrategy.java:146)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.applyTouch(AbstractNodeContainerModificationStrategy.java:252)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.apply(SchemaAwareApplyOperation.java:200)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.mutateChildren(AbstractNodeContainerModificationStrategy.java:139)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.applyTouch(AbstractNodeContainerModificationStrategy.java:252)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.ListEntryModificationStrategy.applyTouch(ListEntryModificationStrategy.java:55)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.apply(SchemaAwareApplyOperation.java:200)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.mutateChildren(AbstractNodeContainerModificationStrategy.java:139)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.applyTouch(AbstractNodeContainerModificationStrategy.java:252)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.apply(SchemaAwareApplyOperation.java:200)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.mutateChildren(AbstractNodeContainerModificationStrategy.java:139)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.applyTouch(AbstractNodeContainerModificationStrategy.java:252)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.apply(SchemaAwareApplyOperation.java:200)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.StructuralContainerModificationStrategy.apply(StructuralContainerModificationStrategy.java:71)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.mutateChildren(AbstractNodeContainerModificationStrategy.java:139)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.applyTouch(AbstractNodeContainerModificationStrategy.java:252)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.apply(SchemaAwareApplyOperation.java:200)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.RootModificationApplyOperation.apply(RootModificationApplyOperation.java:78)
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractDataTreeTip.prepare(AbstractDataTreeTip.java:53)
at org.opendaylight.controller.cluster.datastore.ShardDataTree.startPreCommit(ShardDataTree.java:809)
at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.preCommit(SimpleShardDataTreeCohort.java:115)
at org.opendaylight.controller.cluster.datastore.ChainedCommitCohort.preCommit(ChainedCommitCohort.java:63)
at org.opendaylight.controller.cluster.datastore.FrontendReadWriteTransaction.successfulDirectCanCommit(FrontendReadWriteTransaction.java:212)
at org.opendaylight.controller.cluster.datastore.FrontendReadWriteTransaction$5.onSuccess(FrontendReadWriteTransaction.java:200)
at org.opendaylight.controller.cluster.datastore.FrontendReadWriteTransaction$5.onSuccess(FrontendReadWriteTransaction.java:197)
at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.successfulCanCommit(SimpleShardDataTreeCohort.java:183)
at org.opendaylight.controller.cluster.datastore.ShardDataTree.lambda$processNextPendingTransaction$0(ShardDataTree.java:718)
at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPending(ShardDataTree.java:762)
at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPendingTransaction(ShardDataTree.java:707)
at org.opendaylight.controller.cluster.datastore.ShardDataTree.startPreCommit(ShardDataTree.java:828)
at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.preCommit(SimpleShardDataTreeCohort.java:115)
at org.opendaylight.controller.cluster.datastore.ChainedCommitCohort.preCommit(ChainedCommitCohort.java:63)
at org.opendaylight.controller.cluster.datastore.FrontendReadWriteTransaction.successfulDirectCanCommit(FrontendReadWriteTransaction.java:212)
at org.opendaylight.controller.cluster.datastore.FrontendReadWriteTransaction$5.onSuccess(FrontendReadWriteTransaction.java:200)
at org.opendaylight.controller.cluster.datastore.FrontendReadWriteTransaction$5.onSuccess(FrontendReadWriteTransaction.java:197)
at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.successfulCanCommit(SimpleShardDataTreeCohort.java:183)
at org.opendaylight.controller.cluster.datastore.ShardDataTree.lambda$processNextPendingTransaction$0(ShardDataTree.java:718)
at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPending(ShardDataTree.java:762)
at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPendingTransaction(ShardDataTree.java:707)
at org.opendaylight.controller.cluster.datastore.ShardDataTree.startPreCommit(ShardDataTree.java:828)
at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.preCommit(SimpleShardDataTreeCohort.java:115)
at org.opendaylight.controller.cluster.datastore.ChainedCommitCohort.preCommit(ChainedCommitCohort.java:63)
at org.opendaylight.controller.cluster.datastore.FrontendReadWriteTransaction.successfulDirectCanCommit(FrontendReadWriteTransaction.java:212)
at org.opendaylight.controller.cluster.datastore.FrontendReadWriteTransaction$5.onSuccess(FrontendReadWriteTransaction.java:200)
at org.opendaylight.controller.cluster.datastore.FrontendReadWriteTransaction$5.onSuccess(FrontendReadWriteTransaction.java:197)
at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.successfulCanCommit(SimpleShardDataTreeCohort.java:183)
at org.opendaylight.controller.cluster.datastore.ShardDataTree.lambda$processNextPendingTransaction$0(ShardDataTree.java:718)
at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPending(ShardDataTree.java:762)
at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPendingTransaction(ShardDataTree.java:707)
at org.opendaylight.controller.cluster.datastore.ShardDataTree.startPreCommit(ShardDataTree.java:828)
at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.preCommit(SimpleShardDataTreeCohort.java:115)
at org.opendaylight.controller.cluster.datastore.ChainedCommitCohort.preCommit(ChainedCommitCohort.java:63)
at org.opendaylight.controller.cluster.datastore.FrontendReadWriteTransaction.successfulDirectCanCommit(FrontendReadWriteTransaction.java:212)
at org.opendaylight.controller.cluster.datastore.FrontendReadWriteTransaction$5.onSuccess(FrontendReadWriteTransaction.java:200)
at org.opendaylight.controller.cluster.datastore.FrontendReadWriteTransaction$5.onSuccess(FrontendReadWriteTransaction.java:197)
at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.successfulCanCommit(SimpleShardDataTreeCohort.java:183)
at org.opendaylight.controller.cluster.datastore.ShardDataTree.lambda$processNextPendingTransaction$0(ShardDataTree.java:718)
at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPending(ShardDataTree.java:762)
at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPendingTransaction(ShardDataTree.java:707)
at org.opendaylight.controller.cluster.datastore.ShardDataTree.startPreCommit(ShardDataTree.java:828)
at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.preCommit(SimpleShardDataTreeCohort.java:115)
at org.opendaylight.controller.cluster.datastore.ChainedCommitCohort.preCommit(ChainedCommitCohort.java:63)
at org.opendaylight.controller.cluster.datastore.FrontendReadWriteTransaction.successfulDirectCanCommit(FrontendReadWriteTransaction.java:212)
at org.opendaylight.controller.cluster.datastore.FrontendReadWriteTransaction$5.onSuccess(FrontendReadWriteTransaction.java:200)
at org.opendaylight.controller.cluster.datastore.FrontendReadWriteTransaction$5.onSuccess(FrontendReadWriteTransaction.java:197)
at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.successfulCanCommit(SimpleShardDataTreeCohort.java:183)
at org.opendaylight.controller.cluster.datastore.ShardDataTree.lambda$processNextPendingTransaction$0(ShardDataTree.java:718)
at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPending(ShardDataTree.java:762)
at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPendingTransaction(ShardDataTree.java:707)
at org.opendaylight.controller.cluster.datastore.ShardDataTree.startPreCommit(ShardDataTree.java:828)
at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.preCommit(SimpleShardDataTreeCohort.java:115)
at org.opendaylight.controller.cluster.datastore.ChainedCommitCohort.preCommit(ChainedCommitCohort.java:63)
at org.opendaylight.controller.cluster.datastore.FrontendReadWriteTransaction.successfulDirectCanCommit(FrontendReadWriteTransaction.java:212)
at org.opendaylight.controller.cluster.datastore.FrontendReadWriteTransaction$5.onSuccess(FrontendReadWriteTransaction.java:200)
at org.opendaylight.controller.cluster.datastore.FrontendReadWriteTransaction$5.onSuccess(FrontendReadWriteTransaction.java:197)
at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.successfulCanCommit(SimpleShardDataTreeCohort.java:183)
at org.opendaylight.controller.cluster.datastore.ShardDataTree.lambda$processNextPendingTransaction$0(ShardDataTree.java:718)
at org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPending(Shar

 Comments   
Comment by Peter Gubka [ 11/May/17 ]

Reproducing the same test again without debug log level leads to different/another exceptions

https://jenkins.opendaylight.org/sandbox/job/controller-csit-3node-clustering-only-carbon-4th/10/

https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon-4th/10/archives/odl2_karaf_console.log.gz

Heap dump file created [1392668431 bytes in 11.776 secs]
Exception in thread "AsyncAppender-Dispatcher-Thread-5" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3332)
at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
at java.lang.StringBuffer.append(StringBuffer.java:270)
at org.apache.log4j.helpers.PatternParser$LiteralPatternConverter.format(PatternParser.java:419)
at org.apache.log4j.PatternLayout.format(PatternLayout.java:506)
at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:310)
at org.apache.log4j.RollingFileAppender.subAppend(RollingFileAppender.java:276)
at org.apache.log4j.WriterAppender.append(WriterAppender.java:162)
at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:59)
at org.apache.log4j.AsyncAppender$Dispatcher.run(AsyncAppender.java:552)
at java.lang.Thread.run(Thread.java:745)
SLF4J: Failed toString() invocation on an object of type [org.opendaylight.controller.cluster.access.client.ReconnectingClientConnection]
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3332)
at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
at java.lang.StringBuilder.append(StringBuilder.java:136)
at org.opendaylight.yangtools.yang.data.api.schema.NormalizedNodes.toStringTree(NormalizedNodes.java:102)
at org.opendaylight.yangtools.yang.data.api.schema.NormalizedNodes.toStringTree(NormalizedNodes.java:108)

Comment by Peter Gubka [ 11/May/17 ]

Running the same test with integration/test patch refs/changes/49/56849/2:

Runs with

https://jenkins.opendaylight.org/sandbox/job/controller-csit-3node-clustering-only-carbon-4th/11/

OutOfMemoryError appeared on all 3 nodes
https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon-4th/11/archives/odl1_karaf_console.log.gz
https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon-4th/11/archives/odl2_karaf_console.log.gz
https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-clustering-only-carbon-4th/11/archives/odl3_karaf_console.log.gz

The logs are big. Debug was on for org.opendaylight.controller.cluster.databroker.actors
org.opendaylight.controller.cluster.datastore

Comment by Tom Pantelis [ 11/May/17 ]

OOM is the same as https://bugs.opendaylight.org/show_bug.cgi?id=8424.

StackOverflowError looks that same as https://bugs.opendaylight.org/show_bug.cgi?id=8056.

(In reply to Peter Gubka from comment #1)
> Reproducing the same test again without debug log level leads to
> different/another exceptions
>
> https://jenkins.opendaylight.org/sandbox/job/controller-csit-3node-
> clustering-only-carbon-4th/10/
>
> https://logs.opendaylight.org/sandbox/jenkins091/controller-csit-3node-
> clustering-only-carbon-4th/10/archives/odl2_karaf_console.log.gz
>
> Heap dump file created [1392668431 bytes in 11.776 secs]
> Exception in thread "AsyncAppender-Dispatcher-Thread-5"
> java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Arrays.java:3332)
> at
> java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.
> java:124)
> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
> at java.lang.StringBuffer.append(StringBuffer.java:270)
> at
> org.apache.log4j.helpers.PatternParser$LiteralPatternConverter.
> format(PatternParser.java:419)
> at org.apache.log4j.PatternLayout.format(PatternLayout.java:506)
> at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:310)
> at
> org.apache.log4j.RollingFileAppender.subAppend(RollingFileAppender.java:276)
> at org.apache.log4j.WriterAppender.append(WriterAppender.java:162)
> at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
> at
> org.apache.log4j.helpers.AppenderAttachableImpl.
> appendLoopOnAppenders(AppenderAttachableImpl.java:59)
> at org.apache.log4j.AsyncAppender$Dispatcher.run(AsyncAppender.java:552)
> at java.lang.Thread.run(Thread.java:745)
> SLF4J: Failed toString() invocation on an object of type
> [org.opendaylight.controller.cluster.access.client.
> ReconnectingClientConnection]
> java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Arrays.java:3332)
> at
> java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.
> java:124)
> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
> at java.lang.StringBuilder.append(StringBuilder.java:136)
> at
> org.opendaylight.yangtools.yang.data.api.schema.NormalizedNodes.
> toStringTree(NormalizedNodes.java:102)
> at
> org.opendaylight.yangtools.yang.data.api.schema.NormalizedNodes.
> toStringTree(NormalizedNodes.java:108)

Generated at Wed Feb 07 19:56:08 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.