https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-1498 controller <p>I am running controller in 3 node cluster mode, and when i shutdown the leader node, and try to do some transaction on data store, it fails because of the precondition check failure in a precondition check InMemoryDataTree.java:116. This is the exception i see at the time of OOM.</p> <p>opendaylight-cluster-data-akka.actor.default-dispatcher-5<br/> at java.lang.OutOfMemoryError.&lt;init&gt;()V (OutOfMemoryError.java:48)<br/> at java.util.Arrays.copyOf([CI)[C (Arrays.java:3332)<br/> at java.lang.AbstractStringBuilder.expandCapacity(I)V (AbstractStringBuilder.java:137)<br/> at java.lang.AbstractStringBuilder.ensureCapacityInternal(I)V (AbstractStringBuilder.java:121)<br/> at java.lang.AbstractStringBuilder.append(Ljava/lang/String;)Ljava/lang/AbstractStringBuilder; (AbstractStringBuilder.java:421)<br/> at java.lang.StringBuilder.append(Ljava/lang/String;)Ljava/lang/StringBuilder; (StringBuilder.java:136)<br/> at java.lang.StringBuilder.append(Ljava/lang/Object;)Ljava/lang/StringBuilder; (StringBuilder.java:131)<br/> at java.util.AbstractMap.toString()Ljava/lang/String; (AbstractMap.java:536)<br/> at java.lang.String.valueOf(Ljava/lang/Object;)Ljava/lang/String; (String.java:2994)<br/> at java.lang.StringBuilder.append(Ljava/lang/Object;)Ljava/lang/StringBuilder; (StringBuilder.java:131)<br/> at com.google.common.base.MoreObjects$ToStringHelper.toString()Ljava/lang/String; (MoreObjects.java:359)<br/> at org.opendaylight.yangtools.yang.data.api.schema.tree.spi.AbstractTreeNode.toString()Ljava/lang/String; (AbstractTreeNode.java:45)<br/> at java.lang.String.valueOf(Ljava/lang/Object;)Ljava/lang/String; (String.java:2994)<br/> at java.lang.StringBuilder.append(Ljava/lang/Object;)Ljava/lang/StringBuilder; (StringBuilder.java:131)<br/> at java.util.AbstractMap.toString()Ljava/lang/String; (AbstractMap.java:536)<br/> at java.lang.String.valueOf(Ljava/lang/Object;)Ljava/lang/String; (String.java:2994)<br/> at java.lang.StringBuilder.append(Ljava/lang/Object;)Ljava/lang/StringBuilder; (StringBuilder.java:131)<br/> at com.google.common.base.MoreObjects$ToStringHelper.toString()Ljava/lang/String; (MoreObjects.java:359)<br/> at org.opendaylight.yangtools.yang.data.api.schema.tree.spi.AbstractTreeNode.toString()Ljava/lang/String; (AbstractTreeNode.java:45)<br/> at java.lang.String.valueOf(Ljava/lang/Object;)Ljava/lang/String; (String.java:2994)<br/> at java.lang.StringBuilder.append(Ljava/lang/Object;)Ljava/lang/StringBuilder; (StringBuilder.java:131)<br/> at java.util.AbstractMap.toString()Ljava/lang/String; (AbstractMap.java:536)<br/> at java.lang.String.valueOf(Ljava/lang/Object;)Ljava/lang/String; (String.java:2994)<br/> at java.lang.StringBuilder.append(Ljava/lang/Object;)Ljava/lang/StringBuilder; (StringBuilder.java:131)<br/> at com.google.common.base.MoreObjects$ToStringHelper.toString()Ljava/lang/String; (MoreObjects.java:359)<br/> at org.opendaylight.yangtools.yang.data.api.schema.tree.spi.AbstractTreeNode.toString()Ljava/lang/String; (AbstractTreeNode.java:45)<br/> at java.lang.String.valueOf(Ljava/lang/Object;)Ljava/lang/String; (String.java:2994)<br/> at java.lang.StringBuilder.append(Ljava/lang/Object;)Ljava/lang/StringBuilder; (StringBuilder.java:131)<br/> at com.google.common.base.Preconditions.format(Ljava/lang/String;[Ljava/lang/Object;)Ljava/lang/String; (Preconditions.java:422)<br/> at com.google.common.base.Preconditions.checkState(ZLjava/lang/String;[Ljava/lang/Object;)V (Preconditions.java:197)<br/> at org.opendaylight.yangtools.yang.data.impl.schema.tree.InMemoryDataTree.commit(Lorg/opendaylight/yangtools/yang/data/api/schema/tree/DataTreeCandidate;)V (InMemoryDataTree.java:116)<br/> at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.commit()Lcom/google/common/util/concurrent/ListenableFuture; (SimpleShardDataTreeCohort.java:102)<br/> at org.opendaylight.controller.cluster.datastore.ChainedCommitCohort.commit()Lcom/google/common/util/concurrent/ListenableFuture; (ChainedCommitCohort.java:32)<br/> at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator$CohortEntry.commit()V (ShardCommitCoordinator.java:670)<br/> at org.opendaylight.controller.cluster.datastore.Shard.finishCommit(Lakka/actor/ActorRef;Ljava/lang/String;Lorg/opendaylight/controller/cluster/datastore/ShardCommitCoordinator$CohortEntry;)V (Shard.java:348)<br/> at org.opendaylight.controller.cluster.datastore.Shard.finishCommit(Lakka/actor/ActorRef;Ljava/lang/String;)V (Shard.java:398)<br/> at org.opendaylight.controller.cluster.datastore.Shard.applyState(Lakka/actor/ActorRef;Ljava/lang/String;Ljava/lang/Object;)V (Shard.java:646)<br/> at org.opendaylight.controller.cluster.raft.RaftActor.handleCommand(Ljava/lang/Object;)V (RaftActor.java:228)<br/> at org.opendaylight.controller.cluster.common.actor.AbstractUntypedPersistentActor.onReceiveCommand(Ljava/lang/Object;)V (AbstractUntypedPersistentActor.java:36)<br/> at org.opendaylight.controller.cluster.datastore.Shard.onReceiveCommand(Ljava/lang/Object;)V (Shard.java:273)<br/> at akka.persistence.UntypedPersistentActor.onReceive(Ljava/lang/Object;)V (Eventsourced.scala:430)<br/> at org.opendaylight.controller.cluster.common.actor.MeteringBehavior.apply(Ljava/lang/Object;)V (MeteringBehavior.java:97)<br/> at akka.actor.ActorCell$$anonfun$become$1.applyOrElse(Ljava/lang/Object;Lscala/Function1;)Ljava/lang/Object; (ActorCell.scala:534)<br/> at akka.persistence.Recovery$State$class.process(Lakka/persistence/Recovery$State;Lscala/PartialFunction;Ljava/lang/Object;)V (Recovery.scala:30)<br/> at akka.persistence.ProcessorImpl$$anon$2.process(Lscala/PartialFunction;Ljava/lang/Object;)V (Processor.scala:103)<br/> at akka.persistence.ProcessorImpl$$anon$2.aroundReceive(Lscala/PartialFunction;Ljava/lang/Object;)V (Processor.scala:114)<br/> at akka.persistence.Recovery$class.aroundReceive(Lakka/persistence/Recovery;Lscala/PartialFunction;Ljava/lang/Object;)V (Recovery.scala:265)<br/> at akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(Lscala/PartialFunction;Ljava/lang/Object;)V (Eventsourced.scala:428)<br/> at akka.persistence.Eventsourced$$anon$2.doAroundReceive(Lscala/PartialFunction;Ljava/lang/Object;)V (Eventsourced.scala:82)<br/> at akka.persistence.Eventsourced$$anon$2.aroundReceive(Lscala/PartialFunction;Ljava/lang/Object;)V (Eventsourced.scala:78)<br/> at akka.persistence.Eventsourced$class.aroundReceive(Lakka/persistence/Eventsourced;Lscala/PartialFunction;Ljava/lang/Object;)V (Eventsourced.scala:369)<br/> at akka.persistence.UntypedPersistentActor.aroundReceive(Lscala/PartialFunction;Ljava/lang/Object;)V (Eventsourced.scala:428)<br/> at akka.actor.ActorCell.receiveMessage(Ljava/lang/Object;)V (ActorCell.scala:516)<br/> at akka.actor.ActorCell.invoke(Lakka/dispatch/Envelope;)V (ActorCell.scala:487)<br/> at akka.dispatch.Mailbox.processMailbox(IJ)V (Mailbox.scala:238)<br/> at akka.dispatch.Mailbox.run()V (Mailbox.scala:220)<br/> at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec()Z (AbstractDispatcher.scala:397)<br/> at scala.concurrent.forkjoin.ForkJoinTask.doExec()I (ForkJoinTask.java:260)<br/> at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(Lscala/concurrent/forkjoin/ForkJoinTask;)V (ForkJoinPool.java:1339)<br/> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(Lscala/concurrent/forkjoin/ForkJoinPool$WorkQueue;)V (ForkJoinPool.java:1979)<br/> at scala.concurrent.forkjoin.ForkJoinWorkerThread.run()V (ForkJoinWorkerThread.java:107)<br/> '</p> <p>Precondition fails trying to generate the failure message and it tries to build toString() for oldRoot and currentRoot, and looks like it requires too much memory to build the toString of the thole subtree. I tried to patch it by skipping the toString() of oldRoot and currentRoot in Precondition and things looks stable.</p> <p>Operating System: All<br/> Platform: All</p> CONTROLLER-1498

Controller crashes with OOM. oldRoot/currentRoot, toString() consumes too much of memory.

Bug Resolved Done Tom Pantelis Anil Vishnoi Fri, 11 Mar 2016 22:34:44 +0000 Tue, 25 Jul 2023 08:24:08 +0000 Tue, 14 Jun 2016 14:57:26 +0000 clustering 0 9 <p>Attachment java_pid10114_Leak_Suspects.zip has been added with description: Leak Suspect report of MAT</p> <p>Attachment java_pid10114_Top_Consumers.zip has been added with description: Tom Consume Report of MAT from Headdump</p> <p>All the logs files (karaf.log.10) were consumed in printing the ReadFailedException to string.</p> <p>Attachment node_1_odl_logs.zip has been added with description: Last few minutes logs of cluster node 1</p> <p>Attachment node_2_odl_logs.zip has been added with description: Logs from Controller node 2</p> <p>Attachment node_3_odl_logs.zip has been added with description: Controller Node 3</p> <p>I build a local yangtools-impl jar with this patch and we stop seeing the frequent OOM we were seeing in our environment.</p> <p> <a href="https://git.opendaylight.org/gerrit/36148" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/36148</a></p> <p>I personally feel building subtree in precondition is probably dangerous and can eat up lot of memory and probably performance issue. This fix might not be final, but we need to fix it asap.</p> <p>Severity: Could you elaborate on the severity of this bug? Is this a BLOCKER such that we cannot release Beryllium without it? Is there a workaround such that we can write a release note and fix in Beryllium SR2?<br/> Testing: Could you also elaborate on the testing or this patch? How extensively has this patch been tested? Is it covered by any unit tests or system tests? <br/> Impact: Does this fix impact any dependent projects?</p> <p>The root cause is sal-distributed-datastore violating DataTree invocation sequencing. Current solution just hides the problem by relying on implementation behavior. This needs to be fixed in CDS.</p> <p>Submitted <a href="https://git.opendaylight.org/gerrit/#/c/36172/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/36172/</a> to stable/be as interim fix.</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/36171/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/36171/</a> will implement better solution in Boron.</p> <p>Have we verified that this bug is fixed in the latest build for Beryllium?</p> <p>(In reply to A H from comment #9)<br/> &gt; Have we verified that this bug is fixed in the latest build for Beryllium?</p> <p>I have verified in snapshot distros produced after this yangtools patch<br/> was merged and the problem was not there. Is that good enough?</p> Blocks CONTROLLER-1506 Development External issue ID 5504 External issue URL Issue Type ODL SR Target Milestone Rank 0|i02qxz: