Uploaded image for project: 'controller'
  1. controller
  2. CONTROLLER-1498

Controller crashes with OOM. oldRoot/currentRoot, toString() consumes too much of memory.

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved
    • Resolution: Done
    • None
    • None
    • clustering
    • None
    • Operating System: All
      Platform: All

    • 5504

    Description

      I am running controller in 3 node cluster mode, and when i shutdown the leader node, and try to do some transaction on data store, it fails because of the precondition check failure in a precondition check InMemoryDataTree.java:116. This is the exception i see at the time of OOM.

      opendaylight-cluster-data-akka.actor.default-dispatcher-5
      at java.lang.OutOfMemoryError.<init>()V (OutOfMemoryError.java:48)
      at java.util.Arrays.copyOf([CI)[C (Arrays.java:3332)
      at java.lang.AbstractStringBuilder.expandCapacity(I)V (AbstractStringBuilder.java:137)
      at java.lang.AbstractStringBuilder.ensureCapacityInternal(I)V (AbstractStringBuilder.java:121)
      at java.lang.AbstractStringBuilder.append(Ljava/lang/String;)Ljava/lang/AbstractStringBuilder; (AbstractStringBuilder.java:421)
      at java.lang.StringBuilder.append(Ljava/lang/String;)Ljava/lang/StringBuilder; (StringBuilder.java:136)
      at java.lang.StringBuilder.append(Ljava/lang/Object;)Ljava/lang/StringBuilder; (StringBuilder.java:131)
      at java.util.AbstractMap.toString()Ljava/lang/String; (AbstractMap.java:536)
      at java.lang.String.valueOf(Ljava/lang/Object;)Ljava/lang/String; (String.java:2994)
      at java.lang.StringBuilder.append(Ljava/lang/Object;)Ljava/lang/StringBuilder; (StringBuilder.java:131)
      at com.google.common.base.MoreObjects$ToStringHelper.toString()Ljava/lang/String; (MoreObjects.java:359)
      at org.opendaylight.yangtools.yang.data.api.schema.tree.spi.AbstractTreeNode.toString()Ljava/lang/String; (AbstractTreeNode.java:45)
      at java.lang.String.valueOf(Ljava/lang/Object;)Ljava/lang/String; (String.java:2994)
      at java.lang.StringBuilder.append(Ljava/lang/Object;)Ljava/lang/StringBuilder; (StringBuilder.java:131)
      at java.util.AbstractMap.toString()Ljava/lang/String; (AbstractMap.java:536)
      at java.lang.String.valueOf(Ljava/lang/Object;)Ljava/lang/String; (String.java:2994)
      at java.lang.StringBuilder.append(Ljava/lang/Object;)Ljava/lang/StringBuilder; (StringBuilder.java:131)
      at com.google.common.base.MoreObjects$ToStringHelper.toString()Ljava/lang/String; (MoreObjects.java:359)
      at org.opendaylight.yangtools.yang.data.api.schema.tree.spi.AbstractTreeNode.toString()Ljava/lang/String; (AbstractTreeNode.java:45)
      at java.lang.String.valueOf(Ljava/lang/Object;)Ljava/lang/String; (String.java:2994)
      at java.lang.StringBuilder.append(Ljava/lang/Object;)Ljava/lang/StringBuilder; (StringBuilder.java:131)
      at java.util.AbstractMap.toString()Ljava/lang/String; (AbstractMap.java:536)
      at java.lang.String.valueOf(Ljava/lang/Object;)Ljava/lang/String; (String.java:2994)
      at java.lang.StringBuilder.append(Ljava/lang/Object;)Ljava/lang/StringBuilder; (StringBuilder.java:131)
      at com.google.common.base.MoreObjects$ToStringHelper.toString()Ljava/lang/String; (MoreObjects.java:359)
      at org.opendaylight.yangtools.yang.data.api.schema.tree.spi.AbstractTreeNode.toString()Ljava/lang/String; (AbstractTreeNode.java:45)
      at java.lang.String.valueOf(Ljava/lang/Object;)Ljava/lang/String; (String.java:2994)
      at java.lang.StringBuilder.append(Ljava/lang/Object;)Ljava/lang/StringBuilder; (StringBuilder.java:131)
      at com.google.common.base.Preconditions.format(Ljava/lang/String;[Ljava/lang/Object;)Ljava/lang/String; (Preconditions.java:422)
      at com.google.common.base.Preconditions.checkState(ZLjava/lang/String;[Ljava/lang/Object;)V (Preconditions.java:197)
      at org.opendaylight.yangtools.yang.data.impl.schema.tree.InMemoryDataTree.commit(Lorg/opendaylight/yangtools/yang/data/api/schema/tree/DataTreeCandidate;)V (InMemoryDataTree.java:116)
      at org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.commit()Lcom/google/common/util/concurrent/ListenableFuture; (SimpleShardDataTreeCohort.java:102)
      at org.opendaylight.controller.cluster.datastore.ChainedCommitCohort.commit()Lcom/google/common/util/concurrent/ListenableFuture; (ChainedCommitCohort.java:32)
      at org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator$CohortEntry.commit()V (ShardCommitCoordinator.java:670)
      at org.opendaylight.controller.cluster.datastore.Shard.finishCommit(Lakka/actor/ActorRef;Ljava/lang/String;Lorg/opendaylight/controller/cluster/datastore/ShardCommitCoordinator$CohortEntry;)V (Shard.java:348)
      at org.opendaylight.controller.cluster.datastore.Shard.finishCommit(Lakka/actor/ActorRef;Ljava/lang/String;)V (Shard.java:398)
      at org.opendaylight.controller.cluster.datastore.Shard.applyState(Lakka/actor/ActorRef;Ljava/lang/String;Ljava/lang/Object;)V (Shard.java:646)
      at org.opendaylight.controller.cluster.raft.RaftActor.handleCommand(Ljava/lang/Object;)V (RaftActor.java:228)
      at org.opendaylight.controller.cluster.common.actor.AbstractUntypedPersistentActor.onReceiveCommand(Ljava/lang/Object;)V (AbstractUntypedPersistentActor.java:36)
      at org.opendaylight.controller.cluster.datastore.Shard.onReceiveCommand(Ljava/lang/Object;)V (Shard.java:273)
      at akka.persistence.UntypedPersistentActor.onReceive(Ljava/lang/Object;)V (Eventsourced.scala:430)
      at org.opendaylight.controller.cluster.common.actor.MeteringBehavior.apply(Ljava/lang/Object;)V (MeteringBehavior.java:97)
      at akka.actor.ActorCell$$anonfun$become$1.applyOrElse(Ljava/lang/Object;Lscala/Function1;)Ljava/lang/Object; (ActorCell.scala:534)
      at akka.persistence.Recovery$State$class.process(Lakka/persistence/Recovery$State;Lscala/PartialFunction;Ljava/lang/Object;)V (Recovery.scala:30)
      at akka.persistence.ProcessorImpl$$anon$2.process(Lscala/PartialFunction;Ljava/lang/Object;)V (Processor.scala:103)
      at akka.persistence.ProcessorImpl$$anon$2.aroundReceive(Lscala/PartialFunction;Ljava/lang/Object;)V (Processor.scala:114)
      at akka.persistence.Recovery$class.aroundReceive(Lakka/persistence/Recovery;Lscala/PartialFunction;Ljava/lang/Object;)V (Recovery.scala:265)
      at akka.persistence.UntypedPersistentActor.akka$persistence$Eventsourced$$super$aroundReceive(Lscala/PartialFunction;Ljava/lang/Object;)V (Eventsourced.scala:428)
      at akka.persistence.Eventsourced$$anon$2.doAroundReceive(Lscala/PartialFunction;Ljava/lang/Object;)V (Eventsourced.scala:82)
      at akka.persistence.Eventsourced$$anon$2.aroundReceive(Lscala/PartialFunction;Ljava/lang/Object;)V (Eventsourced.scala:78)
      at akka.persistence.Eventsourced$class.aroundReceive(Lakka/persistence/Eventsourced;Lscala/PartialFunction;Ljava/lang/Object;)V (Eventsourced.scala:369)
      at akka.persistence.UntypedPersistentActor.aroundReceive(Lscala/PartialFunction;Ljava/lang/Object;)V (Eventsourced.scala:428)
      at akka.actor.ActorCell.receiveMessage(Ljava/lang/Object;)V (ActorCell.scala:516)
      at akka.actor.ActorCell.invoke(Lakka/dispatch/Envelope;)V (ActorCell.scala:487)
      at akka.dispatch.Mailbox.processMailbox(IJ)V (Mailbox.scala:238)
      at akka.dispatch.Mailbox.run()V (Mailbox.scala:220)
      at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec()Z (AbstractDispatcher.scala:397)
      at scala.concurrent.forkjoin.ForkJoinTask.doExec()I (ForkJoinTask.java:260)
      at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(Lscala/concurrent/forkjoin/ForkJoinTask;)V (ForkJoinPool.java:1339)
      at scala.concurrent.forkjoin.ForkJoinPool.runWorker(Lscala/concurrent/forkjoin/ForkJoinPool$WorkQueue;)V (ForkJoinPool.java:1979)
      at scala.concurrent.forkjoin.ForkJoinWorkerThread.run()V (ForkJoinWorkerThread.java:107)
      '

      Precondition fails trying to generate the failure message and it tries to build toString() for oldRoot and currentRoot, and looks like it requires too much memory to build the toString of the thole subtree. I tried to patch it by skipping the toString() of oldRoot and currentRoot in Precondition and things looks stable.

      Attachments

        1. node_3_odl_logs.zip
          372 kB
        2. node_2_odl_logs.zip
          484 kB
        3. node_1_odl_logs.zip
          601 kB
        4. java_pid10114_Top_Consumers.zip
          68 kB
        5. java_pid10114_Leak_Suspects.zip
          104 kB

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              tpantelis Tom Pantelis
              Avishnoi Anil Vishnoi
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: