https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-1576 controller <p>Using Stable/Boron.<br/> When datastore grew to a certain size, I received an OOM from the snapshot operation. After analysis the conclusion was that the snapshot is being serialized to a single byte[] and if the size is bigger than 2G, the code of ByteArrayOutputStream will throw an OOM error:<br/> private static int hugeCapacity(int minCapacity) </p> { if (minCapacity &lt; 0) // overflow throw new OutOfMemoryError(); return (minCapacity &gt; MAX_ARRAY_SIZE) ? Integer.MAX_VALUE : MAX_ARRAY_SIZE; } <p>Log:</p> <p>Uncaught error from thread <span class="error">&#91;opendaylight-cluster-data-akka.actor.default-dispatcher-54&#93;</span> shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem<span class="error">&#91;opendaylight-cluster-data&#93;</span><br/> java.lang.OutOfMemoryError<br/> at java.io.ByteArrayOutputStream.hugeCapacity(ByteArrayOutputStream.java:123)<br/> at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:117)<br/> at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)<br/> at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)<br/> at java.io.DataOutputStream.write(DataOutputStream.java:107)<br/> at java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1877)<br/> at java.io.ObjectOutputStream$BlockDataOutputStream.write(ObjectOutputStream.java:1808)<br/> at java.io.DataOutputStream.writeInt(DataOutputStream.java:200)<br/> at java.io.ObjectOutputStream$BlockDataOutputStream.writeInt(ObjectOutputStream.java:1943)<br/> at java.io.ObjectOutputStream.writeInt(ObjectOutputStream.java:797)<br/> at org.opendaylight.controller.cluster.datastore.node.utils.stream.AbstractNormalizedNodeDataOutput.writeInt(AbstractNormalizedNodeDataOutput.java:107)<br/> at org.opendaylight.controller.cluster.datastore.node.utils.stream.NormalizedNodeOutputStreamWriter.writeString(NormalizedNodeOutputStreamWriter.java:60)<br/> at org.opendaylight.controller.cluster.datastore.node.utils.stream.NormalizedNodeOutputStreamWriter.writeQName(NormalizedNodeOutputStreamWriter.java:47)<br/> at org.opendaylight.controller.cluster.datastore.node.utils.stream.AbstractNormalizedNodeDataOutput.writeKeyValueMap(AbstractNormalizedNodeDataOutput.java:400)<br/> at org.opendaylight.controller.cluster.datastore.node.utils.stream.AbstractNormalizedNodeDataOutput.startMapEntryNode(AbstractNormalizedNodeDataOutput.java:250)<br/> at org.opendaylight.yangtools.yang.data.api.schema.stream.NormalizedNodeWriter$OrderedNormalizedNodeWriter.writeMapEntryNode(NormalizedNodeWriter.java:267)<br/> at org.opendaylight.yangtools.yang.data.api.schema.stream.NormalizedNodeWriter.wasProcessedAsCompositeNode(NormalizedNodeWriter.java:208)<br/> at org.opendaylight.yangtools.yang.data.api.schema.stream.NormalizedNodeWriter.write(NormalizedNodeWriter.java:100)<br/> at org.opendaylight.yangtools.yang.data.api.schema.stream.NormalizedNodeWriter.writeChildren(NormalizedNodeWriter.java:171)<br/> at org.opendaylight.yangtools.yang.data.api.schema.stream.NormalizedNodeWriter.wasProcessedAsCompositeNode(NormalizedNodeWriter.java:238)<br/> at org.opendaylight.yangtools.yang.data.api.schema.stream.NormalizedNodeWriter.write(NormalizedNodeWriter.java:100)<br/> at org.opendaylight.yangtools.yang.data.api.schema.stream.NormalizedNodeWriter.writeChildren(NormalizedNodeWriter.java:171)<br/> at org.opendaylight.yangtools.yang.data.api.schema.stream.NormalizedNodeWriter.wasProcessedAsCompositeNode(NormalizedNodeWriter.java:196)<br/> at org.opendaylight.yangtools.yang.data.api.schema.stream.NormalizedNodeWriter.write(NormalizedNodeWriter.java:100)<br/> at org.opendaylight.yangtools.yang.data.api.schema.stream.NormalizedNodeWriter.writeChildren(NormalizedNodeWriter.java:171)<br/> at org.opendaylight.yangtools.yang.data.api.schema.stream.NormalizedNodeWriter$OrderedNormalizedNodeWriter.writeMapEntryNode(NormalizedNodeWriter.java:282)<br/> at org.opendaylight.yangtools.yang.data.api.schema.stream.NormalizedNodeWriter.wasProcessedAsCompositeNode(NormalizedNodeWriter.java:208)<br/> at org.opendaylight.yangtools.yang.data.api.schema.stream.NormalizedNodeWriter.write(NormalizedNodeWriter.java:100)<br/> at org.opendaylight.yangtools.yang.data.api.schema.stream.NormalizedNodeWriter.writeChildren(NormalizedNodeWriter.java:171)<br/> at org.opendaylight.yangtools.yang.data.api.schema.stream.NormalizedNodeWriter.wasProcessedAsCompositeNode(NormalizedNodeWriter.java:238)<br/> at org.opendaylight.yangtools.yang.data.api.schema.stream.NormalizedNodeWriter.write(NormalizedNodeWriter.java:100)<br/> at org.opendaylight.yangtools.yang.data.api.schema.stream.NormalizedNodeWriter.writeChildren(NormalizedNodeWriter.java:171)<br/> at org.opendaylight.yangtools.yang.data.api.schema.stream.NormalizedNodeWriter$OrderedNormalizedNodeWriter.writeMapEntryNode(NormalizedNodeWriter.java:282)<br/> at org.opendaylight.yangtools.yang.data.api.schema.stream.NormalizedNodeWriter.wasProcessedAsCompositeNode(NormalizedNodeWriter.java:208)<br/> at org.opendaylight.yangtools.yang.data.api.schema.stream.NormalizedNodeWriter.write(NormalizedNodeWriter.java:100)<br/> at org.opendaylight.yangtools.yang.data.api.schema.stream.NormalizedNodeWriter.writeChildren(NormalizedNodeWriter.java:171)<br/> at org.opendaylight.yangtools.yang.data.api.schema.stream.NormalizedNodeWriter.wasProcessedAsCompositeNode(NormalizedNodeWriter.java:238)<br/> at org.opendaylight.yangtools.yang.data.api.schema.stream.NormalizedNodeWriter.write(NormalizedNodeWriter.java:100)<br/> at org.opendaylight.yangtools.yang.data.api.schema.stream.NormalizedNodeWriter.writeChildren(NormalizedNodeWriter.java:171)<br/> at org.opendaylight.yangtools.yang.data.api.schema.stream.NormalizedNodeWriter.wasProcessedAsCompositeNode(NormalizedNodeWriter.java:223)<br/> at org.opendaylight.yangtools.yang.data.api.schema.stream.NormalizedNodeWriter.write(NormalizedNodeWriter.java:100)<br/> at org.opendaylight.yangtools.yang.data.api.schema.stream.NormalizedNodeWriter.writeChildren(NormalizedNodeWriter.java:171)<br/> at org.opendaylight.yangtools.yang.data.api.schema.stream.NormalizedNodeWriter$OrderedNormalizedNodeWriter.writeMapEntryNode(NormalizedNodeWriter.java:282)<br/> at org.opendaylight.yangtools.yang.data.api.schema.stream.NormalizedNodeWriter.wasProcessedAsCompositeNode(NormalizedNodeWriter.java:208)<br/> at org.opendaylight.yangtools.yang.data.api.schema.stream.NormalizedNodeWriter.write(NormalizedNodeWriter.java:100)<br/> at org.opendaylight.yangtools.yang.data.api.schema.stream.NormalizedNodeWriter.writeChildren(NormalizedNodeWriter.java:171)<br/> at org.opendaylight.yangtools.yang.data.api.schema.stream.NormalizedNodeWriter.wasProcessedAsCompositeNode(NormalizedNodeWriter.java:238)<br/> at org.opendaylight.yangtools.yang.data.api.schema.stream.NormalizedNodeWriter.write(NormalizedNodeWriter.java:100)<br/> at org.opendaylight.yangtools.yang.data.api.schema.stream.NormalizedNodeWriter.writeChildren(NormalizedNodeWriter.java:171)<br/> at org.opendaylight.yangtools.yang.data.api.schema.stream.NormalizedNodeWriter.wasProcessedAsCompositeNode(NormalizedNodeWriter.java:196)<br/> at org.opendaylight.yangtools.yang.data.api.schema.stream.NormalizedNodeWriter.write(NormalizedNodeWriter.java:100)<br/> at org.opendaylight.yangtools.yang.data.api.schema.stream.NormalizedNodeWriter.writeChildren(NormalizedNodeWriter.java:171)<br/> at org.opendaylight.yangtools.yang.data.api.schema.stream.NormalizedNodeWriter.wasProcessedAsCompositeNode(NormalizedNodeWriter.java:196)<br/> at org.opendaylight.yangtools.yang.data.api.schema.stream.NormalizedNodeWriter.write(NormalizedNodeWriter.java:100)<br/> at org.opendaylight.controller.cluster.datastore.node.utils.stream.AbstractNormalizedNodeDataOutput.writeNormalizedNode(AbstractNormalizedNodeDataOutput.java:157)<br/> at org.opendaylight.controller.cluster.datastore.utils.SerializationUtils.serializeNormalizedNode(SerializationUtils.java:90)<br/> at org.opendaylight.controller.cluster.datastore.persisted.MetadataShardDataTreeSnapshot$Proxy.writeExternal(MetadataShardDataTreeSnapshot.java:56)<br/> at java.io.ObjectOutputStream.writeExternalData(ObjectOutputStream.java:1459)<br/> at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1430)<br/> at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)<br/> at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)<br/> at org.opendaylight.controller.cluster.datastore.persisted.AbstractVersionedShardDataTreeSnapshot.versionedSerialize(AbstractVersionedShardDataTreeSnapshot.java:77)<br/> at org.opendaylight.controller.cluster.datastore.persisted.AbstractVersionedShardDataTreeSnapshot.serialize(AbstractVersionedShardDataTreeSnapshot.java:95)<br/> at org.opendaylight.controller.cluster.datastore.actors.ShardSnapshotActor.handleReceive(ShardSnapshotActor.java:54)<br/> at org.opendaylight.controller.cluster.common.actor.AbstractUntypedActor.onReceive(AbstractUntypedActor.java:26)<br/> at org.opendaylight.controller.cluster.common.actor.MeteringBehavior.apply(MeteringBehavior.java:97)<br/> at akka.actor.ActorCell$$anonfun$become$1.applyOrElse(ActorCell.scala:544)<br/> at akka.actor.Actor$class.aroundReceive(Actor.scala:484)<br/> at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:95)<br/> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)<br/> at akka.actor.ActorCell.invoke(ActorCell.scala:495)<br/> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)<br/> at akka.dispatch.Mailbox.run(Mailbox.scala:224)<br/> at akka.dispatch.Mailbox.exec(Mailbox.scala:234)<br/> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)<br/> at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)<br/> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)<br/> at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)<br/> Suppressed: java.lang.OutOfMemoryError</p> <p>Operating System: All<br/> Platform: All</p> CONTROLLER-1576

OOM when persisting a snapshot of the datastore that is bigger than 2G

Bug Resolved Done Tom Pantelis Guy Sela Wed, 11 Jan 2017 09:22:08 +0000 Tue, 25 Jul 2023 08:24:18 +0000 Wed, 1 Mar 2017 01:26:30 +0000 clustering 0 3 <p>Out of curiosity, how critical do we feel fixing this is? Is it critical to get it fixed in Carbon? I'm just trying to understand how immediate the needs are here.</p> <p>For our deployments, it is the upper bound for the scale currently, so it is very important. <br/> Working with full mesh of tunnels in netvirt is blocked in 100 computes.<br/> Without full mesh, I can't give you the exact number, but from analyzing the histogram of the heap, we will definitely be blocked on a number that is lower than what we need.</p> <p>Patch series: <a href="https://git.opendaylight.org/gerrit/#/q/status:open+project:controller+branch:master+topic:file_backed_snapshots" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/q/status:open+project:controller+branch:master+topic:file_backed_snapshots</a></p> <p>The remaining issue is that akka's snapshot store serializes the snapshot payload to a byte[] before persisting to disk se we'll run into the same limit issue. To fix this we'll have to write our own snapshot store implementation that serializes the payload directly to the file.</p> <p>Full list of patches: <a href="https://git.opendaylight.org/gerrit/#/q/status:open+project:controller+branch:master+topic:bug/7521" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/q/status:open+project:controller+branch:master+topic:bug/7521</a></p> <p>Just for the record: This (or rather the lack of this in an old version) came up again in <a href="https://jira.opendaylight.org/browse/CONTROLLER-1864" title="A 8s &quot;non-GC world stop JVM pause&quot; during snapshot writes" class="issue-link" data-issue-key="CONTROLLER-1864"><del>CONTROLLER-1864</del></a>.</p> Relates CONTROLLER-1864 Development External issue ID 7521 External issue URL Rank 0|i02rfb: