https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-1872 controller <p>One of the very top "TLAB allocation" (cumulative many 10s of GiBs - that's a hell of a lot, no?) observed in a JFR from internal scale testing, which seems to contributed to excessive GC, is the code path below. </p> <p>Is this just "too bad" and "simply the way Akka has to work" and we accept this, or is there any way that we could (get Akka to / work with) to have this optimized to cause less object allocation?</p> <p>There are actually 2 distincts paths, the first bigger one is:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-object">byte</span>[] java.util.Arrays.copyOf(<span class="code-object">byte</span>[], <span class="code-object">int</span>) 55539 void java.io.ByteArrayOutputStream.grow(<span class="code-object">int</span>) 37343 void java.io.ByteArrayOutputStream.ensureCapacity(<span class="code-object">int</span>) 37343 void java.io.ByteArrayOutputStream.write(<span class="code-object">byte</span>[], <span class="code-object">int</span>, <span class="code-object">int</span>) 37343 void java.io.ObjectOutputStream$BlockDataOutputStream.write(<span class="code-object">byte</span>[], <span class="code-object">int</span>, <span class="code-object">int</span>, <span class="code-object">boolean</span>) 27262 void java.io.ObjectOutputStream.write(<span class="code-object">byte</span>[]) 27262 void org.opendaylight.controller.cluster.datastore.persisted.CommitTransactionPayload$Proxy.writeExternal(ObjectOutput) 27262 void java.io.ObjectOutputStream.writeExternalData(Externalizable) 27262 void java.io.ObjectOutputStream.writeOrdinaryObject(<span class="code-object">Object</span>, ObjectStreamClass, <span class="code-object">boolean</span>) 27262 void java.io.ObjectOutputStream.writeObject0(<span class="code-object">Object</span>, <span class="code-object">boolean</span>) 27262 void java.io.ObjectOutputStream.writeObject(<span class="code-object">Object</span>) 27262 void org.opendaylight.controller.cluster.raft.persisted.SimpleReplicatedLogEntry$Proxy.writeExternal(ObjectOutput) 27262 void java.io.ObjectOutputStream.writeExternalData(Externalizable) 27262 void java.io.ObjectOutputStream.writeOrdinaryObject(<span class="code-object">Object</span>, ObjectStreamClass, <span class="code-object">boolean</span>) 27262 void java.io.ObjectOutputStream.writeObject0(<span class="code-object">Object</span>, <span class="code-object">boolean</span>) 27262 void java.io.ObjectOutputStream.writeObject(<span class="code-object">Object</span>) 27262 void akka.serialization.JavaSerializer.$anonfun$toBinary$1(<span class="code-object">Object</span>, ObjectOutputStream) 27262 void akka.serialization.JavaSerializer$$Lambda$862.398419640.apply$mcV$sp() 27262 <span class="code-object">Object</span> scala.runtime.java8.JFunction0$mcV$sp.apply() 27262 <span class="code-object">Object</span> scala.util.DynamicVariable.withValue(<span class="code-object">Object</span>, Function0) 27262 <span class="code-object">byte</span>[] akka.serialization.JavaSerializer.toBinary(<span class="code-object">Object</span>) 27262 MessageFormats$PersistentPayload$Builder akka.persistence.serialization.MessageSerializer.payloadBuilder$1(<span class="code-object">Object</span>) 27262 MessageFormats$PersistentPayload$Builder akka.persistence.serialization.MessageSerializer.$anonfun$persistentPayloadBuilder$1(MessageSerializer, <span class="code-object">Object</span>) 27262 <span class="code-object">Object</span> akka.persistence.serialization.MessageSerializer$$Lambda$878.273383069.apply() 27262 <span class="code-object">Object</span> scala.util.DynamicVariable.withValue(<span class="code-object">Object</span>, Function0) 27262 MessageFormats$PersistentPayload$Builder akka.persistence.serialization.MessageSerializer.persistentPayloadBuilder(<span class="code-object">Object</span>) 27262 MessageFormats$PersistentMessage$Builder akka.persistence.serialization.MessageSerializer.persistentMessageBuilder(PersistentRepr) 27262 <span class="code-object">byte</span>[] akka.persistence.serialization.MessageSerializer.toBinary(<span class="code-object">Object</span>) 27262 <span class="code-object">byte</span>[] akka.serialization.Serialization.$anonfun$serialize$1(Serialization, <span class="code-object">Object</span>) 27262 <span class="code-object">Object</span> akka.serialization.Serialization$$Lambda$877.275215371.apply() 27262 Try scala.util.Try$.apply(Function0) 27262 Try akka.serialization.Serialization.serialize(<span class="code-object">Object</span>) 27262 <span class="code-object">byte</span>[] akka.persistence.journal.leveldb.LeveldbStore.persistentToBytes(PersistentRepr) 27262 <span class="code-object">byte</span>[] akka.persistence.journal.leveldb.LeveldbStore.persistentToBytes$(LeveldbStore, PersistentRepr) 27262 <span class="code-object">byte</span>[] akka.persistence.journal.leveldb.LeveldbJournal.persistentToBytes(PersistentRepr) 27262 void akka.persistence.journal.leveldb.LeveldbStore.addToMessageBatch(PersistentRepr, Set, WriteBatch) 27262 void akka.persistence.journal.leveldb.LeveldbStore.$anonfun$asyncWriteMessages$5(LeveldbStore, ObjectRef, WriteBatch, PersistentRepr) 27262 <span class="code-object">Object</span> akka.persistence.journal.leveldb.LeveldbStore.$anonfun$asyncWriteMessages$5$adapted(LeveldbStore, ObjectRef, WriteBatch, PersistentRepr) 27262 <span class="code-object">Object</span> akka.persistence.journal.leveldb.LeveldbStore$$Lambda$875.1221692884.apply(<span class="code-object">Object</span>) 27262 void scala.collection.immutable.List.foreach(Function1) 27262 void akka.persistence.journal.leveldb.LeveldbStore.$anonfun$asyncWriteMessages$4(LeveldbStore, ObjectRef, ObjectRef, WriteBatch, AtomicWrite) 27262 void akka.persistence.journal.leveldb.LeveldbStore$$Lambda$874.735864755.apply$mcV$sp() 27262 <span class="code-object">Object</span> scala.runtime.java8.JFunction0$mcV$sp.apply() 27262 Try scala.util.Try$.apply(Function0) 27262 Try akka.persistence.journal.leveldb.LeveldbStore.$anonfun$asyncWriteMessages$3(LeveldbStore, ObjectRef, ObjectRef, WriteBatch, AtomicWrite) 27262 <span class="code-object">Object</span> akka.persistence.journal.leveldb.LeveldbStore$$Lambda$873.510803830.apply(<span class="code-object">Object</span>) 27262 Builder scala.collection.TraversableLike.$anonfun$map$1(Function1, Builder, <span class="code-object">Object</span>) 27262 <span class="code-object">Object</span> scala.collection.TraversableLike$$Lambda$283.599282230.apply(<span class="code-object">Object</span>) 27262 void scala.collection.Iterator.foreach(Function1) 27262 void scala.collection.Iterator.foreach$(Iterator, Function1) 27262 void scala.collection.AbstractIterator.foreach(Function1) 27262 void scala.collection.IterableLike.foreach(Function1) 27262 void scala.collection.IterableLike.foreach$(IterableLike, Function1) 27262 void scala.collection.AbstractIterable.foreach(Function1) 27262 <span class="code-object">Object</span> scala.collection.TraversableLike.map(Function1, CanBuildFrom) 27262 <span class="code-object">Object</span> scala.collection.TraversableLike.map$(TraversableLike, Function1, CanBuildFrom) 27262 <span class="code-object">Object</span> scala.collection.AbstractTraversable.map(Function1, CanBuildFrom) 27262 Seq akka.persistence.journal.leveldb.LeveldbStore.$anonfun$asyncWriteMessages$2(LeveldbStore, Seq, ObjectRef, ObjectRef, WriteBatch) 27262 <span class="code-object">Object</span> akka.persistence.journal.leveldb.LeveldbStore$$Lambda$872.1336396232.apply(<span class="code-object">Object</span>) 27262 <span class="code-object">Object</span> akka.persistence.journal.leveldb.LeveldbStore.withBatch(Function1) 27262 <span class="code-object">Object</span> akka.persistence.journal.leveldb.LeveldbStore.withBatch$(LeveldbStore, Function1) 27262 <span class="code-object">Object</span> akka.persistence.journal.leveldb.LeveldbJournal.withBatch(Function1) 27262 Seq akka.persistence.journal.leveldb.LeveldbStore.$anonfun$asyncWriteMessages$1(LeveldbStore, Seq, ObjectRef, ObjectRef) 27262 <span class="code-object">Object</span> akka.persistence.journal.leveldb.LeveldbStore$$Lambda$871.1879377751.apply() 27262</pre> </div></div> <p>The other one is:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-object">byte</span>[] akka.protobuf.AbstractMessageLite.toByteArray() 27218 ByteString akka.remote.transport.AkkaPduProtobufCodec$.constructPayload(ByteString) 9987 <span class="code-object">boolean</span> akka.remote.transport.AkkaProtocolHandle.write(ByteString) 9987 <span class="code-object">boolean</span> akka.remote.EndpointWriter.writeSend(EndpointManager$Send) 9987 <span class="code-object">Object</span> akka.remote.EndpointWriter$$anonfun$4.applyOrElse(<span class="code-object">Object</span>, Function1) 9975 void akka.actor.Actor.aroundReceive(PartialFunction, <span class="code-object">Object</span>) 9975 void akka.actor.Actor.aroundReceive$(Actor, PartialFunction, <span class="code-object">Object</span>) 9975 void akka.remote.EndpointActor.aroundReceive(PartialFunction, <span class="code-object">Object</span>) 9975 void akka.actor.ActorCell.receiveMessage(<span class="code-object">Object</span>) 9975 void akka.actor.ActorCell.invoke(Envelope) 9975 void akka.dispatch.Mailbox.processMailbox(<span class="code-object">int</span>, <span class="code-object">long</span>) 9975 void akka.dispatch.Mailbox.run() 9975 <span class="code-object">boolean</span> akka.dispatch.Mailbox.exec() 9975 <span class="code-object">int</span> akka.dispatch.forkjoin.ForkJoinTask.doExec() 9975 void akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinTask) 9975 void akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool$WorkQueue) 9975 void akka.dispatch.forkjoin.ForkJoinWorkerThread.run() 9975</pre> </div></div> CONTROLLER-1872

Excessive byte array copy in Akka

Bug Medium Resolved Done Tom Pantelis Michael Vorburger Tue, 13 Nov 2018 13:24:17 +0000 Tue, 4 Dec 2018 15:32:49 +0000 Tue, 4 Dec 2018 15:32:49 +0000 Oxygen SR3 Neon Fluorine SR2 0 2 <p>These are normal – we cannot know the final size of the allocation until we have traversed the object. Doing a two-phase traversal wastes CPU cycles <b>more</b> than the incurrect GC/copies, which use exponential resize.</p> <p>It would eliminate some re-allocations if JavaSerializer.toBinary specified a larger initial size to the ByteArrayOutputStream. The is similar to <a href="https://jira.opendaylight.org/browse/CONTROLLER-1870" title="Excessive Arrays.copyOf memory allocation from AbstractNormalizedNodeDataOutput" class="issue-link" data-issue-key="CONTROLLER-1870"><del>CONTROLLER-1870</del></a>. We could hook in our own serializer for SimpleReplicatedLogEntry that does that. The nice part is that we can estimate the serialized size very closely via Payload.size().</p> <p>Submitted <a href="https://git.opendaylight.org/gerrit/#/c/78390/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/78390/</a></p> Development Rank 0|i03kfb: