https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-1870 controller <p>I'm looking at a Java Flight Recording obtained from (internal) scale lab testing, and see extensive "TLAB Allocations" due to what appears to be very excessive Arrays.copyOf memory allocation from AbstractNormalizedNodeDataOutput:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"><span class="code-object">byte</span>[] java.util.Arrays.copyOf(<span class="code-object">byte</span>[], <span class="code-object">int</span>) 1017 void java.io.ByteArrayOutputStream.grow(<span class="code-object">int</span>) 808 void java.io.ByteArrayOutputStream.ensureCapacity(<span class="code-object">int</span>) 808 void java.io.ByteArrayOutputStream.write(<span class="code-object">int</span>) 429 void java.io.DataOutputStream.writeInt(<span class="code-object">int</span>) 373 void com.google.common.io.ByteStreams$ByteArrayDataOutputStream.writeInt(<span class="code-object">int</span>) 373 void org.opendaylight.controller.cluster.datastore.node.utils.stream.AbstractNormalizedNodeDataOutput.writeInt(<span class="code-object">int</span>) 373 void org.opendaylight.controller.cluster.datastore.node.utils.stream.NormalizedNodeOutputStreamWriter.writeString(<span class="code-object">String</span>) 373 void org.opendaylight.controller.cluster.datastore.node.utils.stream.NormalizedNodeOutputStreamWriter.writeQName(QName) 373 void org.opendaylight.controller.cluster.datastore.node.utils.stream.AbstractNormalizedNodeDataOutput.startNode(QName, <span class="code-object">byte</span>) 195 void org.opendaylight.controller.cluster.datastore.node.utils.stream.AbstractNormalizedNodeDataOutput.leafNode(YangInstanceIdentifier$NodeIdentifier, <span class="code-object">Object</span>) 195 <span class="code-object">boolean</span> org.opendaylight.yangtools.yang.data.api.schema.stream.NormalizedNodeWriter.wasProcessAsSimpleNode(NormalizedNode) 195 NormalizedNodeWriter org.opendaylight.yangtools.yang.data.api.schema.stream.NormalizedNodeWriter.write(NormalizedNode) 195 void org.opendaylight.controller.cluster.datastore.node.utils.stream.AbstractNormalizedNodeDataOutput.writeNormalizedNode(NormalizedNode) 195 void org.opendaylight.controller.cluster.datastore.persisted.DataTreeCandidateInputOutput.writeNode(NormalizedNodeDataOutput, DataTreeCandidateNode) 195 void org.opendaylight.controller.cluster.datastore.persisted.DataTreeCandidateInputOutput.writeChildren(NormalizedNodeDataOutput, Collection) 195 void org.opendaylight.controller.cluster.datastore.persisted.DataTreeCandidateInputOutput.writeNode(NormalizedNodeDataOutput, DataTreeCandidateNode) 195 void org.opendaylight.controller.cluster.datastore.persisted.DataTreeCandidateInputOutput.writeChildren(NormalizedNodeDataOutput, Collection) 195 void org.opendaylight.controller.cluster.datastore.persisted.DataTreeCandidateInputOutput.writeNode(NormalizedNodeDataOutput, DataTreeCandidateNode) 195 void org.opendaylight.controller.cluster.datastore.persisted.DataTreeCandidateInputOutput.writeChildren(NormalizedNodeDataOutput, Collection) 195 void org.opendaylight.controller.cluster.datastore.persisted.DataTreeCandidateInputOutput.writeNode(NormalizedNodeDataOutput, DataTreeCandidateNode) 195 void org.opendaylight.controller.cluster.datastore.persisted.DataTreeCandidateInputOutput.writeChildren(NormalizedNodeDataOutput, Collection) 195 void org.opendaylight.controller.cluster.datastore.persisted.DataTreeCandidateInputOutput.writeNode(NormalizedNodeDataOutput, DataTreeCandidateNode) 195 void org.opendaylight.controller.cluster.datastore.persisted.DataTreeCandidateInputOutput.writeChildren(NormalizedNodeDataOutput, Collection) 195 void org.opendaylight.controller.cluster.datastore.persisted.DataTreeCandidateInputOutput.writeNode(NormalizedNodeDataOutput, DataTreeCandidateNode) 195 void org.opendaylight.controller.cluster.datastore.persisted.DataTreeCandidateInputOutput.writeChildren(NormalizedNodeDataOutput, Collection) 195 void org.opendaylight.controller.cluster.datastore.persisted.DataTreeCandidateInputOutput.writeNode(NormalizedNodeDataOutput, DataTreeCandidateNode) 183 void org.opendaylight.controller.cluster.datastore.persisted.DataTreeCandidateInputOutput.writeChildren(NormalizedNodeDataOutput, Collection) 183 void org.opendaylight.controller.cluster.datastore.persisted.DataTreeCandidateInputOutput.writeNode(NormalizedNodeDataOutput, DataTreeCandidateNode) 175 void org.opendaylight.controller.cluster.datastore.persisted.DataTreeCandidateInputOutput.writeChildren(NormalizedNodeDataOutput, Collection) 175 void org.opendaylight.controller.cluster.datastore.persisted.DataTreeCandidateInputOutput.writeNode(NormalizedNodeDataOutput, DataTreeCandidateNode) 175 void org.opendaylight.controller.cluster.datastore.persisted.DataTreeCandidateInputOutput.writeChildren(NormalizedNodeDataOutput, Collection) 175 void org.opendaylight.controller.cluster.datastore.persisted.DataTreeCandidateInputOutput.writeDataTreeCandidate(DataOutput, DataTreeCandidate) 175 CommitTransactionPayload org.opendaylight.controller.cluster.datastore.persisted.CommitTransactionPayload.create(TransactionIdentifier, DataTreeCandidate) 175 void org.opendaylight.controller.cluster.datastore.ShardDataTree.startCommit(SimpleShardDataTreeCohort, DataTreeCandidate) 175 void org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.commit(FutureCallback) 175 void org.opendaylight.controller.cluster.datastore.CohortEntry.commit(FutureCallback) 175 void org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.finishCommit(ActorRef, CohortEntry) 175 void org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator$3.onSuccess(DataTreeCandidate) 175 void org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator$3.onSuccess(<span class="code-object">Object</span>) 175 void org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.successfulPreCommit(DataTreeCandidateTip) 175 void org.opendaylight.controller.cluster.datastore.ShardDataTree$1.onSuccess(<span class="code-object">Void</span>) 175 void org.opendaylight.controller.cluster.datastore.ShardDataTree$1.onSuccess(<span class="code-object">Object</span>) 175 void org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.doUserPreCommit(FutureCallback) 175 void org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.userPreCommit(DataTreeCandidate, FutureCallback) 175 void org.opendaylight.controller.cluster.datastore.ShardDataTree.startPreCommit(SimpleShardDataTreeCohort) 175 void org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.preCommit(FutureCallback) 175 void org.opendaylight.controller.cluster.datastore.CohortEntry.preCommit(FutureCallback) 175 void org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.doCommit(CohortEntry) 175 void org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator$2.onSuccess(<span class="code-object">Void</span>) 175 void org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator$2.onSuccess(<span class="code-object">Object</span>) 175 void org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.successfulCanCommit() 175 void org.opendaylight.controller.cluster.datastore.ShardDataTree.lambda$processNextPendingTransaction$0(ShardDataTree$CommitEntry) 175 void org.opendaylight.controller.cluster.datastore.ShardDataTree$$Lambda$868.1938452935.accept(<span class="code-object">Object</span>) 175 void org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPending(Queue, ShardDataTreeCohort$State, Consumer) 175 void org.opendaylight.controller.cluster.datastore.ShardDataTree.processNextPendingTransaction() 175 void org.opendaylight.controller.cluster.datastore.ShardDataTree.startCanCommit(SimpleShardDataTreeCohort) 175 void org.opendaylight.controller.cluster.datastore.SimpleShardDataTreeCohort.canCommit(FutureCallback) 175 void org.opendaylight.controller.cluster.datastore.CohortEntry.canCommit(FutureCallback) 175 void org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.handleCanCommit(CohortEntry) 175 void org.opendaylight.controller.cluster.datastore.ShardCommitCoordinator.handleReadyLocalTransaction(ReadyLocalTransaction, ActorRef, Shard) 175 void org.opendaylight.controller.cluster.datastore.Shard.handleReadyLocalTransaction(ReadyLocalTransaction) 175 void org.opendaylight.controller.cluster.datastore.Shard.handleNonRaftCommand(<span class="code-object">Object</span>) 175 void org.opendaylight.controller.cluster.raft.RaftActor.handleCommand(<span class="code-object">Object</span>) 175</pre> </div></div> CONTROLLER-1870

Excessive Arrays.copyOf memory allocation from AbstractNormalizedNodeDataOutput

Improvement Medium Resolved Done Tom Pantelis Michael Vorburger Mon, 12 Nov 2018 17:37:01 +0000 Wed, 19 Dec 2018 00:54:51 +0000 Wed, 19 Dec 2018 00:54:51 +0000 Oxygen SR3 Neon 0 4 <p>We can pass in a larger initial size to <em>ByteStreams.newDataOutput</em> in <em>CommitTransactionPayload.create</em> to avoid excessive reallocations.</p> <p>Note that the JFR “stack traces” only provide a partial view:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-object">byte</span>[] java.util.Arrays.copyOf(<span class="code-object">byte</span>[], <span class="code-object">int</span>) 1017 void java.io.ByteArrayOutputStream.grow(<span class="code-object">int</span>) 808 void java.io.ByteArrayOutputStream.ensureCapacity(<span class="code-object">int</span>) 808 void java.io.ByteArrayOutputStream.write(<span class="code-object">int</span>) 429 void java.io.DataOutputStream.writeInt(<span class="code-object">int</span>) 373 void com.google.common.io.ByteStreams$ByteArrayDataOutputStream.writeInt(<span class="code-object">int</span>) 373 </pre> </div></div> <p>means that 1017 calls to copyOf() resulted in a new TLAB; of those 1017 calls, 808 were a result of grow(); of those 808, 429 were a result of write(); of those 429, 373 were a result of writeInt(). All the other calls aren’t explained by this output.</p> <blockquote><p>We can pass in a larger initial size to <em>ByteStreams.newDataOutput</em> in <em>CommitTransactionPayload.create</em> to avoid excessive reallocations.</p></blockquote> <p>You mean fixed? It will change, considerably, based on the exact NN, won't it? Or just be made a configurable max. option? But then we'd overallocate.</p> <p>Perhaps we could do something smarter and "guesstimate" the req size? Like some calculation on the NormalizedNode to pre-determine how many bytes it requires when streamed. (We know the format, and it would be hard-coded to that format.)</p> <p>Or... couldn't we rewrite things to completely avoid a byte[] and only stream? (I haven't looked into the code.)</p> <p>Fixed (and configurable) might suffice, say 512 sounds reasonable - that would eliminate 4 re-allocations. Doing a "guesstimate" would require pre-walking the NN tree which would add expense.</p> <p>The reason it's converted to a byte[] is that the serialized format has a much smaller footprint - the payload is stored in the in-memory journal so that can be significant.</p> <p>Yeah, guestimating the size will lower TLAB allocation, bug will bring down the overall performance, as we have to walk the tree twice – that certainly is not worth it.</p> <p>Submitted <a href="https://git.opendaylight.org/gerrit/#/c/78433/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/78433/</a></p> <p>This is in Neon now, let's close this, for now; we can cherry-pick to Fluorine later, if needed?</p> Development Rank 0|i03kbz: