https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-1983 controller <p>It is impossible to run cluster node on VM with less than 6GiB RAM.</p> <p>When there is less memory we get:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2021-06-08T16:12:16,816 | ERROR | opendaylight-cluster-data-akka.actor.default-dispatcher-6 | ActorSystemImpl | 203 - org.opendaylight.controller.repackaged-akka - 4.0.0.SNAPSHOT | Uncaught error from thread [opendaylight-cluster-data-akka.remote.default-remote-dispatcher-7]: Direct buffer memory, shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[opendaylight-cluster-data] java.lang.OutOfMemoryError: Direct buffer memory at java.nio.Bits.reserveMemory(Bits.java:175) ~[?:?] at java.nio.DirectByteBuffer.&lt;init&gt;(DirectByteBuffer.java:118) ~[?:?] at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:317) ~[?:?] at akka.remote.artery.EnvelopeBufferPool.acquire(EnvelopeBufferPool.scala:34) ~[bundleFile:?] at akka.remote.artery.Encoder$$anon$1.onPush(Codecs.scala:117) ~[bundleFile:?] at akka.stream.impl.fusing.GraphInterpreter.processPush(GraphInterpreter.scala:541) ~[bundleFile:?] at akka.stream.impl.fusing.GraphInterpreter.processEvent(GraphInterpreter.scala:495) ~[bundleFile:?] at akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:390) ~[bundleFile:?] at akka.stream.impl.fusing.GraphInterpreterShell.runBatch(ActorGraphInterpreter.scala:625) ~[bundleFile:?] at akka.stream.impl.fusing.GraphInterpreterShell$AsyncInput.execute(ActorGraphInterpreter.scala:502) ~[bundleFile:?] at akka.stream.impl.fusing.GraphInterpreterShell.processEvent(ActorGraphInterpreter.scala:600) ~[bundleFile:?] at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:773) ~[bundleFile:?] at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$shortCircuitBatch(ActorGraphInterpreter.scala:762) ~[bundleFile:?] at akka.stream.impl.fusing.ActorGraphInterpreter.preStart(ActorGraphInterpreter.scala:753) ~[bundleFile:?] at akka.actor.Actor.aroundPreStart(Actor.scala:548) ~[bundleFile:?] at akka.actor.Actor.aroundPreStart$(Actor.scala:548) ~[bundleFile:?] at akka.stream.impl.fusing.ActorGraphInterpreter.aroundPreStart(ActorGraphInterpreter.scala:691) ~[bundleFile:?] at akka.actor.ActorCell.create(ActorCell.scala:641) ~[bundleFile:?] at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:513) [bundleFile:?] at akka.actor.ActorCell.systemInvoke(ActorCell.scala:535) [bundleFile:?] at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:295) [bundleFile:?] at akka.dispatch.Mailbox.run(Mailbox.scala:230) [bundleFile:?] at akka.dispatch.Mailbox.exec(Mailbox.scala:243) [bundleFile:?] at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) [?:?] at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020) [?:?] at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) [?:?] at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) [?:?] at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183) [?:?] </pre> </div></div> <p> </p> CONTROLLER-1983

Akka artery fails with java.lang.OutOfMemoryError: Direct buffer memory

Bug High Resolved Done Robert Varga Ivan Hrasko pt Wed, 9 Jun 2021 10:23:38 +0000 Thu, 21 Apr 2022 14:42:14 +0000 Thu, 13 Jan 2022 14:41:34 +0000 4.0.0 5.0.0 0 4 <p>This problem has occurred after switching to Akka artery and is similar to: <a href="https://jira.opendaylight.org/browse/CONTROLLER-1575" class="external-link" rel="nofollow">https://jira.opendaylight.org/browse/CONTROLLER-1575</a></p> <p>I have made a test where I am investigating the size of direct memory used and direct memory allocated.</p> <p>Test preparation:<br/> <b>1. run 3 ODL nodes on the VMs.</b></p> <p>VMs configuration: 2048MB RAM, 2 CPUs<br/> JAVA_OPTS=-XX:MaxDirectMemorySize=1024m</p> <p>Statistic at time of ODL deployment (after Karaf starts):<br/> Java Heap size: approx 66MB<br/> Metaspace size: approx. 62MB<br/> Direct buffer usage (non Heap): approx. 90MB</p> <p>This step is without problems.</p> <p><b>2. Install clustering features (odl-netconf-clustered-topology odl-restconf-nb-rfc8040 odl-restconf-nb-bierman02).</b></p> <p>During this step OOM error appears and cluster is not operable.</p> <p>Statistics at time of the crash:<br/> Java Heap size: approx. 199MB<br/> Metaspace size: approx. 135MB<br/> Direct buffer usage (non Heap): approx. 175MB </p> <p>In debugger we can see (<a href="https://github.com/openjdk/jdk/commit/24c70fcf8845b447362aa1da9e672ca896c181c3):" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/openjdk/jdk/commit/24c70fcf8845b447362aa1da9e672ca896c181c3</a>):</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2021-06-22T16:20:44.274684+02:00[Europe/Bratislava] :Cannot reserve 419430400 bytes of direct buffer memory (allocated: 839028241, limit: 1073741824) </pre> </div></div> <p>In jconsole we can see the non-heap usage is less than 200MB. See picture Screenshot_2021-06-22_16-22-46.png. </p> <p><b>The result:</b><br/> The consumption of direct memory is much less than the amout of memory that application is trying to allocate<br/> (approx. 175MB really used compared to more than 1024MB trying to allocate).</p> <p>I think that the application in fact needs less memory than artery is trying to allocate and this is casused<br/> by wrong configuration of AKKA artery (<a href="https://github.com/opendaylight/controller/commit/229c55fbdc5907129c9c35b3e98963b5fa2528f0#diff-5a2d9e6a9a1f74a3fa92b683a519f987d8a6d5d200846028eeb98a5c56e5c2beR109" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/opendaylight/controller/commit/229c55fbdc5907129c9c35b3e98963b5fa2528f0#diff-5a2d9e6a9a1f74a3fa92b683a519f987d8a6d5d200846028eeb98a5c56e5c2beR109</a>).</p> <p>The frame size is set too much high.</p> <p>Akka is trying to allocate the full size of the buffer - 400 MB regardless of the actual needs.</p> <p>We should use lower frame size for most common traffic and large frame size for actors with traffic with very large data.</p> <p>IMO: We should decrease maximum-frame-size and use maximum-large-frame-size.<br/> The another possibility is to decrease the number of buffers because now our direct<br/> memory demand is 128 * 400MB.<br/> See: <a href="https://github.com/akka/akka/issues/22133" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/akka/akka/issues/22133</a></p> <p>When increasing maximum-large-frame-size we have to be aware of:<br/> <a href="https://github.com/akka/akka/issues/22257" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/akka/akka/issues/22257</a>.<br/> We have to increase aeron.term.buffer.length (by default 16MB) and set large-message-destinations.<br/> See: <a href="https://github.com/real-logic/Aeron/wiki/Configuration-Options" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/real-logic/Aeron/wiki/Configuration-Options</a><br/> and <a href="https://stackoverflow.com/questions/32243664/what-is-the-largest-message-aeron-can-process" class="external-link" target="_blank" rel="nofollow noopener">https://stackoverflow.com/questions/32243664/what-is-the-largest-message-aeron-can-process</a></p> <p>See also: <a href="https://doc.akka.io/docs/akka/current/general/configuration-reference.html" class="external-link" target="_blank" rel="nofollow noopener">https://doc.akka.io/docs/akka/current/general/configuration-reference.html</a></p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=rovarga" class="user-hover" rel="rovarga">rovarga</a> Do we know destinations (actors) that require to use large frames? </p> <p>Maybe the best would be to use streams: <a href="https://doc.akka.io/docs/akka/2.6.15/stream/stream-io.html?language=java" class="external-link" target="_blank" rel="nofollow noopener">https://doc.akka.io/docs/akka/2.6.15/stream/stream-io.html?language=java</a></p> <p>Hmm, so these can be any of the AbstractRaftActor subclasses, as well as DataTreeChange actors. Probably others – essentially any actor which sends or receives any message containing a NormalizedNode.</p> <p>Actually what we'd like to leverage <a href="https://doc.akka.io/docs/akka/current/typed/reliable-delivery.html#chunk-large-messages" class="external-link" target="_blank" rel="nofollow noopener">https://doc.akka.io/docs/akka/current/typed/reliable-delivery.html#chunk-large-messages</a> – but that requires switching to Akka Typed, which is a rather large undertaking.</p> <p>Another option is that we ditch akka persistence (which seems very likely, as it is a very bad fit). Based on what that's going to do to messaging patterns and replication capabilities, we may end up with something completely different <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>And yet another option (for a FE/BE) is switching to tell-based protocol, but that requires addressing <a href="https://jira.opendaylight.org/browse/CONTROLLER-1991" title="Tell-based protocol can leak transaction IDs on the backend" class="issue-link" data-issue-key="CONTROLLER-1991"><del>CONTROLLER-1896</del></a>.</p> <p> </p> <p>So the problem here is actually <a href="https://doc.akka.io/docs/akka/current/serialization.html" class="external-link" target="_blank" rel="nofollow noopener">https://doc.akka.io/docs/akka/current/serialization.html</a></p> <p>That API requires us to serialize every message into a byte[], and I think Akka will not fragment that byte[] into Aeron, i.e. it will always send it verbatim.</p> <p>So here is where we really would like to interface Aeron directly with a Shard's journal – this makes utter sense, as we have immutable DTOs on input (NormalizedNode and such) and can serialize into any DataOutput, I think. We therefore can transparently fragment each message as we are serializing it – and it is just a matter having the proper integration surface.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=tcere" class="user-hover" rel="tcere">tcere</a> this is the Aeron part we talked about today. We actually have the tools needed to do automatic fragmentation (<a href="https://github.com/opendaylight/controller/blob/master/opendaylight/md-sal/sal-clustering-commons/src/main/java/org/opendaylight/controller/cluster/io/ChunkedOutputStream.java#L62)" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/opendaylight/controller/blob/master/opendaylight/md-sal/sal-clustering-commons/src/main/java/org/opendaylight/controller/cluster/io/ChunkedOutputStream.java#L62)</a> and obviously defrag as well</p> Issue split CONTROLLER-1987 Relates CONTROLLER-1991 CONTROLLER-2040 Development Rank 0|i03ybr: