https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-1864 controller <p>Attached GCLog.log.0.current has this:</p> <p><tt>2018-09-05T18:23:40.277+0000: 39892.345: Total time for which application threads were stopped: 8.3785112 seconds, Stopping threads took: 8.3765855 seconds</tt></p> <p>Attached Run.jfr (7z) provided by Ericsson, from their internal downstream distribution based on Boron. (This is the same that they made available via the filehosting.org <a href="https://stackoverflow.com/questions/52295844/g1-gc-large-background-i-o-causing-jvm-unresponsive-a-8sec-pause" class="external-link" target="_blank" rel="nofollow noopener">link on this SO QA</a>. Best analyzed using the JDK Mission Control from OpenJDK v11 from <a href="http://jdk.java.net/jmc/" class="external-link" target="_blank" rel="nofollow noopener">http://jdk.java.net/jmc/</a>. Produced via <a href="https://wiki.opendaylight.org/view/HowToProfilePerformance" class="external-link" target="_blank" rel="nofollow noopener">https://wiki.opendaylight.org/view/HowToProfilePerformance</a>.)</p> CONTROLLER-1864

A 8s "non-GC world stop JVM pause" during snapshot writes

Bug Medium Resolved Done Michael Vorburger Michael Vorburger Thu, 20 Sep 2018 02:11:11 +0000 Sat, 22 Sep 2018 14:30:13 +0000 Sat, 22 Sep 2018 14:26:53 +0000 Boron Carbon 0 1 <p>There was no "fix" here, but I'm filing this analysis here as an (immediately closed) JIRA issue for future ref:<br/>  If we very carefully comb over <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/wink.png" height="16" width="16" align="absmiddle" alt="" border="0"/> E// attached initial/akka.conf, we can see that it is missing this line which upstream has:<br/>  <br/> <tt>snapshot-store.local.class = "org.opendaylight.controller.cluster.persistence.LocalSnapshotStore"</tt></p> <p>Thereforefore they were still using the default akka.persistence.snapshot.local.LocalSnapshotStore (seen in the stack trace yesterdday, below) instead of ODL's custom, and optimzed, org.opendaylight.controller.cluster.persistence.LocalSnapshotStore! Either you "just forgot" that, or, more likely, that was made after Boron... see here for the full history of this story in <a href="https://jira.opendaylight.org/browse/CONTROLLER-1576" title="OOM when persisting a snapshot of the datastore that is bigger than 2G" class="issue-link" data-issue-key="CONTROLLER-1576"><del>CONTROLLER-1576</del></a> and:</p> <p>* <a href="https://github.com/opendaylight/controller/blob/master/opendaylight/md-sal/sal-clustering-commons/src/main/java/org/opendaylight/controller/cluster/persistence/LocalSnapshotStore.java" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/opendaylight/controller/blob/master/opendaylight/md-sal/sal-clustering-commons/src/main/java/org/opendaylight/controller/cluster/persistence/LocalSnapshotStore.java</a><br/> * <a href="https://github.com/opendaylight/controller/commits/master/opendaylight/md-sal/sal-clustering-commons/src/main/java/org/opendaylight/controller/cluster/persistence/LocalSnapshotStore.java" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/opendaylight/controller/commits/master/opendaylight/md-sal/sal-clustering-commons/src/main/java/org/opendaylight/controller/cluster/persistence/LocalSnapshotStore.java</a><br/> * <a href="https://git.opendaylight.org/gerrit/#/q/topic:bug/7521" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/q/topic:bug/7521</a></p> <p>Before this fix with a custom LocalSnapshotStore, the default one from Akka would create (x3 ?) a HUGE byte[] instead of "streaming directly to the file like the new custom one, so the JVM was not able to park it mid-stream, as Akka's version may hold up the safepoint while serializing to a byte[]" (summarized from private email exchange with Tom).<br/>  <br/> They als ran ODL with a huge 24 GB heap, which does not help for GC, saying "<span class="error">&#91;Dhiman&#93;</span> The reason for huge heap size is i) to tackle with huge memory allocation during PM counter’s serialization (jolokia)", and that problem was solved in Jolokia 2.5 years ago, see <a href="https://github.com/rhuss/jolokia/issues/391" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/rhuss/jolokia/issues/391</a>; "and ii) Humongous allocation for huge byte array creations before writing the snapshots" which was solved by <a href="https://jira.opendaylight.org/browse/CONTROLLER-1576" title="OOM when persisting a snapshot of the datastore that is bigger than 2G" class="issue-link" data-issue-key="CONTROLLER-1576"><del>CONTROLLER-1576</del></a> (above) 1.5 years ago, according to the JavaDoc on org.opendaylight.controller.cluster.persistence.LocalSnapshotStore. </p> <p>PS: Earlier private email about this problem suspected mmp memory mapping files related issues from LevelDB (native or pure Java impl) blocking the world before GC, but that was a wrong initial conclusion; the snapshot (contrary to the journal) actually has absolutely nothing to do with LevelDB.</p> Relates CONTROLLER-1576 OVSDB-469 Development Rank 0|i03ixz: