https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/OVSDB-469 ovsdb <p>I was suprised to see the stack trace below as the top #3 memory allocator in the JFR from <a href="https://jira.opendaylight.org/browse/CONTROLLER-1864" title="A 8s &quot;non-GC world stop JVM pause&quot; during snapshot writes" class="issue-link" data-issue-key="CONTROLLER-1864"><del>CONTROLLER-1864</del></a> showing up as 1 GB Total Allocation of String, that's a hell of a lot of objects for what org.opendaylight.ovsdb.lib.notation.Version.fromString() does!</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=thapar" class="user-hover" rel="thapar">thapar</a> or <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=skitt" class="user-hover" rel="skitt">skitt</a> or any other OVSDB maintainers, how about rewriting that simpler and without allocation?</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>String java.lang.String.substring(int, int) 2558 CharSequence java.lang.String.subSequence(int, int) 1653 CharSequence java.util.regex.Matcher.getSubSequence(int, int) 1610 String java.util.regex.Matcher.group(int) 1610 Version org.opendaylight.ovsdb.lib.notation.Version.fromString(String) 1567 Version org.opendaylight.ovsdb.lib.schema.typed.TyperUtils.getColumnUntilVersion(Method) 857 void org.opendaylight.ovsdb.lib.schema.typed.TyperUtils.checkColumnSchemaVersion(DatabaseSchema, Method) 857 void org.opendaylight.ovsdb.lib.schema.typed.TyperUtils.access$100(DatabaseSchema, Method) 857 Object org.opendaylight.ovsdb.lib.schema.typed.TyperUtils$1.processGetColumn(Method) 768 Object org.opendaylight.ovsdb.lib.schema.typed.TyperUtils$1.invoke(Object, Method, Object[]) 768 Column com.sun.proxy.$Proxy463.getLocatorColumn() 243 RemoteUcastMacs org.opendaylight.ovsdb.hwvtepsouthbound.transactions.md.HwvtepUcastMacsRemoteUpdateCommand.buildRemoteUcast(UcastMacsRemote) 243 Node org.opendaylight.ovsdb.hwvtepsouthbound.transactions.md.HwvtepUcastMacsRemoteUpdateCommand.buildConnectionNode(Collection) 243 void org.opendaylight.ovsdb.hwvtepsouthbound.transactions.md.HwvtepUcastMacsRemoteUpdateCommand.updateUcastMacsRemote(ReadWriteTransaction, Collection) 243 void org.opendaylight.ovsdb.hwvtepsouthbound.transactions.md.HwvtepUcastMacsRemoteUpdateCommand.execute(ReadWriteTransaction) 243 void org.opendaylight.ovsdb.hwvtepsouthbound.transactions.md.HwvtepOperationalCommandAggregator.execute(ReadWriteTransaction) 243 void org.opendaylight.ovsdb.hwvtepsouthbound.transactions.md.TransactionInvokerImpl.run() 243 void java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) 243 void java.util.concurrent.ThreadPoolExecutor$Worker.run() 243 void java.lang.Thread.run() 243</pre> </div></div> OVSDB-469

Very high String allocation pressure on GC due to org.opendaylight.ovsdb.lib.notation.Version.fromString()

Bug Highest Resolved Done Michael Vorburger Michael Vorburger Thu, 20 Sep 2018 02:16:25 +0000 Mon, 12 Nov 2018 16:41:58 +0000 Tue, 2 Oct 2018 17:04:06 +0000 Oxygen-SR3 Fluorine 0 3 <p>and the top #6 overall memory allocation, with 538 MB of int[] is this, which will also disappear together with above:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>void java.util.regex.Matcher.&lt;init&gt;(Pattern, CharSequence) 4943 Matcher java.util.regex.Pattern.matcher(CharSequence) 4938 Version org.opendaylight.ovsdb.lib.notation.Version.fromString(String) 2960 Version org.opendaylight.ovsdb.lib.schema.typed.TyperUtils.getColumnUntilVersion(Method) 1428 void org.opendaylight.ovsdb.lib.schema.typed.TyperUtils.checkColumnSchemaVersion(DatabaseSchema, Method) 1428 void org.opendaylight.ovsdb.lib.schema.typed.TyperUtils.access$100(DatabaseSchema, Method) 1428 Object org.opendaylight.ovsdb.lib.schema.typed.TyperUtils$1.processGetColumn(Method) 1009 Object org.opendaylight.ovsdb.lib.schema.typed.TyperUtils$1.invoke(Object, Method, Object[]) 1009 Column com.sun.proxy.$Proxy463.getLogicalSwitchColumn() 385 RemoteUcastMacs org.opendaylight.ovsdb.hwvtepsouthbound.transactions.md.HwvtepUcastMacsRemoteUpdateCommand.buildRemoteUcast(UcastMacsRemote) 385 Node org.opendaylight.ovsdb.hwvtepsouthbound.transactions.md.HwvtepUcastMacsRemoteUpdateCommand.buildConnectionNode(Collection) 385 void org.opendaylight.ovsdb.hwvtepsouthbound.transactions.md.HwvtepUcastMacsRemoteUpdateCommand.updateUcastMacsRemote(ReadWriteTransaction, Collection) 385 void org.opendaylight.ovsdb.hwvtepsouthbound.transactions.md.HwvtepUcastMacsRemoteUpdateCommand.execute(ReadWriteTransaction) 385 void org.opendaylight.ovsdb.hwvtepsouthbound.transactions.md.HwvtepOperationalCommandAggregator.execute(ReadWriteTransaction) 385 void org.opendaylight.ovsdb.hwvtepsouthbound.transactions.md.TransactionInvokerImpl.run() 385 void java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) 385 void java.util.concurrent.ThreadPoolExecutor$Worker.run() 385 void java.lang.Thread.run() 385 </pre> </div></div> <p>I believe these are from JFRs shared by E/// team? This is from HWVTEP. This column lists all macs that are available to this particular port. The way schema is designed and based on deployment, this will be every other mac in network*N where N is number of TOR devices.</p> <p>I am not sure if fromString is the culprit here, the string in ColumnVersion will be something like 1.7.12, it is literally the schema version string. It is the data in this column that can grow huge.</p> <p>Are we sure it is the fromVersion that is doing huge allocation and not contents of column itself?</p> <p>Yes, JFRs shared by E/// team (<a href="https://jira.opendaylight.org/browse/CONTROLLER-1864" title="A 8s &quot;non-GC world stop JVM pause&quot; during snapshot writes" class="issue-link" data-issue-key="CONTROLLER-1864"><del>CONTROLLER-1864</del></a>), and Yes the wait I interpret the stack trace shown it's the fromVersion that is doing huge allocation .. I'm not sure what exactly you mean by "contents of column itself", but the stack trace which I see in JMC (above) should look different if it wasn't about the allocation in fromVersion? At the worst, the change I just proposed (<a href="https://git.opendaylight.org/gerrit/#/c/76290/)" class="external-link" target="_blank" rel="nofollow noopener">c/76290)</a> cannot hurt, and should more that looks different in a future JFR.</p> <p>Then something else is wrong. FromVersion can't and shouldn't be doing huge allocations coz version string will never be anything bigger than 'xxx.yyy.zzz'. How can a string like that cause huge allocations?</p> <p>For every table and column in OVSDB we define a from,to versions <span class="error">&#91;e.g. 1.0.0, 2.0.0&#93;</span> to determine if a given switch supports a given column or not. This caode is just to check if column is supported or not. This simple version comparison to see if column falls within range or not.</p> <p>If issue were too many small allocations, I'd understand, but huge allocation is just not possible here. We're missing something significant.</p> <p>Nope, as a wrote, the issue is 1 GB but not of a huge String (I never said that, you interpreted) of many small, substring() String; specifically x3 for each part of the Version. What is the harm in getting c/76290 merged and seeing? It cannot hurt, at worst you are right and it doesn't help, at best I'm right and this overallocation disappears. Worth a try!</p> <p>Aha, makes sense. Sorry for confusion.</p> <p>Trading allocation for performance hit which definitely will be there? There is a reason pattern was used in first place. I'd prefer running tests with a distribution zip off the patch, getting results before we merge.</p> <p>Worst case is not that this doesn't help. Worst case is it makes everything too slow and scale numbers get even worse.</p> <p>EDIT:<br/> Am trying to come up with a patch that can significantly reduce the number of times this method needs to be called to the point where we won't end up with so huge allocation. I think I might be able to figure out a redesign that will give best results.</p> <p>&gt; performance hit which definitely will be there? There is a reason pattern was used in first place.</p> <p>What? What makes you say that? The change proposed in <a href="https://git.opendaylight.org/gerrit/#/c/76290/" class="external-link" target="_blank" rel="nofollow noopener">c/76290</a> will, if anything, make Version fromString() faster, not slower... FYI that parse() method is directly inspired by JDK's Integer.parseInt.  <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=skitt" class="user-hover" rel="skitt">skitt</a> perhaps you would like to weight in here? <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/biggrin.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>&gt; I'd prefer running tests with a distribution zip off the patch, getting results before we merge</p> <p>sure, always a good idea, can you CSIT it? Shout if anything broken (due to this), doubt it.</p> <p>I think Michael’s patch is definitely worth it, thanks. It’s a shame parseInt() can’t deal with substring parsing itself! NumberFormat can, but it’s too expensive.</p> <p>I've back-ported (manual cherry-pick and conflict resolution) <a href="https://git.opendaylight.org/gerrit/#/c/76290/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/76290/</a> from Neon master to Fluorine (<a href="https://git.opendaylight.org/gerrit/#/c/76568)" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/76568)</a> and Oxygen (<a href="https://git.opendaylight.org/gerrit/#/c/76569" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/76569</a>).</p> Relates CONTROLLER-1864 Development Priority Rank 0|i03iy7: