https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-1746 controller <p>On a single-node Carbon SR1 setup, we got an OOM over the weekend. Looking at the heap dump shows 1.1GB occupied in the frontend history's closed transactions map, with 33M entries.</p> <p>I'll upload the heap dump and add a link to it.</p> <p>Operating System: All<br/> Platform: All</p> CONTROLLER-1746

OOM with large number of closed transactions

Bug Resolved Done Robert Varga Stephen Kitt Mon, 7 Aug 2017 13:45:19 +0000 Thu, 11 Oct 2018 11:04:06 +0000 Tue, 2 Oct 2018 18:31:25 +0000 Carbon Neon Fluorine SR1 Oxygen SR4 mdsal 0 7 <p>The heap dump is on <a href="https://www.sk2.org/java_pid2098.hprof.xz" class="external-link" target="_blank" rel="nofollow noopener">https://www.sk2.org/java_pid2098.hprof.xz</a> (269MB).</p> <p>Affects both Carbon and Nitrogen. Blocking as this this a regression.</p> <p>Confirmed by analyzing the dump reported in <a href="https://git.opendaylight.org/gerrit/#/c/61034/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/61034/</a>.</p> <p>This is ask-based protocol and we are looking at transactions which have been closed but not purged. Tell-based protocol issues explicit purges from the frontend, for ask-based protocol this has to be done on the backend.</p> <p>Examining the data, these are free-standing transactions, which have been committed &#8211; hence we are missing the purge step.</p> <p>master: <a href="https://git.opendaylight.org/gerrit/61284" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/61284</a></p> <p>skitt: it’s a Carbon SR1 regression so anything older is something else</p> <p>skitt: it’s easy to spot, Eclipse MAT tells you the biggest leak culprit is a huge completedTransactions HashMap$Node[]</p> <p>(In reply to Michael Vorburger from comment #5)<br/> &gt; skitt: it’s a Carbon SR1 regression so anything older is something else<br/> &gt; <br/> &gt; skitt: it’s easy to spot, Eclipse MAT tells you the biggest leak culprit is<br/> &gt; a huge completedTransactions HashMap$Node[]</p> <p>closeTransactions, not completeTransactions</p> <p>(In reply to Stephen Kitt from comment #6)<br/> &gt; (In reply to Michael Vorburger from comment #5)<br/> &gt; &gt; skitt: it’s a Carbon SR1 regression so anything older is something else<br/> &gt; &gt; <br/> &gt; &gt; skitt: it’s easy to spot, Eclipse MAT tells you the biggest leak culprit is<br/> &gt; &gt; a huge completedTransactions HashMap$Node[]<br/> &gt; <br/> &gt; closeTransactions, not completeTransactions</p> <p>close*d*Transactions, not complete*d*Transactions</p> <p>&gt; close*d*Transactions, not complete*d*Transactions</p> <p>closedTransactions &lt;= Dear Bugzilla indexer, please dig this.. <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/wink.png" height="16" width="16" align="absmiddle" alt="" border="0"/> <span class="error">&#91;We will not find this bug in the future when searching for &quot;closedTransactions&quot; without this.&#93;</span></p> <p>carbon: <a href="https://git.opendaylight.org/gerrit/61433" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/61433</a></p> <p>Awaiting confirmation the fix works.</p> <p>The fix looks OK, I’m seeing transactions being purged properly.</p> <p>AR builds are failing on carbon and only one patch that has been merged recently. Please confirm if this change could have potentially caused an issue:</p> <p><a href="https://lists.opendaylight.org/pipermail/release/2017-August/011971.html" class="external-link" target="_blank" rel="nofollow noopener">https://lists.opendaylight.org/pipermail/release/2017-August/011971.html</a></p> <p>Yes, as the change introduces asynchronous variance to production code under normal operation. This introduces a time window of ~100ms, where UT code can observe a transient state.</p> <p>As far as we can tell the issue lies solely with the UT suite, which needs to be fixed to account for the variance in behavior between autorelease VM, verify/merge VMs and developer environment.</p> <p>Of the two issues seen in autorelease, one already has a fix, <a href="https://git.opendaylight.org/gerrit/61626" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/61626</a>, which has been reviewed. The other failure is under investigation.</p> <p>By no means can we revert the fix: it fixes a long-term stability regression we introduced between Carbon and Carbon SR1. It has been reported from the field.</p> <p><a href="https://git.opendaylight.org/gerrit/61720" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/61720</a> should fix the second UT failure</p> <p>I apologize for accidentally reopen this bug when review it. Hope not make any trouble.</p> <p>This bug wasn't fixed. Still seeing a huge amount of closedTransactions in Oxygen-sr3.</p> <p>The code in FrontendHistoryMetadataBuilder:</p> <p> </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> void onTransactionCommitted(<span class="code-keyword">final</span> TransactionIdentifier txId) { closedTransactions.put(UnsignedLong.fromLongBits(txId.getTransactionId()), <span class="code-object">Boolean</span>.TRUE); } void onTransactionPurged(<span class="code-keyword">final</span> TransactionIdentifier txId) { <span class="code-keyword">final</span> UnsignedLong id = UnsignedLong.fromLongBits(txId.getTransactionId()); closedTransactions.remove(id); purgedTransactions.add(Range.closedOpen(id, UnsignedLong.ONE.plus(id))); } </pre> </div></div> <p> </p> <p>closedTransaction will be added on transaction committed and removed on transaction purged.</p> <p>But it seems that these two operations have been disordered.</p> <p>log:set debug org.opendaylight.controller.cluster.datastore.FrontendClientMetadataBuilder</p> <p> </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> 2018-09-20 16:17:50,466 | DEBUG | dispatcher-24 | lientMetadataBuilder | member-1-shard-inventory-operational: Purged transaction member-1-datastore-operational-fe-0-chn-7-txn-58-0 2018-09-20 16:17:50,466 | DEBUG | dispatcher-24 | lientMetadataBuilder | member-1-shard-inventory-operational: Committed transaction member-1-datastore-operational-fe-0-chn-7-txn-58-0 2018-09-20 16:17:50,483 | DEBUG | dispatcher-24 | lientMetadataBuilder | member-1-shard-inventory-operational: Purged transaction member-1-datastore-operational-fe-0-chn-7-txn-59-0 2018-09-20 16:17:50,486 | DEBUG | dispatcher-24 | lientMetadataBuilder | member-1-shard-inventory-operational: Committed transaction member-1-datastore-operational-fe-0-chn-7-txn-59-0 2018-09-20 16:17:50,496 | DEBUG | dispatcher-24 | lientMetadataBuilder | member-1-shard-inventory-operational: Purged transaction member-1-datastore-operational-fe-0-chn-6-txn-90-0 2018-09-20 16:17:50,497 | DEBUG | dispatcher-24 | lientMetadataBuilder | member-1-shard-inventory-operational: Committed transaction member-1-datastore-operational-fe-0-chn-6-txn-90-0 2018-09-20 16:17:50,498 | DEBUG | dispatcher-24 | lientMetadataBuilder | member-1-shard-inventory-operational: Purged transaction member-1-datastore-operational-fe-0-chn-6-txn-91-0 2018-09-20 16:17:50,498 | DEBUG | dispatcher-24 | lientMetadataBuilder | member-1-shard-inventory-operational: Committed transaction member-1-datastore-operational-fe-0-chn-6-txn-91-0 2018-09-20 16:17:50,514 | DEBUG | dispatcher-24 | lientMetadataBuilder | member-1-shard-inventory-operational: Purged transaction member-1-datastore-operational-fe-0-chn-7-txn-60-0 2018-09-20 16:17:50,519 | DEBUG | dispatcher-24 | lientMetadataBuilder | member-1-shard-inventory-operational: Committed transaction member-1-datastore-operational-fe-0-chn-7-txn-60-0 2018-09-20 16:17:50,520 | DEBUG | dispatcher-24 | lientMetadataBuilder | member-1-shard-inventory-operational: Purged transaction member-1-datastore-operational-fe-0-chn-7-txn-61-0 2018-09-20 16:17:50,520 | DEBUG | dispatcher-24 | lientMetadataBuilder | member-1-shard-inventory-operational: Committed transaction member-1-datastore-operational-fe-0-chn-7-txn-61-0 2018-09-20 16:17:50,577 | DEBUG | dispatcher-24 | lientMetadataBuilder | member-1-shard-inventory-operational: Purged transaction member-1-datastore-operational-fe-0-chn-5-txn-85-0 2018-09-20 16:17:50,577 | DEBUG | dispatcher-24 | lientMetadataBuilder | member-1-shard-inventory-operational: Committed transaction member-1-datastore-operational-fe-0-chn-5-txn-85-0 </pre> </div></div> <p>Purged is always executed before committed for the same transaction. So the closed transactions keep being added back.</p> <p> </p> <p>I think we need frontend debugs as well to understand what is going on. Frontend should not be issuing a purge before it sees the transaction committed.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jamescch" class="user-hover" rel="jamescch">jamescch</a> is this reproducible with the configuration data store?</p> <p>So this is a single node, with operational store. Unless this is reproducible with three nodes or with configuration data store, it would seem to indicate that sal-akka-raft is playing tricks by invoking us back.</p> <p>Actually this is a problem with ShardDataTree.applyReplicatedPayload(), where we go down through payloadReplicationComplete(), but invoke allMetadataCommittedTransaction() only once we return.</p> <p>This works with the Shard.replicatePayload() shortcut to subvert us: payloadReplicationComplete() will apply the state, but will also run through completion callbacks, which will issue a purge payload replication, which will bounce on stack too – hence it will completely execute before we get to invoking allMetadataCommittedTransaction(). Thus the mis-ordering will happen. Replication/persistence will thwart the replicatePayload() shortcut, hence we are good with three nodes or persistence.</p> <p>Great! The fix works well. The order looks correct now.</p> Development External issue ID 8941 External issue URL Issue Type ODL SR Target Milestone Rank 0|i02sh3: