[#CONTROLLER-1879] OOM due to huge RangeSet of purgedTransactions in FrontendHistoryMetadataBuilder

[CONTROLLER-1879] OOM due to huge RangeSet of purgedTransactions in FrontendHistoryMetadataBuilder Created: 24/Dec/18 Updated: 09/Nov/21 Resolved: 23/May/19
Status:	Resolved
Project:	controller
Component/s:	clustering
Affects Version/s:	Fluorine SR1
Fix Version/s:	Sodium, Fluorine SR3, Neon SR2

Type:

Bug

Priority:

Highest

Reporter:

Huafei Zhang

Assignee:

Tomas Cere

Resolution:

Done

Votes:

Labels:

None

Remaining Estimate:

Not Specified

Time Spent:

Not Specified

Original Estimate:

Not Specified

Attachments:

Clip(12-12-13-42-56)(9).png

Issue Links:

Issue split
split to	~~CONTROLLER-1991~~	Tell-based protocol can leak transact...	Resolved

Description

Submitting a great mount of ReadOnlyTransaction and WritingTransaction in turns causes OOM.

We think it is the transaction id of ReadOnlyTransaction that splits the purged transaction range into lots of segments.

Comments

Comment by Robert Varga [ 24/Dec/18 ]

This is a controller issue, if confirmed. What are the steps to reproduce? Are you sure those ReadOnlyTransactions are being closed?

Comment by Huafei Zhang [ 26/Dec/18 ]

Yes, ReaOnlyTransaction is explicitly closed.

We have wrote a endless loop structure to reproduce this issue.

in each loop, we create a ReadOnlyTransaction and WriteTransaction to read and write the same network-topology node.

in not a very long time , the 2G memory runs out.

Comment by Robert Varga [ 07/May/19 ]

There are two issues here:

in the remote case, aborting a read-only transaction causes a leak in closedTransactions, as the transaction will be tracked as closed but not purged
in the local case, aborting a read-only transaction causes a hole in purgedTransactions to appear, as the abort will be purely-local and backend will never see the aborted transaction – which leads to the reported leak.

Comment by Robert Varga [ 07/May/19 ]

Generated at Wed Feb 07 19:56:41 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.