[NETVIRT-883] Umbrella parent issue for grouping all suspected transaction leaks Created: 04/Sep/17  Updated: 19/Feb/18  Resolved: 19/Feb/18

Status: Resolved
Project: netvirt
Component/s: General
Affects Version/s: Nitrogen
Fix Version/s: None

Type: Bug Priority: Medium
Reporter: Michael Vorburger Assignee: Michael Vorburger
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: All
Platform: All

Attachments: Text File open-transactions.txt     Text File open-transactions.txt    
Issue Links:
Blocks
blocks NETVIRT-878 CSIT should help to detect possible m... In Progress
is blocked by CONTROLLER-1756 OOM due to huge Map in ShardDataTree Resolved
is blocked by CONTROLLER-1760 Tooling to find the real root cause c... Resolved
is blocked by CONTROLLER-1765 trace:transactions should suppress fa... Resolved
is blocked by GENIUS-89 Suspected WriteOnlyTransaction leak i... Resolved
is blocked by NETVIRT-884 Suspected WriteOnlyTransaction leak i... Resolved
is blocked by NETVIRT-885 Suspected ReadOnlyTransaction leak in... Resolved
is blocked by NETVIRT-886 Suspected ReadOnlyTransaction leak in... Resolved
is blocked by NETVIRT-887 Suspected WriteOnlyTransaction leak i... Resolved
is blocked by OPNFLWPLUG-933 IllegalStateException: Attempted to c... Resolved
is blocked by OPNFLWPLUG-934 ReadOnlyTransaction created in FlowCa... Resolved
is blocked by OPNFLWPLUG-935 TransactionChain created in Operation... Resolved
is blocked by OPNFLWPLUG-940 Suspected (not sure) TransactionChain... Resolved
is blocked by OVSDB-423 TransactionChain created in Transacti... Resolved
is blocked by OVSDB-424 TransactionChain created in hwvtepsou... Resolved
is blocked by OVSDB-425 Suspected ReadOnlyTransaction leak in... Resolved
is blocked by OVSDB-426 Suspected ReadOnlyTransaction leak in... Resolved
is blocked by NETVIRT-1000 Transaction Leaks in neutronvpn, fibm... Resolved
is blocked by NETVIRT-888 Suspected WriteOnlyTransaction leaks ... Resolved
is blocked by NETVIRT-985 java.lang.OutOfMemoryError: Java heap... Resolved
External issue ID: 9096

 Description   

This new parent issue will have all suspected DataBroker transaction leaks we find using the CONTROLLER-1760 tooling linked to it as blocking issues.

Some of the issues I'm about to link here were originally linked to as blocking CONTROLLER-1756 (which is what originally prompted this investigation), but has subsequently been found to not or unlikely be the cause of that particular OOM (in our scenario of stress tests with OpenStack Rally) - but should ideally still all be fixed - thus this new bug.

NB discussion in CONTROLLER-1765 re. possible false positives from CONTROLLER-1760.

NETVIRT-878 will probably need all of these to be fixed.

 Comments   
Comment by Michael Vorburger [ 04/Sep/17 ]

I've created new depends on linked issues for some of that appears on https://paste.fedoraproject.org/paste/rZ5iTpiP5Eqm4aoYVFyzHA (except for the only x1 TransactionChain, due to CONTROLLER-1765). We expect to get a new such dump tomorrow from a system after a full Rally load test, and will have more interesting Nx in there, which will probably help to understand priorities among these.

Comment by Michael Vorburger [ 04/Sep/17 ]

> new such dump tomorrow from a system after a full Rally load test,
> and will have more interesting Nx in there, which will probably help
> to understand priorities among these.

Attached new open-transactions.txt from a (quote Sridhar, tx!) "single iteration of rally scenario with the following configuration/scenarios" :

Concurrency: 16, 32
times: 300
a. create-list-network
b. create-list-port
c. create-list-router
d. create-list-security-group
e. create-list-subnet

shows 100s and in some VTN-19-2000 non-closed ReadOnlyTransaction and WriteOnlyTransaction obtained directly from DataBroker, and a handful (few) open TransactionChain.

Comment by Michael Vorburger [ 04/Sep/17 ]

Attachment open-transactions.txt has been added with description: CONTROLLER-1760 trace:transactions output from rally scale test scenario

Comment by Michael Vorburger [ 07/Sep/17 ]

Attaching latest trace:transactions - list is already getting smaller - good!

Comment by Michael Vorburger [ 07/Sep/17 ]

Attachment open-transactions.txt has been added with description: New trace:transactions output re-test of a custom distribution of latest (yesterday's) carbon/stable together with c/62674/5

Comment by Stephen Kitt [ 12/Sep/17 ]

https://git.opendaylight.org/gerrit/63027 should help with this in general.

Comment by Michael Vorburger [ 19/Feb/18 ]

This is considered Done and complete - until we find the next ones (which NETVIRT-878 should help prevent).

Generated at Wed Feb 07 20:22:43 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.