[GENIUS-102] New OOM due to more TX leaks seen in the nova-boot scenario (interfacemanager) Created: 30/Nov/17  Updated: 31/Jul/18  Resolved: 31/Jul/18

Status: Resolved
Project: genius
Component/s: None
Affects Version/s: Carbon, Nitrogen, Oxygen
Fix Version/s: Oxygen-SR3, Oxygen, Fluorine

Type: Bug Priority: Highest
Reporter: Michael Vorburger Assignee: Stephen Kitt
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File controller-0_open-transactions.txt     Text File controller-1_open-transactions.txt     Text File controller-2_open-transactions.txt    
Issue Links:
Blocks
blocks GENIUS-176 Convert all of genius to managed tran... In Progress
blocks NETVIRT-878 CSIT should help to detect possible m... In Progress
blocks NETVIRT-1384 Umbrella: Numerous new transaction leaks Resolved
is blocked by NETVIRT-985 java.lang.OutOfMemoryError: Java heap... Resolved
Cloners
clones OVSDB-435 New OOM due to more TX leaks seen in ... Resolved

 Description   

Following the (many!) fixes that we did in NETVIRT-985, a new round of testing on the latest stable/carbon using the "nova-boot scenario" has hit an OOM again. A new "trace:transaction" output reveals more TX leaks which we need to plug; see attached.

The biggest one with 273x open newReadOnlyTransaction likely causing the new OOM this time appears in (1) ovsdb; https://git.opendaylight.org/gerrit/#/c/65567/ seems to not have helped for that one - curious... or we made a mess in downstream porting?!

Filing this in JIRA project ovsdb due to above; but the attached shows a number of other leaks in (2) openflowplugin and (3) genius.lockmanager & (4) genius.interfacemanager (x90!) which we need to plug as part of this JIRA as well.

 Comments   
Comment by Stephen Kitt [ 30/Nov/17 ]

https://git.opendaylight.org/gerrit/65662 will fix the Genius issues.

Comment by Michael Vorburger [ 30/Nov/17 ]

https://git.opendaylight.org/gerrit/#/c/66096/ takes care of (4) in SouthboundUtils.removeTerminationEndPoint(); that one was due to a completely UNUSED newWriteOnlyTransaction()!

skitt and I in discussion just realized that his WIP https://git.opendaylight.org/gerrit/#/c/65662/ will address other leaks in genius.interfacemanager including e.g. the 35x newWriteOnlyTransaction in OvsInterfaceStateAddHelper, so he'll wrap that one up for master, and cherry-pick and work out conflicts for carbon and nitrogen.

https://git.opendaylight.org/gerrit/#/q/topic:OVSDB-435 will have all changes re. this new OOM accross ovsdb, genius and openflowplugin.

Comment by Michael Vorburger [ 30/Nov/17 ]

Attachments show a small (x4) leak OvsInterfaceStateRemoveHelper which does not appear in c/65662, because it is removed in master; perhaps that can be look into as part of the c/65662 back-ports?

Totally separate is a suspected (small) leak in ITM which was fixed in master only; that will now be in nitrogen <https://git.opendaylight.org/gerrit/#/c/66100> and carbon <https://git.opendaylight.org/gerrit/#/c/66097> as well.

Comment by Michael Vorburger [ 19/Feb/18 ]

This is considered Done and complete - until we find the next ones (which NETVIRT-878 should help prevent).

Comment by Michael Vorburger [ 25/Jul/18 ]

re-opening this because https://git.opendaylight.org/gerrit/#/c/70071/ still needs to go into Oxygen SR3... and link to NETVIRT-1384.

Comment by Faseela K [ 25/Jul/18 ]

vorburger, skitt Thanks for tracking this to completion.

Comment by Michael Vorburger [ 26/Jul/18 ]

also https://git.opendaylight.org/gerrit/#/c/70072/ - we'll CSIT both of these together, then (hopefully) merge.

Comment by Michael Vorburger [ 31/Jul/18 ]

now all merged

Generated at Wed Feb 07 19:59:55 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.