Operating System: All
Platform: All

I think the root cause for this is in SubstatementContext.copy

Statements and GroupingUtils.needToCopyByUses(). The check is a weird and undocumented one and does not take effect in situations like:

grouping foo {
  leaf foo {
    description "foo";
  }
}

Note leaf is copied, but since description's getParentContext() points to a leaf, the second part check fails and the description ends up being copied.

If this is true, the fix for this issue will also mean we will cut down temporary object use, too.

A simple unit test for this can be created. With the following models:

module foo {
  grouping foo {
    leaf foo {
      type string;
      units foo;
    }
  }

  grouping export {
    uses grouping foo;
  }
}

module bar {
  import foo;

  container bar-export {
    uses foo:export;
  }

  container bar-foo {
    uses foo:foo;
  }
}

module baz {
  import foo;

  container baz-export {
    uses foo:export;
  }

  container baz-foo {
    uses foo:foo;
  }
}

All units visible from effective model should point to the same UnitsEffectiveStatementImpls bject.

We have eliminated all obvious duplicates during assembly, with the exception of duplication of StatementContextBase, which is performed when statements are inlined via uses statement.

This is caused by the fact we copy all declared/effective statements eagerly, no matter if they will be touched in future – just to update their parent, path and copy history.

We need to create StatementContextBase equivalent proxy, which will perform the copy lazily as needed and add the changed CopyHistory and similar details to createEffective(), so it ends up reusing statement definitions if they are not touched by an augment, refine or similar.

YANGTOOLS-779 shows that this particular issue can be hit out in the field. ~28MB of models, which are presumably grouping/uses-heavy can cause us to run out 4GB of heap.

A memory snapshot taken due to an OOM during FULL_DECLARATION parsing phase shows 1.5GB memory usage, with 6.2M SubstatementContext instances retaining ~1.1GB.

While this memory usage is temporary (BuildGlobalContext shows 800MB retained at this point), we obviously need to address this to keep our resource usage within reasonable bounds.

I am not sure we can deliver this by Carbon SR1.

Solving this issue requires re-wiring 'uses', 'augment', 'refine' and 'deviate' mechanics, which deals with namespace management and child nodes. Namespace management there is tricky and we essentially want a copy-on-write StatementContextBase.

There are three sides to this problem:

1) the side we are investigating, which is reuse and lazily-populated StatementContextBase objects (which will need to be refactored). This involves dealing with CopyHistory and instantiation namespaces. Since this is traversed in a tree-like manner, this is a relatively simple problem, as all it boils down to is using forwarding objects in UsesStatementImpl's resolution phase.

2) the side of augment/refine/deviate making modifications to objects, which really expands forwarding objects with lazy instantiation to cope with modifications.

3) the side of actually triggering augment/deviate/uses phase actions, which is driven by reactor and SchemaNodeIdentifierBuildNamespace, which is populated by StmtContext.Mutable object coming and going.

It is the combination of 1) and 3) which makes this hard, as there are competing requirements. If 'uses' stops instantiating StatementContextBases, it stops feeding the namespace, which means augment/deviate/uses triggers will no longer trigger when they should.

Turning this into a milestone, as this will require multiple steps, which we do not want to do in one go simply because of the complexities involved.

Part 3) would be solved with YANGTOOLS-859, that issue needs to have a phase action trigger when an unsupported node is added – which can be also used to move a registration downwards until it hits the appropriate proxy, eventually either fully resolving the node, or knowing further resolution is impossible.

https://git.opendaylight.org/gerrit/73972 solves the triggering problem. All triggers now follow ChildSchemaNodeNamespace, hence lazy StatementContextBase equivalent class can be plugged in: it can implement namespaces and it can also act as a statement – hence 3) will not be a problem.

Part 2) is being solved in YANGTOOLS-724, and that is the next step we need to take.

So, I've run unit test with next models:

module bar {
  namespace bar-ns;
  prefix bar;  
  
  import foo { prefix foo;}  
  
  container bar-export {
    uses foo:export;
  }  
  container bar-foo {
    uses foo:foo;
  }
}

module baz {
  namespace baz-ns;
  prefix baz;  
  
  import foo { prefix foo;}  
  
  container baz-export {
    uses foo:export;
  }  
  
  container baz-foo {
    uses foo:foo;
  }
}

module foo {
  namespace foo-ns;
  prefix foo;  
  
  grouping foo {
    leaf foo {
      type string;
      units foo;
    }
  }  
  grouping export {
    uses foo;
  }
} 

And memory usage less than 50 Mb.

rovarga any ideas how to reproduce it?

Well, you need to analyze a heap dump to see if there are duplicate objects – using YourKit, for example.

Profile looks normal. There aren't thousands of duplicates.

heap.bin here is heap memory dump

As you can see there are 6 identical objects, expanded from a single grouping. This may not look like much, but it certainly adds up when you work with something like https://github.com/Juniper/yang/tree/master/19.2/19.2R1/junos/conf .

To add some colour to the picture: a SchemaContext can easily weigh 375MiB, as noted (as a matter of course) in YANGTOOLS-1004 – and these things are per device-type, so they directly affect scalability to no small extent.

There are three remaining steps in this task:

1. clarify StatementSupport.applyCopyPolicy() behavior
2. audit all StatementSupport classes to check whether they are using correct CopyPolicy
3. address the FIXME in CONTEXT_INDEPENDENT case of StatementContextBase.copyAsChildOf(), so that completely independent statements are not copied. Note the second FIXME should be taken care of by YANGTOOLS-784.

So where are here:

the highlighted statements should really be squashed, as at a number of them are duplicates, as confirmed by examining declared instance references:

This issue should take care of at least these.