[YANGTOOLS-1474] Fail to process augmentation with empty grouping Created: 04/Jan/23 Updated: 30/Jan/23 Resolved: 30/Jan/23 |
|
| Status: | Resolved |
| Project: | yangtools |
| Component/s: | parser |
| Affects Version/s: | 8.0.9, 10.0.2, 9.0.6 |
| Fix Version/s: | 8.0.10, 10.0.3, 9.0.7 |
| Type: | Bug | Priority: | Medium |
| Reporter: | Sangwook Ha | Assignee: | Robert Varga |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||
| Description |
|
Parsing of YANG models may fail with the following conditions:
For example, parsing of the following two YANG modules with no supported features - the augmentation effectively does nothing but there is a such data model actually used in the device, believe it or not: foo.yang module foo {
namespace "urn:foo";
prefix "foo";
container foo {
leaf foo-leaf {
type string;
}
}
}
bar.yang module bar {
namespace "urn:bar";
prefix "bar";
import foo {
prefix foo;
}
grouping bar-group {
}
augment "/foo:foo" {
uses bar-group;
}
}
may trigger the following exception: Caused by: org.opendaylight.yangtools.yang.parser.spi.meta.SomeModifiersUnresolvedException: Some of EFFECTIVE_MODEL modifiers for statements were not resolved. at org.opendaylight.yangtools.yang.parser.reactor@10.0.3-SNAPSHOT/org.opendaylight.yangtools.yang.parser.stmt.reactor.BuildGlobalContext.propagateException(BuildGlobalContext.java:233) at org.opendaylight.yangtools.yang.parser.reactor@10.0.3-SNAPSHOT/org.opendaylight.yangtools.yang.parser.stmt.reactor.BuildGlobalContext.transformEffective(BuildGlobalContext.java:263) at org.opendaylight.yangtools.yang.parser.reactor@10.0.3-SNAPSHOT/org.opendaylight.yangtools.yang.parser.stmt.reactor.BuildGlobalContext.buildEffective(BuildGlobalContext.java:208) at org.opendaylight.yangtools.yang.parser.reactor@10.0.3-SNAPSHOT/org.opendaylight.yangtools.yang.parser.stmt.reactor.CrossSourceStatementReactor$BuildAction.buildEffective(CrossSourceStatementReactor.java:232) at org.opendaylight.yangtools.yang.parser.rfc7950@10.0.3-SNAPSHOT/org.opendaylight.yangtools.yang.stmt.TestUtils.parseYangSource(TestUtils.java:95) at org.opendaylight.yangtools.yang.parser.rfc7950@10.0.3-SNAPSHOT/org.opendaylight.yangtools.yang.stmt.AbstractYangTest.assertEffectiveModel(AbstractYangTest.java:44) ... 72 more Caused by: com.google.common.base.VerifyException: expected a non-null reference at com.google.common@31.1-jre/com.google.common.base.Verify.verifyNotNull(Verify.java:503) at com.google.common@31.1-jre/com.google.common.base.Verify.verifyNotNull(Verify.java:479) at org.opendaylight.yangtools.yang.parser.reactor@10.0.3-SNAPSHOT/org.opendaylight.yangtools.yang.parser.stmt.reactor.AbstractResumedStatement.mutableDeclaredSubstatements(AbstractResumedStatement.java:67) at org.opendaylight.yangtools.yang.parser.spi@10.0.3-SNAPSHOT/org.opendaylight.yangtools.yang.parser.spi.meta.StmtContext$Mutable.declaredSubstatements(StmtContext.java:234) at org.opendaylight.yangtools.yang.parser.spi@10.0.3-SNAPSHOT/org.opendaylight.yangtools.yang.parser.spi.meta.StmtContextUtils.checkFeatureSupport(StmtContextUtils.java:221) at org.opendaylight.yangtools.yang.parser.reactor@10.0.3-SNAPSHOT/org.opendaylight.yangtools.yang.parser.stmt.reactor.ReactorStmtCtx.isSupportedByFeatures(ReactorStmtCtx.java:490) at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:178) at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179) at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133) at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1845) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) at org.opendaylight.yangtools.yang.parser.spi@10.0.3-SNAPSHOT/org.opendaylight.yangtools.yang.parser.spi.meta.AbstractStatementSupport.declaredSubstatements(AbstractStatementSupport.java:174) at org.opendaylight.yangtools.yang.parser.spi@10.0.3-SNAPSHOT/org.opendaylight.yangtools.yang.parser.spi.meta.AbstractStatementSupport.createEffective(AbstractStatementSupport.java:84) at org.opendaylight.yangtools.yang.parser.reactor@10.0.3-SNAPSHOT/org.opendaylight.yangtools.yang.parser.stmt.reactor.AbstractResumedStatement.createEffective(AbstractResumedStatement.java:131) at org.opendaylight.yangtools.yang.parser.reactor@10.0.3-SNAPSHOT/org.opendaylight.yangtools.yang.parser.stmt.reactor.AbstractResumedStatement.createEffective(AbstractResumedStatement.java:116) at org.opendaylight.yangtools.yang.parser.reactor@10.0.3-SNAPSHOT/org.opendaylight.yangtools.yang.parser.stmt.reactor.StatementContextBase.createEffective(StatementContextBase.java:426) at org.opendaylight.yangtools.yang.parser.reactor@10.0.3-SNAPSHOT/org.opendaylight.yangtools.yang.parser.stmt.reactor.ReactorStmtCtx.loadEffective(ReactorStmtCtx.java:375) at org.opendaylight.yangtools.yang.parser.reactor@10.0.3-SNAPSHOT/org.opendaylight.yangtools.yang.parser.stmt.reactor.ReactorStmtCtx.buildEffective(ReactorStmtCtx.java:371) at org.opendaylight.yangtools.yang.parser.reactor@10.0.3-SNAPSHOT/org.opendaylight.yangtools.yang.parser.stmt.reactor.BuildGlobalContext.transformEffective(BuildGlobalContext.java:261) ... 76 more One thing to note is that the two conditions listed above appear to be necessary but not sufficient because the error is not triggered predictably. |
| Comments |
| Comment by Sangwook Ha [ 04/Jan/23 ] |
|
Among the 4 test cases from https://git.opendaylight.org/gerrit/c/yangtools/+/103918/1, 3 test cases failed and one passed although the 4 test cases use the exactly same YANG files: 06:47:34 [ERROR] Failures: 06:47:34 [ERROR] YT1474Test.testEmptyGroupingBarFoo1:32->AbstractYangTest.assertEffectiveModel:47 Failed to assemble effective model 06:47:34 [ERROR] YT1474Test.testEmptyGroupingFooBar1:22->AbstractYangTest.assertEffectiveModel:47 Failed to assemble effective model 06:47:34 [ERROR] YT1474Test.testEmptyGroupingFooBar2:27->AbstractYangTest.assertEffectiveModel:47 Failed to assemble effective model 06:47:34 [INFO] 06:47:34 [ERROR] Tests run: 565, Failures: 3, Errors: 0, Skipped: 0 When the tests in YT1474Test are run separately, different test cases fail. For example, on MacOS, the only one test case (testEmptyGroupingBarFoo2) failed: [ERROR] Failures: [ERROR] YT1474Test.testEmptyGroupingBarFoo2:37->AbstractYangTest.assertEffectiveModel:47 Failed to assemble effective model [INFO] [ERROR] Tests run: 4, Failures: 1, Errors: 0, Skipped: 0 But a different test case (testEmptyGroupingFooBar1) failed when the tests were run in a Linux container: [ERROR] Failures: [ERROR] YT1474Test.testEmptyGroupingFooBar1:22->AbstractYangTest.assertEffectiveModel:47 Failed to assemble effective model [INFO] [ERROR] Tests run: 4, Failures: 1, Errors: 0, Skipped: 0 |
| Comment by Robert Varga [ 17/Jan/23 ] |
|
The variance depends on what the processing order is going to be. In the failure case 'foo.yang' is built first successfully and the failure occurs when 'bar.yang' is being built. The grouping's refcount should be held down by parent's (module) statement's implicit reference, but that does not seem to be the case here. For non-empty groupings we end up being pinned down by our children (and hence the sweep takes a different codepath). Recording the sweep stack trace and propagating into the VerifyException shows the following codepath: Caused by: com.google.common.base.VerifyException: Substatements no longer available in SubstatementContext{definition=StatementDefinitionContext{statement=(urn:ietf:params:xml:ns:yang:yin:1)grouping}, argument=(urn:bar)bar-group, refCount=SWEPT}
at org.opendaylight.yangtools.yang.parser.reactor/org.opendaylight.yangtools.yang.parser.stmt.reactor.AbstractResumedStatement.mutableDeclaredSubstatements(AbstractResumedStatement.java:78)
at org.opendaylight.yangtools.yang.parser.spi@10.0.3-SNAPSHOT/org.opendaylight.yangtools.yang.parser.spi.meta.StmtContext$Mutable.declaredSubstatements(StmtContext.java:234)
at org.opendaylight.yangtools.yang.parser.spi@10.0.3-SNAPSHOT/org.opendaylight.yangtools.yang.parser.spi.meta.StmtContextUtils.checkFeatureSupport(StmtContextUtils.java:221)
at org.opendaylight.yangtools.yang.parser.reactor/org.opendaylight.yangtools.yang.parser.stmt.reactor.ReactorStmtCtx.isSupportedByFeatures(ReactorStmtCtx.java:490)
at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:178)
at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)
at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133)
at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1845)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
at org.opendaylight.yangtools.yang.parser.spi@10.0.3-SNAPSHOT/org.opendaylight.yangtools.yang.parser.spi.meta.AbstractStatementSupport.declaredSubstatements(AbstractStatementSupport.java:174)
at org.opendaylight.yangtools.yang.parser.spi@10.0.3-SNAPSHOT/org.opendaylight.yangtools.yang.parser.spi.meta.AbstractStatementSupport.createEffective(AbstractStatementSupport.java:84)
at org.opendaylight.yangtools.yang.parser.reactor/org.opendaylight.yangtools.yang.parser.stmt.reactor.AbstractResumedStatement.createEffective(AbstractResumedStatement.java:142)
at org.opendaylight.yangtools.yang.parser.reactor/org.opendaylight.yangtools.yang.parser.stmt.reactor.AbstractResumedStatement.createEffective(AbstractResumedStatement.java:127)
at org.opendaylight.yangtools.yang.parser.reactor/org.opendaylight.yangtools.yang.parser.stmt.reactor.StatementContextBase.createEffective(StatementContextBase.java:426)
at org.opendaylight.yangtools.yang.parser.reactor/org.opendaylight.yangtools.yang.parser.stmt.reactor.ReactorStmtCtx.loadEffective(ReactorStmtCtx.java:375)
at org.opendaylight.yangtools.yang.parser.reactor/org.opendaylight.yangtools.yang.parser.stmt.reactor.ReactorStmtCtx.buildEffective(ReactorStmtCtx.java:371)
at org.opendaylight.yangtools.yang.parser.reactor/org.opendaylight.yangtools.yang.parser.stmt.reactor.BuildGlobalContext.transformEffective(BuildGlobalContext.java:261)
... 75 more
Caused by: java.lang.Throwable: Drop org.opendaylight.yangtools.yang.parser.stmt.reactor.SubstatementContext@7048f722 name SubstatementContext{definition=StatementDefinitionContext{statement=(urn:ietf:params:xml:ns:yang:yin:1)grouping}, argument=(urn:bar)bar-group, refCount=SWEEPING}
at org.opendaylight.yangtools.yang.parser.reactor/org.opendaylight.yangtools.yang.parser.stmt.reactor.AbstractResumedStatement.dropDeclaredSubstatements(AbstractResumedStatement.java:213)
at org.opendaylight.yangtools.yang.parser.reactor/org.opendaylight.yangtools.yang.parser.stmt.reactor.OriginalStmtCtx.sweepSubstatements(OriginalStmtCtx.java:145)
at org.opendaylight.yangtools.yang.parser.reactor/org.opendaylight.yangtools.yang.parser.stmt.reactor.ReactorStmtCtx.sweepState(ReactorStmtCtx.java:898)
at org.opendaylight.yangtools.yang.parser.reactor/org.opendaylight.yangtools.yang.parser.stmt.reactor.ReactorStmtCtx.sweepOnDecrement(ReactorStmtCtx.java:783)
at org.opendaylight.yangtools.yang.parser.reactor/org.opendaylight.yangtools.yang.parser.stmt.reactor.ReactorStmtCtx.loadEffective(ReactorStmtCtx.java:380)
at org.opendaylight.yangtools.yang.parser.reactor/org.opendaylight.yangtools.yang.parser.stmt.reactor.ReactorStmtCtx.buildEffective(ReactorStmtCtx.java:371)
at org.opendaylight.yangtools.yang.parser.rfc7950/org.opendaylight.yangtools.yang.parser.rfc7950.stmt.uses.UsesStatementSupport.createEffective(UsesStatementSupport.java:143)
at org.opendaylight.yangtools.yang.parser.rfc7950/org.opendaylight.yangtools.yang.parser.rfc7950.stmt.uses.UsesStatementSupport.createEffective(UsesStatementSupport.java:1)
at org.opendaylight.yangtools.yang.parser.spi@10.0.3-SNAPSHOT/org.opendaylight.yangtools.yang.parser.spi.meta.AbstractStatementSupport.createEffective(AbstractStatementSupport.java:85)
at org.opendaylight.yangtools.yang.parser.reactor/org.opendaylight.yangtools.yang.parser.stmt.reactor.AbstractResumedStatement.createEffective(AbstractResumedStatement.java:142)
at org.opendaylight.yangtools.yang.parser.reactor/org.opendaylight.yangtools.yang.parser.stmt.reactor.AbstractResumedStatement.createEffective(AbstractResumedStatement.java:127)
at org.opendaylight.yangtools.yang.parser.reactor/org.opendaylight.yangtools.yang.parser.stmt.reactor.StatementContextBase.createEffective(StatementContextBase.java:426)
at org.opendaylight.yangtools.yang.parser.reactor/org.opendaylight.yangtools.yang.parser.stmt.reactor.ReactorStmtCtx.loadEffective(ReactorStmtCtx.java:375)
at org.opendaylight.yangtools.yang.parser.reactor/org.opendaylight.yangtools.yang.parser.stmt.reactor.ReactorStmtCtx.buildEffective(ReactorStmtCtx.java:371)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
at org.opendaylight.yangtools.yang.parser.spi@10.0.3-SNAPSHOT/org.opendaylight.yangtools.yang.parser.spi.meta.AbstractStatementSupport.buildEffectiveSubstatements(AbstractStatementSupport.java:139)
at org.opendaylight.yangtools.yang.parser.spi@10.0.3-SNAPSHOT/org.opendaylight.yangtools.yang.parser.spi.meta.AbstractStatementSupport.createEffective(AbstractStatementSupport.java:83)
at org.opendaylight.yangtools.yang.parser.reactor/org.opendaylight.yangtools.yang.parser.stmt.reactor.AbstractResumedStatement.createEffective(AbstractResumedStatement.java:142)
at org.opendaylight.yangtools.yang.parser.reactor/org.opendaylight.yangtools.yang.parser.stmt.reactor.AbstractResumedStatement.createEffective(AbstractResumedStatement.java:127)
at org.opendaylight.yangtools.yang.parser.reactor/org.opendaylight.yangtools.yang.parser.stmt.reactor.StatementContextBase.createEffective(StatementContextBase.java:426)
at org.opendaylight.yangtools.yang.parser.reactor/org.opendaylight.yangtools.yang.parser.stmt.reactor.ReactorStmtCtx.loadEffective(ReactorStmtCtx.java:375)
at org.opendaylight.yangtools.yang.parser.reactor/org.opendaylight.yangtools.yang.parser.stmt.reactor.ReactorStmtCtx.buildEffective(ReactorStmtCtx.java:371)
at org.opendaylight.yangtools.yang.parser.reactor/org.opendaylight.yangtools.yang.parser.stmt.reactor.ReplicaStatementContext.createEffective(ReplicaStatementContext.java:54)
at org.opendaylight.yangtools.yang.parser.reactor/org.opendaylight.yangtools.yang.parser.stmt.reactor.ReactorStmtCtx.loadEffective(ReactorStmtCtx.java:375)
at org.opendaylight.yangtools.yang.parser.reactor/org.opendaylight.yangtools.yang.parser.stmt.reactor.ReactorStmtCtx.buildEffective(ReactorStmtCtx.java:371)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
at org.opendaylight.yangtools.yang.parser.spi@10.0.3-SNAPSHOT/org.opendaylight.yangtools.yang.parser.spi.meta.AbstractStatementSupport.buildEffectiveSubstatements(AbstractStatementSupport.java:139)
at org.opendaylight.yangtools.yang.parser.spi@10.0.3-SNAPSHOT/org.opendaylight.yangtools.yang.parser.spi.meta.AbstractStatementSupport.createEffective(AbstractStatementSupport.java:83)
at org.opendaylight.yangtools.yang.parser.reactor/org.opendaylight.yangtools.yang.parser.stmt.reactor.AbstractResumedStatement.createEffective(AbstractResumedStatement.java:142)
at org.opendaylight.yangtools.yang.parser.reactor/org.opendaylight.yangtools.yang.parser.stmt.reactor.AbstractResumedStatement.createEffective(AbstractResumedStatement.java:127)
at org.opendaylight.yangtools.yang.parser.reactor/org.opendaylight.yangtools.yang.parser.stmt.reactor.StatementContextBase.createEffective(StatementContextBase.java:426)
at org.opendaylight.yangtools.yang.parser.reactor/org.opendaylight.yangtools.yang.parser.stmt.reactor.ReactorStmtCtx.loadEffective(ReactorStmtCtx.java:375)
at org.opendaylight.yangtools.yang.parser.reactor/org.opendaylight.yangtools.yang.parser.stmt.reactor.ReactorStmtCtx.buildEffective(ReactorStmtCtx.java:371)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
at org.opendaylight.yangtools.yang.parser.spi@10.0.3-SNAPSHOT/org.opendaylight.yangtools.yang.parser.spi.meta.AbstractStatementSupport.buildEffectiveSubstatements(AbstractStatementSupport.java:139)
at org.opendaylight.yangtools.yang.parser.rfc7950/org.opendaylight.yangtools.yang.parser.rfc7950.stmt.module.ModuleStatementSupport.buildEffectiveSubstatements(ModuleStatementSupport.java:189)
at org.opendaylight.yangtools.yang.parser.spi@10.0.3-SNAPSHOT/org.opendaylight.yangtools.yang.parser.spi.meta.AbstractStatementSupport.createEffective(AbstractStatementSupport.java:83)
... 81 more
The "Drop " exception is recorded when the substatements are dropped. |
| Comment by Robert Varga [ 17/Jan/23 ] |
|
Okay, so the problem here is that:
now, some time later we resume building 'bar' and iterate its children ... and boom.
|