https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/YANGTOOLS-745 yangtools <p>BUG-7128 adds basic lifecycle mapping for yang-maven-plugin, which does not include execution during increment builds.</p> <p>Based on <a href="https://wiki.eclipse.org/M2E_compatible_maven_plugins#BuildContext" class="external-link" target="_blank" rel="nofollow noopener">https://wiki.eclipse.org/M2E_compatible_maven_plugins#BuildContext</a>, we cannot enable execution during incremental builds because our generator plugins do not correctly account for generator plugins interactions.</p> <p>This issue tracks the final enablement of incremental builds, which can happen only after the dependencies have been taken care of.</p> <p>Operating System: All<br/> Platform: All</p> YANGTOOLS-745

Enable execution M2E in incremental builds

New Feature Highest Resolved Done Ruslan Kashapov Robert Varga Mon, 20 Feb 2017 22:33:13 +0000 Mon, 13 Mar 2023 07:06:10 +0000 Mon, 13 Mar 2023 07:06:10 +0000 11.0.0 maven plugin 0 2 <p>Due to the plugin API invokes files generation out of <ins>modules not files</ins>,<br/> the decision to include the module in scope of re-build (module being updated)<br/> should take into account the following</p> <p>a) is yang file of module itself was updated<br/> b) is any of submodules (if module contains any) was updated<br/> c) if other module which ougments current one was updated<br/> d) if expected output for this module is absent for some reason</p> <p>while (a-c) can be checked out of effective model context within <em>yang-maven-plugin</em><br/> module, the d) is responsibility of actual <tt>FileGenerator</tt> implementation,<br/> and cannot be determined unless the input-to-output mapping (on per module basis)<br/> is returned from file generator. In current implementation only full list of <br/> generated files is returned, no details on which output file generated based on <br/> which input module.</p> <p>The approaches to check the expected output before re-build (file re-generation <br/> based on updated input):</p> <p><b>1.</b> Do not check for expected outputs at plugin core level, <b>delegate</b> the responsibility<br/> on handling updated only inputs to file generator (i.e mdsal-binding-java-api-generator)</p> <p>in this case the information on input module states will be provided using <b>same API</b> <br/> via extended <tt>ContextHolder</tt>, which will provide file generator with information either <br/> input module was updated after last build. Also full scope of input will be available for <br/> file generator, because it's required (<em>mdsal-binding-java-api-generator</em> case) to build <br/> aggregate output file (model binding providers list under /META-INF/services)</p> <p>this approach also makes unnecessary output stream capture on a plugin level,<br/> because this data won't be used prior the file generator invocation. file generator <br/> implementation will be responsible for check either expected output exists and regenerate<br/> files if necessary. </p> <p>Also output files and mappings can be persisted via context object (the one providing <br/> information on modified inputs). This may help managing states on a file generator level,<br/> but the implementation is not forced to do it.</p> <p><b>2.</b> <b>Enforce</b> file generator to return input-to-output mapping via <b>API update</b>. Currently <br/> the result of file generation is a <tt>Table&lt;GeneratedFileType, GeneratedFilePath, GeneratedFile&gt;</tt>,<br/> the update may contain 4th element referencing source module (i.e. <tt>SourceIdentifier</tt>)</p> <p>in this case the output file to input relation can be mapped on a yang-maven-plugin level,<br/> as result the decision on either to include module into a scope for generator also can be made<br/> on a plugin level, not generator implementation.</p> <p>this will require the generator to be updated for both</p> <ul class="alternate" type="square"> <li>implement updated API to return input-to-output mappings</li> <li>support partial file generation (result of multiple runs for parts to be same as for <br/>   single run for full scope)</li> </ul> <p>shortly:</p> <p>1. api remain same, generator to decide the scope based on provided information<br/>   (which module is modified) <br/> 2. api is modified, plugin logic limits scope, generator processes what is given</p> <p>So the idea here is that we have two-stage processing and there are complex bits which we have little control over.</p> <p>The first stage is assembling an EffectiveModelContext, the dependencies of which are:</p> <ul> <li>local files</li> <li>project dependencies and files therein</li> <li>the set of codegen plugins and their preference of module resolution (currently unused)</li> </ul> <p>If we know EffectiveModelContext has not changed, we know nothing in codegen could change and we can immediately terminate execution. That is what <a href="https://jira.opendaylight.org/browse/YANGTOOLS-1165" title="Capture yang-maven input streams" class="issue-link" data-issue-key="YANGTOOLS-1165"><del>YANGTOOLS-1165</del></a> is about.</p> <p>The second stage is the codegen phase, where we run each plugin, each of which results in a set of files being generated, dependencies of which are:</p> <ul> <li>EffectiveModelContext</li> <li>Each codegen plugin's configuration (not really used, IIRC)</li> </ul> <p>This results in a set of files being generated by each codegen plugin – which is expressed in plugin-generator-api. If we have a new EffectiveModelContext or new codegen plugin configuration, we must let the plugin run.</p> <p>Codegen plugins must never have a dependency on anything maveney – we should be able to create a Gradle plugin (for example), which does the same thing as the Maven plugin and uses the same codegen plugins. This is why we have removed yang-maven-plugin-spi in <a href="https://jira.opendaylight.org/browse/YANGTOOLS-747" title="Remove yang-maven-plugin-spi APIs" class="issue-link" data-issue-key="YANGTOOLS-747"><del>YANGTOOLS-747</del></a>.</p> <p>Now, we cannot know what exactly the logic in codegen plugin does – and we should not try to express that in an API boundary. It is also important that the codegen plugin does not have (official) access to workspace state, as that improves repeatability.</p> <p>Once a codegen plugin has done its thing, though, we need to deal with the task of updating target/generated-sources files with the new view. This is where <a href="https://jira.opendaylight.org/browse/YANGTOOLS-1166" title="Capture yang-maven generated files" class="issue-link" data-issue-key="YANGTOOLS-1166"><del>YANGTOOLS-1166</del></a> comes in. One thing is that we need to delete files that are on the filesystem but were not emitted by the codegen plugin, which is easy. The other thing is writing the files which were emitted.</p> <p>This is tricky, because all other maven plugins look at file timestamps to determine whether they were updated. Eclipse integration is sensitive to when a file was updated via BuildContext. So when we write out files, we need to look at their current content and compare it (quickly!) with the newly-generated content – and if it has not changed, not touch the file at all. BuildContext.hasDelta() and BuildContext.isUptodate() – they are not really that nice, IIRC.</p> <p>With all of this in place, if I have a project open with Eclipse and make a whitespace change in src/main/yang/foo.yang, nothing should change on the filesystem. If I make any significant change, only the files affected by that change (at codegen plugin's discretion) should change. If I delete foo.yang, all the files generated for it should disappear.</p> <p>Finally, there is one more step, which is to store/retrieve state of <a href="https://jira.opendaylight.org/browse/YANGTOOLS-1165" title="Capture yang-maven input streams" class="issue-link" data-issue-key="YANGTOOLS-1165"><del>YANGTOOLS-1165</del></a> and <a href="https://jira.opendaylight.org/browse/YANGTOOLS-1166" title="Capture yang-maven generated files" class="issue-link" data-issue-key="YANGTOOLS-1166"><del>YANGTOOLS-1166</del></a> via BuildContext's setValue()/getValue() methods – that allows us to see "hey, what were the deps/output in the last incremental run?"</p> <p>Once all this works, we can update  <font color="#000000">src/main/resources/META-INF/m2e/lifecycle-mapping-metadata.xml</font> to say  &lt;runOnIncremental&gt;true&lt;/runOnIncremental&gt; – because we will not cause compilation spikes or anything.</p> <p>With that, this issue will be done and dusted. The command-line integration and/or IntelliJ integration (which works with pure maven, AFAIR) will then something <a href="https://jira.opendaylight.org/browse/YANGTOOLS-60" title="Milestone: enable takari lifecycle or similar command-line incremental builds" class="issue-link" data-issue-key="YANGTOOLS-60">YANGTOOLS-60</a> will address.</p> <p>One wrinkle we need to deal is that there can be actually multiple plugin executions configured. We should handle these seamlessly by acquiring a <a href="https://stackoverflow.com/a/62705361" class="external-link" target="_blank" rel="nofollow noopener">MojoExecution from maven</a> and propagating as appropriate.</p> Blocks YANGTOOLS-1491 YANGTOOLS-60 YANGTOOLS-747 YANGTOOLS-1490 YANGTOOLS-1166 YANGTOOLS-1165 Development Epic Link YANGTOOLS-746 External issue ID 7825 External issue URL Issue Type Rank 0|i0294f: