Operating System: All
Platform: All

Due to the plugin API invokes files generation out of modules not files,
the decision to include the module in scope of re-build (module being updated)
should take into account the following

a) is yang file of module itself was updated
b) is any of submodules (if module contains any) was updated
c) if other module which ougments current one was updated
d) if expected output for this module is absent for some reason

while (a-c) can be checked out of effective model context within yang-maven-plugin
module, the d) is responsibility of actual FileGenerator implementation,
and cannot be determined unless the input-to-output mapping (on per module basis)
is returned from file generator. In current implementation only full list of 
generated files is returned, no details on which output file generated based on 
which input module.

The approaches to check the expected output before re-build (file re-generation 
based on updated input):

1. Do not check for expected outputs at plugin core level, delegate the responsibility
on handling updated only inputs to file generator (i.e mdsal-binding-java-api-generator)

in this case the information on input module states will be provided using same API 
via extended ContextHolder, which will provide file generator with information either 
input module was updated after last build. Also full scope of input will be available for 
file generator, because it's required (mdsal-binding-java-api-generator case) to build 
aggregate output file (model binding providers list under /META-INF/services)

this approach also makes unnecessary output stream capture on a plugin level,
because this data won't be used prior the file generator invocation. file generator 
implementation will be responsible for check either expected output exists and regenerate
files if necessary. 

Also output files and mappings can be persisted via context object (the one providing 
information on modified inputs). This may help managing states on a file generator level,
but the implementation is not forced to do it.

2. Enforce file generator to return input-to-output mapping via API update. Currently 
the result of file generation is a Table<GeneratedFileType, GeneratedFilePath, GeneratedFile>,
the update may contain 4th element referencing source module (i.e. SourceIdentifier)

in this case the output file to input relation can be mapped on a yang-maven-plugin level,
as result the decision on either to include module into a scope for generator also can be made
on a plugin level, not generator implementation.

this will require the generator to be updated for both

• implement updated API to return input-to-output mappings
• support partial file generation (result of multiple runs for parts to be same as for 
    single run for full scope)

shortly:

1. api remain same, generator to decide the scope based on provided information
  (which module is modified) 
2. api is modified, plugin logic limits scope, generator processes what is given

So the idea here is that we have two-stage processing and there are complex bits which we have little control over.

The first stage is assembling an EffectiveModelContext, the dependencies of which are:

• local files
• project dependencies and files therein
• the set of codegen plugins and their preference of module resolution (currently unused)

If we know EffectiveModelContext has not changed, we know nothing in codegen could change and we can immediately terminate execution. That is what YANGTOOLS-1165 is about.

The second stage is the codegen phase, where we run each plugin, each of which results in a set of files being generated, dependencies of which are:

• EffectiveModelContext
• Each codegen plugin's configuration (not really used, IIRC)

This results in a set of files being generated by each codegen plugin – which is expressed in plugin-generator-api. If we have a new EffectiveModelContext or new codegen plugin configuration, we must let the plugin run.

Codegen plugins must never have a dependency on anything maveney – we should be able to create a Gradle plugin (for example), which does the same thing as the Maven plugin and uses the same codegen plugins. This is why we have removed yang-maven-plugin-spi in YANGTOOLS-747.

Now, we cannot know what exactly the logic in codegen plugin does – and we should not try to express that in an API boundary. It is also important that the codegen plugin does not have (official) access to workspace state, as that improves repeatability.

Once a codegen plugin has done its thing, though, we need to deal with the task of updating target/generated-sources files with the new view. This is where YANGTOOLS-1166 comes in. One thing is that we need to delete files that are on the filesystem but were not emitted by the codegen plugin, which is easy. The other thing is writing the files which were emitted.

This is tricky, because all other maven plugins look at file timestamps to determine whether they were updated. Eclipse integration is sensitive to when a file was updated via BuildContext. So when we write out files, we need to look at their current content and compare it (quickly!) with the newly-generated content – and if it has not changed, not touch the file at all. BuildContext.hasDelta() and BuildContext.isUptodate() – they are not really that nice, IIRC.

With all of this in place, if I have a project open with Eclipse and make a whitespace change in src/main/yang/foo.yang, nothing should change on the filesystem. If I make any significant change, only the files affected by that change (at codegen plugin's discretion) should change. If I delete foo.yang, all the files generated for it should disappear.

Finally, there is one more step, which is to store/retrieve state of YANGTOOLS-1165 and YANGTOOLS-1166 via BuildContext's setValue()/getValue() methods – that allows us to see "hey, what were the deps/output in the last incremental run?"

Once all this works, we can update  src/main/resources/META-INF/m2e/lifecycle-mapping-metadata.xml to say  <runOnIncremental>true</runOnIncremental> – because we will not cause compilation spikes or anything.

With that, this issue will be done and dusted. The command-line integration and/or IntelliJ integration (which works with pure maven, AFAIR) will then something YANGTOOLS-60 will address.

One wrinkle we need to deal is that there can be actually multiple plugin executions configured. We should handle these seamlessly by acquiring a MojoExecution from maven and propagating as appropriate.