I think the best way to handle this case to compare the list of int/dist projects with the list of autorelease projects and flag if they don't match.
The tricky part here will be to handle projects that have subprojects like honeycomb/vbd and ofextensions/circuitsw kind of projects. So special care and attention should be paid to this kind of case.
> While we get some script to do the magic, is it possible to fail the build if we get these
> dependencies WARNING? if this is difficult to achieve then we just wait until someone has the
> cycles for writing the script.
I was not able to find any flag which can magically do this with mvn, also confirmed this with zxiiro. I work on a script to get this sorted out.
Something I do not understand is if a project (e.g. NIC) tries to add itself to int/dist but it is not in autorelease, the distribution project autorelease-validate fails like here:
However the opposite case, when a project is removed from autorelease but it is still in int/dist, autorelease project autorelease-validate does not fail.
If a project adds itself to int/dist validate-autorelease needs to know that it's being added so that it can include them in the validate-autorelease build.
In the other case if it's removed from autorelease the validate-autorelease job should still fail it's probably a timing thing that's allowing it to pass like maybe validate-autorelease ran before the project was actually removed from autorelease.
Edit: Validate-autorelease could be improved to detect that it is running against a project that is not in autorelease currently and adds them (only for the build, does not need to do it for real). This way validate-autorelease will always build the project that's being validated regardless of it's really in autorelease or not. This will make it easier to test projects as if they are in autorelease.
To add ontop of this if we have a job that compares distribution with autorelease we can ensure that we are always in sync in an automated way.
ecelgp Please confirm if this (integration/distribution/features/repos/index/pom.xml) is the best file to read to get a list of all the projects in int/dist?
Also I see that is AR has a few extra modules which are SR projects, can we exclude or skip them as a part of the validation?
integration/distribution
mdsal
odlparent
yangtools
The validation should be simple: when removing a project from AR, make sure it does not have an active template in integration/distribution/features/repos/index/pom.xml
For your comment: odlparent and yangtools should not be in AR from Oxygen onwards as they are not part of the SR. mdsal project can never be removed from AR unless we want to get rid of all projects as this is now at the root of the SR. int/dist is the distribution itself so no point to remove it unless we want to remove all projects again.
Check project removal in validate autorelease
https://git.opendaylight.org/gerrit/69105
ecelgp When I run the validation with stable/oxygen or master branch.
I get the following differences in projects:
---> validate-projects.sh ERROR: List of projects mismatch in releng/autorelease and integration/distribution: mdsal nic unimgr ocpplugin faas eman releng/autorelease: mdsal integration/distribution: nic unimgr ocpplugin faas eman
I think `mdsal` is not in the list for `integration/distribution` or required directly? Do we need to remove `nic unimgr ocpplugin faas eman` projects from int/dist aswell?
mdsal does not have a feature by itself and that is why you do not see it in int/dist. We should allow this mismatch.
The other projects you mention are disabled in int/dist index pom.xml: https://git.opendaylight.org/gerrit/gitweb?p=integration/distribution.git;a=blob;f=features/repos/index/pom.xml;h=6982eff18abf0f2abe6600b807faa9e1e482884f;hb=refs/heads/stable/oxygen
So there must be some issue in the script.
ecelgp Any specific reason they have been disabled? Can we remove the projects from the int/dist index pom.xml to get them on par with AR, instead of them being disabled?
ecelgp Another thing I noticed if that `integration/distribution/features/repos/index/pom.xml` pom file structure is completely different with every release and more consistent from stable/oxygen onwards. For instance, the activeByDefault tag is not available in the previous release. Maybe we should only enable this script for releases which supports them in the pom file and going forward?
2 answers:
- The maven templates in int/dist were introduced in Oxygen which means it is enough if script works from Oxygen onwards.
- The way I see the templates will work going forward is: all projects part of the new Managed snapshot integrated group (what we call today SR) will have a template in int/dist. If a project miss-behave like break others or we have an upstream bump that requires removing projects from int/dist, we will do all this by just disabling/enabling templates. So this means 2 things:
1) In general AR script has to be template enable/disable aware.
2) While the above should work, we can also go ahead and remove from int/dist those projects that are out of Oxygen: nic unimgr ocpplugin faas eman.
ecelgp thanks for the response. I am been trying with xmlstarlet if its possible to work around with the have that activation tag checked and set to 'true', but hitting some issues hopefully should get resolved.
zxiiro Would you know why the first and last element is concatenated in the xmlstarlet output? The pom file looks good to me. Please note, this is on stable/oxygen branch.
$ xmlstarlet sel -N x=http://maven.apache.org/POM/4.0.0 -t -m '/x:project/x:profiles/x:profile/x:activation[x:activeByDefault="true"]' --if /x:project/x:dependencies/x:depende/x:groupId -v /x:project/x:dependencies/x:dependency/x:groupId --elif /x:project/x:profiles/x:profile/x:id -v /x:project/x:profiles/x:profile/x:id --else -o '' integration/distribution/features/repos/index/pom.xml | sort -u aaa alto bgpcep bier coe controller daexim distribution dlux dluxapps eman faas genius groupbasedpolicy honeycombvbd infrautils jsonrpc l2switch lispflowmapping nemo netconf netvirt neutron nic ocpplugin odlparent ofconfig openflowplugin ovsdb p4plugin packetcable sfc snmp snmp4sdn sxp tsdr unimgr usc vtn vtnaaa
zxiiro The issue with `xmlstarlet` is now resolved.
ecelgp This is tested here on sandbox.
https://jenkins.opendaylight.org/sandbox/job/autorelease-validate-autorelease-oxygen/4/consoleFull
Please review/merge:
https://git.opendaylight.org/gerrit/69105
ecelgp I have closed this issue as resolved. Please feel free to reopen the issue if something related to this task is still pending.
Thanks, this works as far as I can tell 