mvn_evaluate.txt
[RELENG-43] Improve ar-notify script to find groupId without <name> field Created: 06/Mar/17 Updated: 11/Apr/17 Resolved: 11/Apr/17 |
|
| Status: | Resolved |
| Project: | releng |
| Component/s: | Autorelease |
| Affects Version/s: | unspecified |
| Fix Version/s: | None |
| Type: | Improvement | ||
| Reporter: | Thanh Ha (zxiiro) | Assignee: | Anil Belur |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
Operating System: All |
||
| Attachments: |
mvn_evaluate.txt
|
| Description |
|
In light of discussions in [0], Robert suggested exploring the possibility of improving the script to detect failure from parsing more of the logs. We can infer some additional information from the build by parsing the maven ERROR lines. — Maven prints the command line args to resume the build: [ERROR] After correcting the problems, you can resume the build with the command hence you can use: to get: [INFO] — maven-help-plugin:2.2:evaluate (default-cli) @ config-manager — org.opendaylight.controller |
| Comments |
| Comment by Thanh Ha (zxiiro) [ 06/Mar/17 ] |
|
Forgot to link [0] |
| Comment by Anil Belur [ 10/Mar/17 ] |
|
With the recent failure on autorelease from [1.], we have features-config which under controller project: ./controller/features/config/features-config As recommended, I tried `mvn evaluate` along with artifactid obtained from the console logs, from the autorelease root directory. $ cd /w/workspace/autorelease-release-carbon However this does not seem to return the `org.opendaylight.controller` from the output file attached. What am I missing ? [1.] https://jenkins.opendaylight.org/releng/job/autorelease-release-carbon/197/ |
| Comment by Anil Belur [ 10/Mar/17 ] |
|
Attachment mvn_evaluate.txt has been added with description: mvn_evaluate.txt |
| Comment by Thanh Ha (zxiiro) [ 10/Mar/17 ] |
|
(In reply to Anil Belur from comment #2) The command should be `mvn help:evaluate -Dexpression=project.groupId -rf :config-manager`. Evaluate is a goal inside of the Maven Help plugin: https://maven.apache.org/plugins/maven-help-plugin/evaluate-mojo.html |
| Comment by Anil Belur [ 10/Mar/17 ] |
|
Thanh, yes I have used the same command: $ cd /w/workspace/autorelease-release-carbon |
| Comment by Thanh Ha (zxiiro) [ 11/Mar/17 ] |
|
(In reply to Anil Belur from comment #2) This is a link to the releng server. Did you mean to paste a link to the sandbox logs? |
| Comment by Anil Belur [ 12/Mar/17 ] |
|
(In reply to Thanh Ha from comment #5) The logs I wanted to share earlier on sandbox jobs have been purged over the weekend, here are the updated logs in [1.]. There errors seen in the attached log file in [2.], are these due to incorrect ~/.m2/settings.xml file while running `mvn help:evaluate ...` ? I have reused the same autorelease-settings to maven-target. Downloaded: https://repo.maven.apache.org/maven2/biz/aQute/bnd/biz.aQute.bndlib/3.2.0/biz.aQute.bndlib-3.2.0.jar (2319 KB at 3079.4 KB/sec) [1.] https://logs.opendaylight.org/sandbox/jenkins091/autorelease-release-carbon/2/ |
| Comment by Anil Belur [ 14/Mar/17 ] |
|
Hello: I tried your suggestion on improvements for the AR notify script, with `mvn help:evaluate -Dexpression="project.groupId" -rf :bundles-test | grep -v Download | grep -e '[[]'` on the autorelease root folder. The command seems to works perfectly, but does not return the expected parent of the artifactid which is failing. Instead returns the `org.opendaylight.releng.autorelease` which may not be very useful for determining the $PROJECT mailing list. We need run the `mvn help:evaluate ...` within the folder which contains the `pom.xml` for `<module>bundles-test</module>` or artifactid. One possible work around is to manually find the project directory manually with the pom.xml and run `mvn help:evaluate ...`. However, a caveat is not finding an artifactid with inside <module> tag for instance with features-tsdr, can again cause issue in [1.]. Regards, [1. ] https://logs.opendaylight.org/sandbox/jenkins091/autorelease-release-carbon/9/console.log.gz $ mvn clean verify $ mvn help:evaluate -Dexpression=project.groupId -rf :bundles-test ~/autorelease (master)*$ mvn help:evaluate -Dexpression="project.groupId" -rf :bundles-test | grep -v Download | grep -e '[[]' ~/autorelease (master)*$ cd odlparent/ $ find . -name pom.xml | xargs grep '<module>bundles-test</module>' | awk -F: ' { print $1 }' |
| Comment by Thanh Ha (zxiiro) [ 14/Mar/17 ] |
|
So clearly -rf doesn't do what we want or thought it did. Looks like help:evaluate evaluates only the pom file that it is passed (default: pom.xml). The artifactId can also be used to find the directory as a pom.xml files artifactId typically matches the directory name so we can do something like `find . -type d -name ${artifactId}` to find the relative path containing the directory and pass that path to mvn's -f parameter. What if we did something like this (using bundles-test as example): mvn help:evaluate -Dexpression="project.groupId" -f $(find . -type d -name bundles-test) | grep -v Download | grep -e '[[]' For me running it from autorelease root results in org.opendaylight.odlparent groupId to be returned. |
| Comment by Anil Belur [ 15/Mar/17 ] |
|
Thanh, the suggestion works for ':bundles-test', since there is folder under the project odlparent. However for other scenarios like 'features-tsdr' would not return anything. Maybe searching for a "string" in the pom.xml to determine if its the correct pom to use with `mvn help:evaluate ...' maybe a better alternative ? // fails to find dir features-tsdr $ find . -name pom.xml | xargs grep "$ARTIFACTID" I was thinking if we should search the pom.xml file instead (below), however all $artifactId don't have <module> tag in pom.xml
|
| Comment by Anil Belur [ 19/Mar/17 ] |
|
The issue with using "mvn help:evalute ..." from the autorelease root is this requires ${artifactId}/pom.xml file as in input, which in turn requires manually finding the directory. We thought of using xpaths and xmlstarlet to search xpath's in all pom.xml and determine if ${project.artifactId} == $ARTIFACTID, then return ${project.groupId}, which can be used to derive project mailing list. However using this approach may require updating the groupId to several pom files, since it requires all the pom files to have ${project.groupId} and ${project.artifactId} xpath. Many pom files are missing the project.groupId from the xpath, which possibly requires to be updated. Below are the list of pom.xml's missing the ${project.groupId}. for p in $(find . -name pom.xml); do Expected script output should be something like: <path/to/dir/pom.xml>:<artifactId>:<groupId> ./tsdr/persistence-cassandra/pom.xml:tsdr-persistence-cassandra:NULL |
| Comment by Thanh Ha (zxiiro) [ 19/Mar/17 ] |
|
Pom files inherit the groupId and version of the parent pom if those GAV items are not specified. So if a pom returns NULL for groupId or version then check project.parent.groupId which should always return a value. |
| Comment by Anil Belur [ 19/Mar/17 ] |
|
I have explored that option too, but several of the pom files return an different parent from what is expected, for instance the below poms return project.groupId == org.opendaylight.contoller for the poms under neutron. Is this the expected output ? ./neutron/neutron-hostconfig/vpp/pom.xml:org.opendaylight.controller |
| Comment by Anil Belur [ 19/Mar/17 ] |
|
s/project.groupId/project.parent.groupId/ == org.opendaylight.contoller |
| Comment by Thanh Ha (zxiiro) [ 20/Mar/17 ] |
|
(In reply to Anil Belur from comment #12) Maybe you missed it in my previous message but I said the groupId is inherited IF not declared. Which means you need to do an if-else-if statement. I'll try being more clear.
You will find all of the cases below meet condition 1. We only need to use condition 2 on select situations. > ./neutron/neutron-hostconfig/vpp/pom.xml:org.opendaylight.controller project.groupId = org.opendaylight.neutron as expected. Condition 1 is set. > ./neutron/neutron-hostconfig/pom.xml:org.opendaylight.controller project.groupId = org.opendaylight.neutron as expected. Condition 1 is set. > ./neutron/neutron-hostconfig/ovs/pom.xml:org.opendaylight.controller project.groupId = org.opendaylight.neutron as expected. Condition 1 is set. > ./neutron/neutron-hostconfig/utils/pom.xml:org.opendaylight.controller project.groupId = org.opendaylight.neutron as expected. Condition 1 is set. > ./neutron/neutron-logger/pom.xml:org.opendaylight.controller project.groupId = org.opendaylight.neutron as expected. Condition 1 is set. > ./sxp/features/odl-sxp-robot/pom.xml:org.opendaylight.odlparent project.groupId = org.opendaylight.sxp as expected. Condition 1 is set. > ./sxp/features/odl-sxp-controller/pom.xml:org.opendaylight.odlparent project.groupId = org.opendaylight.sxp as expected. Condition 1 is set. > ./sxp/features/pom.xml:org.opendaylight.odlparent project.groupId = org.opendaylight.sxp as expected. Condition 1 is set. > ./sxp/features/odl-sxp-core/pom.xml:org.opendaylight.odlparent project.groupId = org.opendaylight.sxp as expected. Condition 1 is set. > ./sxp/features/features-sxp/pom.xml:org.opendaylight.odlparent project.groupId = org.opendaylight.sxp as expected. Condition 1 is set. Does this make sense? |
| Comment by Anil Belur [ 22/Mar/17 ] |
|
// suggestion from Thanh Ha on gerrit change #53380 So a situation that might occur is there might be more than 1 project that uses the same name for it's ARTIFACT_ID. For example many projects have an ARTIFACT_ID called "distribution-karaf". In which case we still don't know project is the cause. |
| Comment by Anil Belur [ 22/Mar/17 ] |
|
Yes, we would have artifactId dups under several projects to be handled, here is the list: applications: repeats: 3 |
| Comment by Thanh Ha (zxiiro) [ 11/Apr/17 ] |
|
I think this is resolved? |
| Comment by Anil Belur [ 11/Apr/17 ] |
|
Yes, closing as resolved. |