[RELENG-46] Failed to create Carbon staging repo Created: 20/Apr/17  Updated: 19/Oct/17  Resolved: 07/Jun/17

Status: Resolved
Project: releng
Component/s: Autorelease
Affects Version/s: unspecified
Fix Version/s: None

Type: Bug
Reporter: Thanh Ha (zxiiro) Assignee: Unassigned
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: All
Platform: All


External issue ID: 8265

 Description   

The Carbon Autorelease build is failing to generate a staging repo. See:

https://logs.opendaylight.org/releng/jenkins092/autorelease-release-carbon/249/archives/deploy-staged-repository.log.gz

One theory is this might be related to our max file size limit of 500 MB in Nexus.



 Comments   
Comment by Thanh Ha (zxiiro) [ 20/Apr/17 ]

Submitted patch https://git.opendaylight.org/gerrit/55751 which should create a log of files that are larger than 200 MB so that we can inspect file sizes.

Comment by Thanh Ha (zxiiro) [ 20/Apr/17 ]

Colin suggests we archive the last build in Jenkins to unblock testing.

Comment by Thanh Ha (zxiiro) [ 20/Apr/17 ]

Submitted a patch to archive 1 copy of artifacts from the last successful build.

https://git.opendaylight.org/gerrit/55757

Comment by Thanh Ha (zxiiro) [ 20/Apr/17 ]

In the interest in checking the file sizes log I started a build with -DskipTests=true to see if we can have a quick copy of that log file to inspect today.

Comment by Thanh Ha (zxiiro) [ 21/Apr/17 ]

Largest file appears to be about 300MB so the issue must be something else. I'm not sure what else it could be at the moment. We need to bring up a VM to troubleshoot.

210456576 /w/workspace/autorelease-release-carbon/hide/from/pom/files/m2repo/org/opendaylight/bier/bier-karaf/0.1.0-Carbon/bier-karaf-0.1.0-Carbon.tar.gz
212033536 /w/workspace/autorelease-release-carbon/hide/from/pom/files/m2repo/org/opendaylight/bier/bier-karaf/0.1.0-Carbon/bier-karaf-0.1.0-Carbon.zip
315871232 /w/workspace/autorelease-release-carbon/hide/from/pom/files/m2repo/org/opendaylight/integration/distribution-karaf/0.6.0-Carbon/distribution-karaf-0.6.0-Carbon.tar.gz
318402560 /w/workspace/autorelease-release-carbon/hide/from/pom/files/m2repo/org/opendaylight/integration/distribution-karaf/0.6.0-Carbon/distribution-karaf-0.6.0-Carbon.zip
213438464 /w/workspace/autorelease-release-carbon/hide/from/pom/files/m2repo/org/opendaylight/vtn/distribution.vtn-manager.karaf/0.6.0-Carbon/distribution.vtn-manager.karaf-0.6.0-Carbon.tar.gz
215134208 /w/workspace/autorelease-release-carbon/hide/from/pom/files/m2repo/org/opendaylight/vtn/distribution.vtn-manager.karaf/0.6.0-Carbon/distribution.vtn-manager.karaf-0.6.0-Carbon.zip

Comment by Thanh Ha (zxiiro) [ 21/Apr/17 ]

So it turns out with -DskipTests the build actually successfully deploys.

https://logs.opendaylight.org/releng/jenkins092/autorelease-release-carbon/253/archives/deploy-staged-repository.log.gz

So the only difference between this build #253 and the build from #249 is that:

1) We ran with -DskipTests
2) We removed tsdr & cardinal from the build

Comment by Anil Belur [ 21/Apr/17 ]

I am unsure if this is related to the issue in question, but we see several fatal errors in console logs of carbon build #249 and #247 in [3.] & [4.] which is not seen in #253 (where the artifacts are uploaded onto nexus successfully with the tests skipped)

2017-04-18 23:55:16.840642: [23193]: FATAL: Creating Database Handler Failed!!
2017-04-18 23:55:16.840819: [23193]: FATAL: Creating Database Handler Failed!!
2017-04-18 23:55:16.840893: [23193]: FATAL: Creating Database Handler Failed!!
...

Not sure, if this is the cause the issue ?

[3.] https://logs.opendaylight.org/releng/jenkins092/autorelease-release-carbon/249/console.log.gz
[4.] https://logs.opendaylight.org/releng/jenkins092/autorelease-release-carbon/247/console.log.gz

Comment by Colin Dixon [ 24/Apr/17 ]

(In reply to Thanh Ha from comment #6)
> So it turns out with -DskipTests the build actually successfully deploys.
>
> https://logs.opendaylight.org/releng/jenkins092/autorelease-release-carbon/
> 253/archives/deploy-staged-repository.log.gz
>
>
> So the only difference between this build #253 and the build from #249 is
> that:
>
> 1) We ran with -DskipTests
> 2) We removed tsdr & cardinal from the build

Given this and the number of sporadic/timing issues we're seeing, e.g.,:
https://lists.opendaylight.org/pipermail/release/2017-April/010290.html

Maybe we should run autorelease-carbon with -DskipTests first, then publish artifacts, then run it normally. How hard would that be?

Comment by Anil Belur [ 25/Apr/17 ]

It would be straightforward to create a separate job "autorelease-release-carbon-notests" job (temporarily) with the `-Dskiptest` enabled instead of incorporating both into the same job. This is because the any failures in the builder section would not allow the job to continue the next builder section.

Comment by Anil Belur [ 28/Apr/17 ]

Uploading artifacts to nexus is no longer reproducible, therefore this should no longer be a blocker bug. I am presently reducing the priority to normal.

https://logs.opendaylight.org/releng/jenkins092/autorelease-release-notests-carbon/1/archives/deploy-staged-repository.log.gz

Comment by Anil Belur [ 28/Apr/17 ]

Atrifacts for the build: https://nexus.opendaylight.org/content/repositories/autorelease-1765/org/opendaylight/integration/distribution-karaf/0.6.0-Carbon/

Comment by Colin Dixon [ 02/May/17 ]

Anil, if you believe this is fixed. You can close it. We can always reopen it later.

Generated at Wed Feb 07 20:37:21 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.