[INTPAK-136] Fix intermittent failure in puppet REST API test Created: 31/Jan/18 Updated: 03/Jul/18 |
|
| Status: | Open |
| Project: | integration-packaging |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Low |
| Reporter: | Daniel Farrell | Assignee: | Daniel Farrell |
| Resolution: | Unresolved | Votes: | 0 |
| Labels: | None | ||
| Σ Remaining Estimate: | Not Specified | Remaining Estimate: | Not Specified |
| Σ Time Spent: | Not Specified | Time Spent: | Not Specified |
| Σ Original Estimate: | Not Specified | Original Estimate: | Not Specified |
| Sub-Tasks: |
|
| Description |
|
Something like 50% of test runs are failing for the puppet-opendaylight check that curls the NB REST API. Figure out what's going wrong and make the test robust. |
| Comments |
| Comment by Daniel Farrell [ 06/Mar/18 ] |
|
This is still happening. Looked into the logs in some detail, all I see is that the ones that work take 25-35 seconds and the ones that fail timeout after 300 seconds. The timeouts happen with curl exit code 7. Still seems to be about 50% failure rate. It always passes the first test run (installing Oxygen from CBS) but fails on the second run (installing RPM from Nexus). https://jenkins.opendaylight.org/releng/job/puppet-test-full-master/ |
| Comment by Daniel Farrell [ 06/Mar/18 ] |
|
Have you seen anything like this trozet? |
| Comment by Tim Rozet [ 06/Mar/18 ] |
|
No I haven't. I've been using the Nexus RPMs for Oxygen exclusively. So when it fails the second run, is this the same host, and is this right? 1. CBS rpm is installed 2. Some tests are run 3. CBS rpm uninstalled 4. Nexus RPM installed 5. Curl test fails
|
| Comment by Daniel Farrell [ 09/Mar/18 ] |
|
No, it's happening in a fresh container. So it's just a normal install, config, test. The job has passed the last 4 runs, but streaks like that are actually to be expected if it's really random. I'll trigger a few more manually and see how it looks. |
| Comment by Daniel Farrell [ 04/Jun/18 ] |
|
This happened in both of the last two daily puppet-test-full jobs. The full job history was blue before those. https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/puppet-test-full-oxygen/73 |
| Comment by Daniel Farrell [ 04/Jun/18 ] |
|
Both of these failures were on stable/oxygen, master is still blue. |
| Comment by Daniel Farrell [ 29/Jun/18 ] |
|
Lowering prio because it hasn't happened recently. |
| Comment by Daniel Farrell [ 03/Jul/18 ] |
|
Looking in detail, it hasn't happened recently because I disabled the test in https://git.opendaylight.org/gerrit/#/c/71063/. Will open up a subtask to track re-enabling that test, getting it stable. |