https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/RELENG-75 releng <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jluhrsen" class="user-hover" rel="jluhrsen">jluhrsen</a> mentioned that CSIT jobs are often failing to come online and failing job builds for example see the history of <span class="error">&#91;0&#93;</span>. It seems to be failing often enough.</p> <p><span class="error">&#91;0&#93;</span> <a href="https://jenkins.opendaylight.org/releng/user/jluhrsen/my-views/view/netvirt%20csit/job/netvirt-csit-1node-openstack-ocata-upstream-stateful-snat-conntrack-oxygen/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/user/jluhrsen/my-views/view/netvirt%20csit/job/netvirt-csit-1node-openstack-ocata-upstream-stateful-snat-conntrack-oxygen/</a></p> RELENG-75

Heat scripts fail to bring nodes online

Story Medium Resolved Done Thanh Ha (zxiiro) Thanh Ha (zxiiro) Thu, 4 Jan 2018 20:28:48 +0000 Fri, 12 Jan 2018 15:58:22 +0000 Fri, 12 Jan 2018 15:58:22 +0000 Jenkins Job Builder 0 3 <p>Not sure if this is the cause but Heat stack cleanup scripts currently run every 15 minutes. There's definitely a race condition that's possible here between when a stack is in progress creating while the VM is coming fully online and is passed back to the CSIT job. It's possible that the script's list of stacks contains stacks that are in this in progress state while it's coming online and deleting the stacks before it can be passed back to the CSIT job.</p> <p>We should re-evaluate this script and see if we can add some smarts into it to improve things.</p> <p><a href="https://github.com/opendaylight/releng-builder/blob/master/jjb/opendaylight-infra-cleanup-stale-stacks.sh" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/opendaylight/releng-builder/blob/master/jjb/opendaylight-infra-cleanup-stale-stacks.sh</a></p> <p>Proposed patches: </p> <ul> <li><a href="https://git.opendaylight.org/gerrit/66881" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/66881</a></li> <li><a href="https://git.opendaylight.org/gerrit/66884" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/66884</a></li> </ul> <p>Merged.</p> <p>I suspect the previous patch might not solve the problem after poking at it more deeply. I decided to additionally add <a href="https://git.opendaylight.org/gerrit/66883" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/66883</a> patch to inprove the debug output to make it more useful. Hopefully we can get more useful info the next time it happens.</p> <p>With the additional logging we got from the stack patches last night we know what the real issue is now. The error message is:</p> <p>Resource CREATE failed: ResourceInError: resources.vm_0_group.resources<span class="error">&#91;0&#93;</span>.resources.instance: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500</p> <p>Which tells me we're using too many robot systems. Taking a look at releng/builder I noticed that we now have both 1c and 2c robot nodes each allowing 25 parallel robots to run. This is not a good idea. Unfortunately with Robot systems every job should use the same robot vm so that we can properly limit the max nodes. I think next steps here are:</p> <p>1. Contact the cloud provider (Done)<br/> 2. Switch all releng/builder jobs to the 2c robot nodes<br/> 3. If necessary reduce the limit of robot vms further</p> <p>Proposed patch to move all robots to the same type <a href="https://git.opendaylight.org/gerrit/66904" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/66904</a></p> <p>Add a test to check that we only ever use 1 robot node <a href="https://git.opendaylight.org/gerrit/66905" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/66905</a> this ensures future changes does not miss this detail.</p> <p>Infra seems to be stable now so closing this off as resolved. It seems the changes we made last week made things a lot better.</p> Development Rank 0|i039b3: