https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/RELENG-83 releng <p><a href="https://jenkins.opendaylight.org/releng/job/netvirt-csit-1node-openstack-queens-upstream-stateful-fluorine/140/console" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/job/netvirt-csit-1node-openstack-queens-upstream-stateful-fluorine/140/console</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>01:31:38 1: Waiting for 15 minutes to create releng-netvirt-csit-1node-openstack-queens-upstream-stateful-fluorine-140.01:32:42 1: CREATE_FAILED*01:32:45* ERROR: Failed to initialize infrastructure. Reason: Resource CREATE failed: OverLimit: resources.vm_0_group.resources[0].resources.volume: VolumeSizeExceedsAvailableQuota: Requested volume or snapshot exceeds allowed gigabytes quota. Requested 40G, quota is 8192G and 8160G has been consumed. (HTTP 413) (Request-ID: req-6dfd4fc5-fd7f-4974-a289-074fb3ec2ace)</pre> </div></div> <p>  </p> <p> </p> RELENG-83

releng infra is failing: Resource CREATE failed: OverLimit: resources.vm_0_group.resources[0].resources.volume: VolumeSizeExceedsAvailableQuota:

Bug Medium Closed Done Thanh Ha (zxiiro) Sam Hague Mon, 26 Mar 2018 15:29:01 +0000 Fri, 27 Apr 2018 15:55:52 +0000 Fri, 27 Apr 2018 15:55:52 +0000 Jenkins Job Builder 0 4 <p>Pinged Mo on IRC and he cleaned up the volumes. Can you try again and see if it's cleared up?</p> <p>do we want a new Jira for each type of infra failure? This <a href="https://jenkins.opendaylight.org/releng/user/jluhrsen/my-views/view/netvirt%20csit/job/netvirt-csit-1node-openstack-queens-upstream-stateful-oxygen/383/console" class="external-link" target="_blank" rel="nofollow noopener">one</a> from a few hours ago is not the VolumeSizeExceedsAvailableQuota. I can<br/> open a new Jira, or we can use this one as a place to log these sort of similar problems.</p> <p> </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">*21:21:25* ERROR: Failed to initialize infrastructure. Reason: Resource CREATE failed: ResourceInError: resources.vm_0_group.resources[0].resources.instance: Went to status ERROR due to <span class="code-quote">"Message: No valid host was found. There are not enough hosts available., Code: 500"</span></pre> </div></div> <p>Past few days the volume problem was hitting again. mnaser cleared it up:</p> <p><a href="https://jenkins.opendaylight.org/releng/view/netvirt-csit/job/netvirt-csit-1node-openstack-ocata-upstream-stateful-carbon/544/console" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/netvirt-csit/job/netvirt-csit-1node-openstack-ocata-upstream-stateful-carbon/544/console</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Requested 40G, quota is 8192G and 8160G has been consumed</pre> </div></div> <p>two more cases of this in one job I was looking at:</p> <p><a href="https://jenkins.opendaylight.org/releng/user/jluhrsen/my-views/view/netvirt%20csit/job/netvirt-csit-1node-openstack-queens-upstream-stateful-snat-conntrack-oxygen/247/console" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/user/jluhrsen/my-views/view/netvirt%20csit/job/netvirt-csit-1node-openstack-queens-upstream-stateful-snat-conntrack-oxygen/247/console</a><br/> <a href="https://jenkins.opendaylight.org/releng/user/jluhrsen/my-views/view/netvirt%20csit/job/netvirt-csit-1node-openstack-queens-upstream-stateful-snat-conntrack-oxygen/248/console" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/user/jluhrsen/my-views/view/netvirt%20csit/job/netvirt-csit-1node-openstack-queens-upstream-stateful-snat-conntrack-oxygen/248/console</a></p> <p>These 2 Jiras are about the same issue.</p> <p>which is the other jira?</p> <p>The one that was cross linked from LF Jira. I guess when you link it just leaves your comment but doesn't mention that it setup the link. Look under the "Issue Links" of this Jira.</p> <p>Hitting this again right now:<br/> <a href="https://jenkins.opendaylight.org/releng/view/bgpcep/job/bgpcep-csit-verify-1node-userfeatures/607/console" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/bgpcep/job/bgpcep-csit-verify-1node-userfeatures/607/console</a></p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> Requested 40G, quota is 10240G and 10240G has been consumed. </pre> </div></div> <p>Seems like it's a bit early again since we were hitting it yesterday, especially considering it's 10TB now.</p> <p>Are all of these just failed deletes? I had this happen today <a href="https://jenkins.opendaylight.org/sandbox/job/tomas-bgpcep-csit-1node-userfeatures-2-all-fluorine/6/console" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/sandbox/job/tomas-bgpcep-csit-1node-userfeatures-2-all-fluorine/6/console</a> with failed delete. I presume these stay in storage and eventually it always fills up?</p> <p>I cleared it up. It seems the situation is at least improved. The Volumes are no longer stuck but are "Available" instead. Which seems to indicate that they are not properly getting cleaned up when the stack is deleted.</p> <p>got this in the <a href="https://jenkins.opendaylight.org/sandbox/job/jamo-netvirt-csit-1node-openstack-pike-sfc-oxygen/2/console" class="external-link" target="_blank" rel="nofollow noopener">sandbox </a> just now:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> 21:00:38 ERROR: Failed to initialize infrastructure. Reason: Resource CREATE failed: OverLimit: resources.vm_0_group.resources[0].resources.volume: VolumeSizeExceedsAvailableQuota: Requested volume or snapshot exceeds allowed gigabytes quota. Requested 40G, quota is 10240G and 10230G has been consumed. (HTTP 413) (Request-ID: req-62167042-f53f-406c-908f-c3d8001ef523)</pre> </div></div> <p>and <a href="https://jenkins.opendaylight.org/sandbox/job/jamo-netvirt-csit-1node-openstack-pike-sfc-oxygen/3/console" class="external-link" target="_blank" rel="nofollow noopener">again </a>:</p> <p> <br/> <b>21:25:53</b> 2: Waiting for 15 minutes to create sandbox-jamo-netvirt-csit-1node-openstack-pike-sfc-oxygen-3.<b>21:26:56</b> 1: CREATE_FAILED*21:26:59* ERROR: Failed to initialize infrastructure. Reason: Resource CREATE failed: OverLimit: resources.vm_1_group.resources<span class="error">&#91;0&#93;</span>.resources.volume: VolumeSizeExceedsAvailableQuota: Requested volume or snapshot exceeds allowed gigabytes quota. Requested 40G, quota is 10240G and 10230G has been consumed. (HTTP 413) (Request-ID: req-7ba1f95c-301b-4fd8-a6a5-7679fa6244fc)</p> <p>We haven't seen volume issues for awhile so considering this issue resolved. Please open new Jiras if other issues persist.</p> Development Rank 0|i03cbz: