Pinged Mo on IRC and he cleaned up the volumes. Can you try again and see if it's cleared up?

do we want a new Jira for each type of infra failure? This one from a few hours ago is not the VolumeSizeExceedsAvailableQuota. I can
open a new Jira, or we can use this one as a place to log these sort of similar problems.

*21:21:25* ERROR: Failed to initialize infrastructure. Reason: Resource CREATE failed: ResourceInError: resources.vm_0_group.resources[0].resources.instance: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500"

Past few days the volume problem was hitting again. mnaser cleared it up:

https://jenkins.opendaylight.org/releng/view/netvirt-csit/job/netvirt-csit-1node-openstack-ocata-upstream-stateful-carbon/544/console

Requested 40G, quota is 8192G and 8160G has been consumed

two more cases of this in one job I was looking at:

https://jenkins.opendaylight.org/releng/user/jluhrsen/my-views/view/netvirt%20csit/job/netvirt-csit-1node-openstack-queens-upstream-stateful-snat-conntrack-oxygen/247/console
https://jenkins.opendaylight.org/releng/user/jluhrsen/my-views/view/netvirt%20csit/job/netvirt-csit-1node-openstack-queens-upstream-stateful-snat-conntrack-oxygen/248/console

These 2 Jiras are about the same issue.

which is the other jira?

The one that was cross linked from LF Jira. I guess when you link it just leaves your comment but doesn't mention that it setup the link. Look under the "Issue Links" of this Jira.

Hitting this again right now:
https://jenkins.opendaylight.org/releng/view/bgpcep/job/bgpcep-csit-verify-1node-userfeatures/607/console

 Requested 40G, quota is 10240G and 10240G has been consumed.

Seems like it's a bit early again since we were hitting it yesterday, especially considering it's 10TB now.

Are all of these just failed deletes? I had this happen today https://jenkins.opendaylight.org/sandbox/job/tomas-bgpcep-csit-1node-userfeatures-2-all-fluorine/6/console with failed delete. I presume these stay in storage and eventually it always fills up?

I cleared it up. It seems the situation is at least improved. The Volumes are no longer stuck but are "Available" instead. Which seems to indicate that they are not properly getting cleaned up when the stack is deleted.

got this in the sandbox just now:

21:00:38 ERROR: Failed to initialize infrastructure. Reason: Resource CREATE failed: OverLimit: resources.vm_0_group.resources[0].resources.volume: VolumeSizeExceedsAvailableQuota: Requested volume or snapshot exceeds allowed gigabytes quota. Requested 40G, quota is 10240G and 10230G has been consumed. (HTTP 413) (Request-ID: req-62167042-f53f-406c-908f-c3d8001ef523)

and again :

 
21:25:53 2: Waiting for 15 minutes to create sandbox-jamo-netvirt-csit-1node-openstack-pike-sfc-oxygen-3.21:26:56 1: CREATE_FAILED*21:26:59* ERROR: Failed to initialize infrastructure. Reason: Resource CREATE failed: OverLimit: resources.vm_1_group.resources[0].resources.volume: VolumeSizeExceedsAvailableQuota: Requested volume or snapshot exceeds allowed gigabytes quota. Requested 40G, quota is 10240G and 10230G has been consumed. (HTTP 413) (Request-ID: req-7ba1f95c-301b-4fd8-a6a5-7679fa6244fc)

We haven't seen volume issues for awhile so considering this issue resolved. Please open new Jiras if other issues persist.