[NETVIRT-897] CSIT Sporadic failures - tempest.scenario.test_server_basic_ops.TestServerBasicOps.test_server_basic_ops Created: 08/Sep/17  Updated: 06/Apr/18  Resolved: 06/Apr/18

Status: Resolved
Project: netvirt
Component/s: General
Affects Version/s: Carbon
Fix Version/s: None

Type: Bug Priority: Medium
Reporter: Jamo Luhrsen Assignee: Janki Chhatbar
Resolution: Done Votes: 0
Labels: csit:failures
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: All
Platform: All

External issue ID: 9131
Priority: High

 Description   

looks like a FIP was not found in this job:

https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-pike-upstream-stateful-nitrogen/3/log.html.gz

tempest snippet:

tempest.scenario.test_server_basic_ops.TestServerBasicOps.test_server_basic_ops[compute,id-7fff3fb3-91d8-4fd0-bd7d-0204f1f180ba,network,smoke]
----------------------------------------------------------------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
Traceback (most recent call last):
File "tempest/common/utils/_init_.py", line 89, in wrapper
return f(self, *func_args, **func_kwargs)
File "tempest/scenario/test_server_basic_ops.py", line 145, in test_server_basic_ops
self.verify_ssh(keypair)
File "tempest/scenario/test_server_basic_ops.py", line 60, in verify_ssh
self.fip = self.create_floating_ip(self.instance)['ip']
File "tempest/scenario/manager.py", line 624, in create_floating_ip
create_floating_ip(pool=pool_name)['floating_ip'])
File "tempest/lib/services/compute/floating_ips_client.py", line 65, in create_floating_ip
resp, body = self.post(url, post_body)
File "tempest/lib/common/rest_client.py", line 270, in post
return self.request('POST', url, extra_headers, headers, body, chunked)
File "tempest/lib/services/compute/base_compute_client.py", line 48, in request
method, url, extra_headers, headers, body, chunked)
File "tempest/lib/common/rest_client.py", line 659, in request
self._error_checker(resp, resp_body)
File "tempest/lib/common/rest_client.py", line 765, in _error_checker
raise exceptions.NotFound(resp_body, resp=resp)
tempest.lib.exceptions.NotFound: Object not found
Details:

{u'message': u'Floating IP pool not found.', u'code': 404}

 Comments   
Comment by Jamo Luhrsen [ 27/Sep/17 ]

another example [0] with it's relevant karaf log [1], neutron server log [2] and tempest log [3]

[0] https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-pike-upstream-stateful-carbon/27/log.html.gz#s1-s2-s1-t16
[1] https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-pike-upstream-stateful-carbon/27/odl1_karaf.log.gz
[2] https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-pike-upstream-stateful-carbon/27/control_1/q-svc.log.2017-09-27-132116.gz
[3] https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-pike-upstream-stateful-carbon/27/tempest/tempest_output_tempest.scenario.test_server_basic_ops.TestServerBasicOps.test_server_basic_ops.log.gz

Comment by Chetan Arakere Gowdru [ 07/Oct/17 ]

From the logs, it looks more like a openstack issue(not an odl issue). The error been thrown during Floating IP creation(neutron floating-ip create) in this use-case. From q-svc logs, I see there is no floating-ip JSON request been sent to ODL.

Since for the full tempest suite, the external-network and external-subnet been created earlier, and floating-ip been are carved out from this subnet. Strange to see only in this TC, fip creation fails and been successful with other tempest run.

I do hit with this same issue locally with Pike + CarbonSR2 while other tempest TC involving FIP creation getting passed.

Comment by Jamo Luhrsen [ 24/Oct/17 ]

I have been working on a patch that will pause the tempest test cases before teardown to collect debugs. I have something
in the sandbox for this bug that may help. See here:

https://logs.opendaylight.org/sandbox/jenkins091/netvirt-csit-1node-openstack-pike-jamo-upstream-stateful-carbon/7/log.html.gz#s1-s2-s1-t5

that should take you to this specific test_server_basic_ops test case. If you drill down in to it you'll see two keywords
OpenStackOperations . Show Debugs
OpenStackOperations . Get Test Teardown Debugs

Those are executed at the point where the test has failed, but before tempest does it's clean up. You can find
some better debugs/details in there (flow dumps, openstack cli outputs, etc)

this is all in the sandbox and that log will be gone in aprox 5 days from now. Eventually my patch will be done and
we'll see this running in releng where the logs stay for 6 months.

Comment by Jamo Luhrsen [ 01/Nov/17 ]

looks like pike needs an extra config in tempest.conf for this test to pass. here is a patch
to try for that. once merged and we see this test case passing reliably we can close this
issue:

https://git.opendaylight.org/gerrit/#/c/65009/

Generated at Wed Feb 07 20:22:45 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.