[tempest] Fix phantom test results from timing-data archive - #412
Conversation
|
Skipping CI for Draft Pull Request. |
When discover-tempest-config fails and tempest run never executes, stestr last --subunit picks up old test results from the timing-data archive and reports them as if they passed in this run. Two fixes: - Extract only the timing database files (times.dbm*) from the archive instead of the entire .stestr directory which includes old test result files - Track whether tempest run actually executed and skip result collection when it did not Closes: https://redhat.atlassian.net/browse/OSPNW-1674 Signed-off-by: Itay Matza <imatza@redhat.com>
299125a to
3bdad31
Compare
kstrenkova
left a comment
There was a problem hiding this comment.
The code itself looks good to me. Normally, I would also test it locally when I am reviewing a patch. However, due to having a load of work assigned at the moment, I will believe you tested the functionality 😄
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: abays, imatza-rh, kstrenkova The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
0f06bc7
into
openstack-k8s-operators:main
|
/cherry-pick antelope |
|
@imatza-rh: new pull request created: #413 DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
When
discover-tempest-configfails andtempest runnever executes,stestr last --subunitpicks up old test results from the timing-dataarchive and reports them as if they passed in this run.
Root Cause
The timing-data archive
contains the full
.stestr/directory from previous runs — including old testresult files — not just the timing database that stestr uses for test scheduling.
When
discover-tempest-configfails, those old results are picked up by theunconditional
generate_test_resultscall.Example: https://sf.apps.int.gpc.ocp-hub.prod.psi.redhat.com/logs/25f/components-integration/25ffb2f08d194ccba0123053d8dfb21b/
Fix
Two changes in
run_tempest.sh:Extract only timing database files from the archive (
times.dbm*)instead of the entire
.stestr/directory. stestr usesdbm.dumbwhichstores timing data as three files:
.bak,.dat,.dir.Guard result collection — track whether
tempest runactually executedand skip
move_tempest_log+generate_test_resultswhen it did not.Both layers are complementary: the tar filter prevents old results from being
extracted, and the execution guard prevents result collection when tests did
not run.
Testing
stestrsource: usesfrom dbm import dumb as my_dbmfortimes.dbmsave_config_filesoutputtimes.dbm.*files extracted)RETURN_VALUEsemantics are preserved (3 cases: both succeed, discover fails, tempest fails)ifcondition suppresses ERR trap (matching original&&behavior)rerun_failed_testsandcheck_expected_failureshandle missingFAILED_TESTS_FILEcorrectlyCloses: https://redhat.atlassian.net/browse/OSPNW-1674