Skip to content

[tempest] Fix phantom test results from timing-data archive - #412

Merged
openshift-merge-bot[bot] merged 1 commit into
openstack-k8s-operators:mainfrom
imatza-rh:fix/phantom-tempest-results
Aug 5, 2026
Merged

[tempest] Fix phantom test results from timing-data archive#412
openshift-merge-bot[bot] merged 1 commit into
openstack-k8s-operators:mainfrom
imatza-rh:fix/phantom-tempest-results

Conversation

@imatza-rh

Copy link
Copy Markdown
Contributor

When discover-tempest-config fails and tempest run never executes,
stestr last --subunit picks up old test results from the timing-data
archive and reports them as if they passed in this run.

Root Cause

The timing-data archive
contains the full .stestr/ directory from previous runs — including old test
result files — not just the timing database that stestr uses for test scheduling.
When discover-tempest-config fails, those old results are picked up by the
unconditional generate_test_results call.

Example: https://sf.apps.int.gpc.ocp-hub.prod.psi.redhat.com/logs/25f/components-integration/25ffb2f08d194ccba0123053d8dfb21b/

Fix

Two changes in run_tempest.sh:

  1. Extract only timing database files from the archive (times.dbm*)
    instead of the entire .stestr/ directory. stestr uses dbm.dumb which
    stores timing data as three files: .bak, .dat, .dir.

  2. Guard result collection — track whether tempest run actually executed
    and skip move_tempest_log + generate_test_results when it did not.

Both layers are complementary: the tar filter prevents old results from being
extracted, and the execution guard prevents result collection when tests did
not run.

Testing

  • Verified stestr source: uses from dbm import dumb as my_dbm for times.dbm
  • Verified archive structure matches save_config_files output
  • Tested tar wildcard extraction locally (only times.dbm.* files extracted)
  • Verified RETURN_VALUE semantics are preserved (3 cases: both succeed, discover fails, tempest fails)
  • Verified bash ERR trap interaction: if condition suppresses ERR trap (matching original && behavior)
  • Verified rerun_failed_tests and check_expected_failures handle missing FAILED_TESTS_FILE correctly
  • bashate passes clean

Closes: https://redhat.atlassian.net/browse/OSPNW-1674

@openshift-ci

openshift-ci Bot commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@imatza-rh
imatza-rh marked this pull request as ready for review August 3, 2026 08:33
@openshift-ci
openshift-ci Bot requested review from rabi and stuggi August 3, 2026 08:33
@imatza-rh
imatza-rh requested a review from kstrenkova August 3, 2026 08:48
Comment thread container-images/tcib/base/os/tempest/run_tempest.sh
Comment thread container-images/tcib/base/os/tempest/run_tempest.sh
When discover-tempest-config fails and tempest run never executes,
stestr last --subunit picks up old test results from the timing-data
archive and reports them as if they passed in this run.

Two fixes:
- Extract only the timing database files (times.dbm*) from the
  archive instead of the entire .stestr directory which includes
  old test result files
- Track whether tempest run actually executed and skip result
  collection when it did not

Closes: https://redhat.atlassian.net/browse/OSPNW-1674
Signed-off-by: Itay Matza <imatza@redhat.com>
@imatza-rh
imatza-rh force-pushed the fix/phantom-tempest-results branch from 299125a to 3bdad31 Compare August 5, 2026 10:21

@kstrenkova kstrenkova left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code itself looks good to me. Normally, I would also test it locally when I am reviewing a patch. However, due to having a load of work assigned at the moment, I will believe you tested the functionality 😄

Comment thread container-images/tcib/base/os/tempest/run_tempest.sh
Comment thread container-images/tcib/base/os/tempest/run_tempest.sh

@abays abays left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@openshift-ci

openshift-ci Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: abays, imatza-rh, kstrenkova

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added the approved label Aug 5, 2026
@openshift-merge-bot
openshift-merge-bot Bot merged commit 0f06bc7 into openstack-k8s-operators:main Aug 5, 2026
6 checks passed
@imatza-rh

Copy link
Copy Markdown
Contributor Author

/cherry-pick antelope

@openshift-cherrypick-robot

Copy link
Copy Markdown

@imatza-rh: new pull request created: #413

Details

In response to this:

/cherry-pick antelope

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants