[chalmerlowe]

Problem

In the monorepo CI pipeline, gapic-generator runs two Showcase system test suites:

• showcase: Installs the latest released version of google-api-core from PyPI to verify backward compatibility of generated clients.
• showcase-canary: Installs the local monorepo packages/google-api-core (INSTALL_LOCAL_CORE=true) to verify integration between current gapic-generator and google-api-core changes.

Because google-api-core and gapic-generator release on independent schedules, packages/google-api-core in the monorepo frequently contains unreleased changes that are not yet available on PyPI. When a Showcase system test asserts behavior that depends on unreleased google-api-core changes, showcase-canary passes against the local monorepo source, but the non-canary showcase job fails against the older PyPI release until the next google-api-core release is published.

Previously, individual test files worked around this misalignment using ad-hoc, feature-specific hasattr checks.

Solution

This pull request introduces a general, feature-agnostic mechanism in packages/gapic-generator/tests/system/conftest.py to detect and manage misalignment between the installed PyPI google-api-core package and the monorepo packages/google-api-core source:

1. Monorepo vs. Installed google-api-core Sync Check (conftest.py):

   • Added _installed_core_matches_monorepo() and INSTALLED_CORE_MATCHES_MONOREPO to compare the active google.api_core installation against packages/google-api-core/google/api_core in the monorepo.
   • Added a skip_on_core_mismatch() context manager that always runs the wrapped assertion(s) first and only converts an AssertionError into pytest.skip(...) when the installed PyPI google-api-core lags behind the monorepo source:
     • If a change in google-api-core does not affect a test assertion, the assertion runs and passes without skipping.
     • If an assertion fails against the older PyPI package while google-api-core has unreleased changes, only that failing assertion is skipped in non-canary showcase, while showcase-canary (INSTALL_LOCAL_CORE=true) strictly enforces it.
     • Once the next google-api-core release is published to PyPI, INSTALLED_CORE_MATCHES_MONOREPO automatically becomes True again and all assertions are strictly enforced in both showcase and showcase-canary with no version-marker cleanup required.

2. Initial Application (test_span_compliance.py):

   • Wrapped the final matrix assertion step (assert_span_matches_row) in with conftest.skip_on_core_mismatch(): so upcoming pull requests that adjust google-api-core telemetry attributes do not fail non-canary showcase CI prior to the next google-api-core PyPI release, while still executing all RPC scenarios and unaffected matrix rows against PyPI google-api-core.

[gemini-code-assist]

Code Review

This pull request introduces a mechanism to detect whether the installed google-api-core package matches the monorepo source, skipping specific system and span compliance tests when a mismatch is found. The review feedback suggests replacing binary file comparison (filecmp.cmp) with text-based comparison (read_text) to prevent false mismatches caused by cross-platform line ending differences (CRLF vs LF) on Windows.

[daniel-sanche]

It seems like these tests will always be skipped on non-canary, even years from now, when this feature has long-since been released. Do you think we should add a parameter to this decorator with a future release version, so the tests can enable themselves on that version or greater? @pytest.mark.requires_core_version("2.55.0")

That would give us a signal when the marker can be removed, and allow the tests to run in the future if forgotten

I'm still getting my head around this though. Let me know if that would cause issues, or if I'm misunderstanding

[chalmerlowe]

@daniel-sanche

I just pushed an update to ensure that assert_span_matches_row always runs first, and we only skip a test in showcase if it raises AssertionError AND the installed PyPI google-api-core code differs from the local monorepo code:

How does this play out across various scenarios?:

Unrelated google-api-core changes: If someone the code in a way that is completely unrelated to any of the observability tests that happen in test_span_compliance.py such as changing a docstring, then assert_span_matches_row still runs and we expected to pass for all 38 rows. Thus 0 tests are skipped.

Unreleased google-api-core changes that DO affect spans: All unaffected rows in the matrix still run and pass on showcase; only matrix rows whose assertions fail against the older PyPI package are skipped in showcase (NOTE: showcase-canary runs against local google-api-core and still fully enforces all 38 rows).

After google-api-core changes are released to PyPI: The PyPI and monorepo files match again (INSTALLED_CORE_MATCHES_MONOREPO == True), so all 38 rows enforce strict assertions on both showcase and showcase-canary automatically — without needing to track or clean up version numbers over time.

[daniel-sanche]

So it sounds like failing tests would be marked as skipped? That could hide real errors in the future, right? That sounds dangerous

My suggestion is to add a quick min_version parameter to the decorator, and actually raise errors after that version is released, rather than skipping

If that's too difficult, I think the old behavior of always skipping is actually better, because skips are more likely to be noticed and result in the decorator being removed, rather than silent guaranteed passes

[daniel-sanche]

Approving because this is test code, and I don't want to block the PRs waiting on this. But consider my comments about reverting the "skip on fail" behavior

[chalmerlowe]

I will revisit with you and we can find a better long-term solution, thanks Daniel.