feat(ci): monitors marker + three-way Better Stack heartbeat split#641
Open
ari-nz wants to merge 3 commits into
Open
feat(ci): monitors marker + three-way Better Stack heartbeat split#641ari-nz wants to merge 3 commits into
ari-nz wants to merge 3 commits into
Conversation
ari-nz
requested review from
a team and
helmut-hoffer-von-ankershoffen
as code owners
There was a problem hiding this comment.
Pull request overview
This PR introduces a monitors(...) pytest marker intended to tag scheduled E2E tests by the Better Stack monitor they should feed, and updates the hourly scheduled CI workflow to split test execution and emit separate heartbeats (SDK vs. application-specific).
Changes:
- Added
@pytest.mark.monitors("he-tme" | "test-app")to selected platform E2E scheduled tests. - Registered the new
monitorsmarker inpyproject.tomlpytest configuration. - Split the hourly scheduled workflow into multiple pytest invocations and added a dedicated Better Stack heartbeat URL secret for HE-TME (propagated via the staging/production wrapper workflows).
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
tests/aignostics/platform/e2e_test.py |
Tags scheduled E2E tests with monitors(...) to support routing results/heartbeats by monitored system. |
pyproject.toml |
Registers the new pytest marker so --strict-markers runs remain valid. |
.github/workflows/_scheduled-test-hourly.yml |
Splits scheduled tests into separate runs and sends separate Better Stack heartbeats (SDK + HE-TME), plus combined status to Sentry. |
.github/workflows/scheduled-testing-staging-hourly.yml |
Passes through the new HE-TME Better Stack heartbeat secret to the reusable workflow. |
.github/workflows/scheduled-testing-production-hourly.yml |
Passes through the new HE-TME Better Stack heartbeat secret to the reusable workflow. |
Comment on lines
+123
to
+128
| # set +e so a test failure does not abort the step — we capture the exit code | ||
| # manually and send it to Better Stack regardless of outcome. | ||
| set +e | ||
| make test_scheduled | ||
| EXIT_CODE=$? | ||
| XDIST_WORKER_FACTOR=1 uv run --all-extras nox -s test -- \ | ||
| -m "(scheduled or scheduled_only) and monitors and not stress_only" \ | ||
| --junit-xml=reports/junit_he_tme.xml |
Comment on lines
+117
to
+129
| - name: Test / scheduled / he-tme | ||
| id: test_he_tme | ||
| env: | ||
| BETTERSTACK_HEARTBEAT_URL: "${{ inputs.platform_environment == 'staging' && secrets.BETTERSTACK_HEARTBEAT_URL_STAGING || secrets.BETTERSTACK_HEARTBEAT_URL_PRODUCTION }}" | ||
| SENTRY_DSN: ${{ secrets.SENTRY_DSN }} | ||
| shell: bash | ||
| run: | | ||
| # set +e so a test failure does not abort the step — we capture the exit code | ||
| # manually and send it to Better Stack regardless of outcome. | ||
| set +e | ||
| make test_scheduled | ||
| EXIT_CODE=$? | ||
| XDIST_WORKER_FACTOR=1 uv run --all-extras nox -s test -- \ | ||
| -m "(scheduled or scheduled_only) and monitors and not stress_only" \ | ||
| --junit-xml=reports/junit_he_tme.xml | ||
| echo "exit_code=$?" >> $GITHUB_OUTPUT |
| "unit: Solitary unit tests - test a layer of a module in isolation with all dependencies mocked, except interaction with shared utils and the systems module. Unit tests must be able to pass offline, i.e. not calls to external services. The timeout should not be bigger than the default 10s, and must be <5 min.", | ||
| "integration: Sociable integration tests - test interactions across architectural layers (e.g. CLI/GUI→Service, Service→Utils) or between modules (e.g. Application→Platform), using real SDK collaborators, real file I/O, real subprocesses, and real Docker containers. Integration test must be able to pass offline, i.e. mock external services (Aignostics Platform API, Auth0, S3/GCS buckets, IDC). The timeout should not be bigger than the default 10s, and must be <5 min.", | ||
| "e2e: End-to-end tests - test complete workflows with real external network services (Aignostics Platform API, cloud storage, IDC, etc). If the test timeout is >= 5 min and < 60 min, additionally mark as `long_running`, if >= 60min mark as 'very_long_running'.", | ||
| "monitors: Tag a scheduled test with the application it monitors, e.g. @pytest.mark.monitors('he-tme'). Tests without this marker are considered SDK-layer health checks. Used to route Better Stack heartbeats to the correct monitor.", |
Comment on lines
+288
to
+289
| reports/junit_sdk.xml | ||
| reports/junit_he_tme.xml |
ari-nz
changed the title
Codecov Report✅ All modified and coverable lines are covered by tests. ❌ Your project check has failed because the head coverage (65.75%) is below the target coverage (70.00%). You can increase the head coverage or adjust the target coverage.
|
Add @pytest.mark.monitors("he-tme") marker to tag tests that exercise
the HE-TME application end-to-end. Tests without the marker are SDK
health checks (auth, listing, connectivity).
The hourly scheduled workflow now runs two separate pytest invocations
and sends two independent Better Stack heartbeats so an HE-TME outage
no longer pollutes the SDK monitor and vice versa.
The HE-TME heartbeat URLs are optional secrets; the step skips
gracefully until the monitors are created in Better Stack.
Extends the monitors marker to the three test-app e2e tests so they route to a test-app Better Stack monitor independently from the SDK health checks.
…lications
Tests are now routed to three separate Better Stack heartbeat monitors:
- SDK (no monitors marker): token management, service wiring
- Platform API (monitors_platform_api): auth, listing, connectivity
- test_cli_health_json (system)
- test_cli_application_list_verbose (application)
- test_cli_run_list_limit_10 (application)
- HE-TME / applications (monitors): application processing
- existing he-tme and test-app tests (unchanged)
Adds monitors_platform_api pytest marker alongside the existing
monitors("platform-api") string for pytest -m expression filtering.
Adds BETTERSTACK_HEARTBEAT_URL_PLATFORM_API_* secrets (optional,
heartbeat step skips gracefully when absent).
ari-nz
force-pushed
the
feat/monitors-marker-split
branch
from
129c9aa to
0683245
Compare
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
monitorspytest marker so scheduled tests can declare which system they monitor (he-tme,test-app,platform-api). Tests without the marker are considered SDK-layer health checks.not monitors and not monitors_platform_api): token management, service wiringmonitors_platform_api): health check, application listing, run listingmonitors and not monitors_platform_api): HE-TME and test-app processing testsmonitors_platform_apias a companion boolean marker (alongsidemonitors("platform-api")) so the workflow can filter with standard-mexpressions — pytest's-msyntax cannot filter on marker argument values.BETTERSTACK_HEARTBEAT_URL_PLATFORM_API_{STAGING,PRODUCTION}are optional; heartbeat steps skip gracefully when absent.if: always()so they fire even if an upstream step errors unexpectedly — missing a heartbeat would cause a false "down" alert.|| '1'fallback per step output to prevent false-healthy heartbeats when a step never ran.Test plan
uv run pytest --collect-only -m "monitors_platform_api"→ 3 testsuv run pytest --collect-only -m "monitors and not monitors_platform_api"→ 6 tests (3 he-tme + 3 test-app)uv run pytest --collect-only -m "not monitors and not monitors_platform_api" -m "scheduled or scheduled_only"→ SDK-only scheduled testsPosted by Claude claude-sonnet-4-6 via Claude Code on behalf of ari@aignostics.com