Skip to content

Add: tool-smoke gate inside each DFX scene test#771

Merged
ChaoWao merged 1 commit into
hw-native-sys:mainfrom
ChaoWao:dfx-tool-smoke
May 13, 2026
Merged

Add: tool-smoke gate inside each DFX scene test#771
ChaoWao merged 1 commit into
hw-native-sys:mainfrom
ChaoWao:dfx-tool-smoke

Conversation

@ChaoWao
Copy link
Copy Markdown
Collaborator

@ChaoWao ChaoWao commented May 13, 2026

Summary

Each DFX capture pipeline (dep_gen, l2_swimlane, tensor_dump) ships with a consumer script under simpler_setup/tools/. The corresponding scene test now invokes the consumer against the artifact it just produced and asserts exit code 0. A future schema change that breaks a tool fails the same CI step that captured the artifact — no silent tooling rot.

  • Smoke is exit-code-only. HTML / PDF / Perfetto JSON content is not validated. The contract is "the tool still parses this schema".
  • PMU has no smoke — there's no consumer tool for raw pmu.csv.
  • sched_overhead_analysis is intentionally skipped — it needs a real device log and would false-positive on sim. Reserve for a future hardware DFX smoke.

Wiring

Each test calls the consumer via a plain subprocess.run([sys.executable, "-m", "simpler_setup.tools.<name>", artifact, ...], check=True, timeout=60) — stdlib only, no helper module:

  • tests/.../dfx/dep_gen/test_dep_gen.py — smokes deps_to_graph in both default and --show-tensor-info modes, guarded by shutil.which("dot") so dev machines without graphviz skip cleanly.
  • tests/.../dfx/l2_swimlane/test_l2_swimlane.py — smokes swimlane_converter against l2_perf_records.json.
  • tests/.../dfx/tensor_dump/test_tensor_dump.py — smokes dump_viewer against the captured tensor_dump/ directory.
  • .github/workflows/ci.yml — installs graphviz on the github-hosted sim runners (Linux apt + macOS brew); the self-hosted onboard a2a3 runner emits a ::warning:: if dot is missing, then the shutil.which guard skips that one smoke without failing CI.

Test plan

  • All 4 dfx tests pass via pytest --platform a2a3sim on macOS arm64 with corresponding --enable-* flags
  • _smoke_deps.html, _smoke_deps_with_tensors.html, _smoke_swimlane.json produced (presence verified)
  • dump_viewer smoke runs (test passes with the call in place)
  • Pre-commit gate clean (clang-format, clang-tidy, cpplint, markdownlint, ruff, pyright, check-headers, etc.)
  • CI ubuntu + macOS sim smokes green
  • Self-hosted a2a3 onboard smokes green (or warning if graphviz unavailable on the runner)

Follow-up to #769.

Copy link
Copy Markdown

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a smoke testing utility in simpler_setup/tools/_smoke.py designed to verify that DFX capture pipeline tools can successfully parse schemas. These smoke tests have been integrated into several scene tests, including those for deps_to_graph, swimlane_converter, and dump_viewer. The reviewer suggested improving the robustness of the run_tool function by explicitly handling subprocess.TimeoutExpired to capture output during timeouts, and recommended removing Path from the __all__ export to maintain a cleaner public API.

Comment thread simpler_setup/tools/_smoke.py Outdated
Comment thread simpler_setup/tools/_smoke.py Outdated
Each DFX capture pipeline (dep_gen / l2_swimlane / tensor_dump) ships
with a consumer script under simpler_setup/tools/. The scene test for
that pipeline now invokes the consumer against the artifact it just
produced, asserting exit code 0. If a future schema change breaks the
tool, the failure attributes to the same CI step that captured the
artifact rather than surfacing later as a silent tooling rot.

Smoke is exit-code-only — HTML / PDF / diagram content is NOT validated.
The contract is "does the tool still parse this schema", not "is the
rendered output correct".

Wiring
- simpler_setup/tools/_smoke.py: run_tool + has_binary helpers shared
  by all DFX tests.
- tests/.../dfx/dep_gen/test_dep_gen.py: deps_to_graph smoked in both
  default and --show-tensor-info modes (guarded by has_binary("dot")
  so dev machines without graphviz skip cleanly).
- tests/.../dfx/l2_swimlane/test_l2_swimlane.py: swimlane_converter
  smoked against l2_perf_records.json.
- tests/.../dfx/tensor_dump/test_tensor_dump.py: dump_viewer smoked
  against the captured tensor_dump/ directory.
- pmu has no consumer tool; no smoke added (raw csv is the artifact).

sched_overhead_analysis is intentionally NOT smoked — it requires a
real device log and would false-positive on sim. Reserve for a future
hardware DFX smoke.

CI: graphviz installed on the github-hosted sim runners (both Linux
and macOS) so deps_to_graph can render. The self-hosted onboard a2a3
runner warns if graphviz is missing instead of failing — the runner
admin should install it for full coverage, otherwise the has_binary
guard skips that one smoke.

Issue: follow-up to hw-native-sys#769
@ChaoWao ChaoWao merged commit a94d514 into hw-native-sys:main May 13, 2026
14 checks passed
@ChaoWao ChaoWao deleted the dfx-tool-smoke branch May 13, 2026 09:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant