Skip to content

product: add verified setup diagnostics and a provider capture matrix #239

Description

@Siddhant-K-code

Goal

Turn installation into a bounded, verifiable first local trace and publish exactly what each supported capture path can and cannot observe.

This follows the activation and trust actions in the Telemetry.dev learning. It is not a request for more integration logos or a hosted onboarding flow.

Problem

agent-strace setup can generate or install provider hooks, and the setup guide describes several limitations. A user still has to infer whether the active configuration works, which events were actually captured, whether the session closed cleanly, and how the observed coverage differs by provider.

The current integration table is not a versioned verification record. Broad feature wording can therefore outrun the behavior of a particular provider version.

Scope

Verified activation

Add a supported verification path, such as agent-strace setup --verify --cli <provider> or a narrowly scoped doctor command, that:

  • resolves the configuration file and executable that the provider will actually use;
  • confirms that generated commands exist and match the installed AgentTrace version;
  • guides the user through one representative local session;
  • reports the captured session ID, provider, capture method, observed event types, start/end markers, redaction state, and known gaps; and
  • exits non-zero when setup is missing, stale, or cannot produce reviewable evidence.

Do not claim success merely because a config file was written.

Capture matrix

Publish a human-readable and machine-readable matrix for the supported CLI adapters, initially Claude Code and Codex, covering:

  • provider and tested version;
  • setup/config location;
  • session, prompt, response, tool, file, command, error, and stop coverage;
  • fields that are provider-defined, optional, or unavailable;
  • privacy/redaction behavior;
  • rollback/removal steps; and
  • last verified date and fixture/test reference.

Keep provider-specific semantics visible rather than flattening them into a single “full capture” claim.

Reproducibility

  • Add clean-environment tests for the documented setup and rollback commands.
  • Use only commands shipped in the current package.
  • Make stale matrix entries visible in CI or release review.

Out of scope

  • Adding new providers merely to increase integration count.
  • Hosted signup, ingestion, or account management.
  • Treating provider hooks as access to hidden reasoning or complete ground truth.

Acceptance criteria

  • A clean supported environment can follow one documented path from install to a verified local session.
  • Verification distinguishes “config written” from “events observed.”
  • The result names the session, provider, event types, redaction state, and known gaps.
  • Failures return actionable diagnostics and a non-zero exit status.
  • Claude Code and Codex have versioned capture-matrix entries with rollback and last-verified metadata.
  • Human and machine-readable matrix outputs are checked for consistency.
  • README/setup claims link to the matrix instead of claiming undifferentiated full capture.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentationenhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions