Scribe

Observability and evaluation starter kit for the boringcomputers agents (Ruth, Ezekiel, Bezalel).

Scribe is a template and governance source, not a runtime package. It publishes tagged, checksummed recipes — telemetry contract, scrub catalog, corpus writer, Collector configuration, evaluation adapter, and operational skills — that consumers copy into their repos and verify with verify-recipes against .scribe.json. The core is vendor-neutral OpenTelemetry; Dynatrace is an optional exporter and evaluation backend.

See PLAN.md for the full 90-day plan, interfaces, and acceptance criteria.

Layout

• packages/kit — all recipe sources
  • src/contract — AgentInteractionRecordV1, validation, attribute namespaces, principal hashing
  • src/propagation — W3C traceparent helpers for the Eve → MCP → Bezalel boundary
  • src/scrub — versioned scrub catalog (every pattern ships with tested examples) + engine
  • src/corpus — telemetry policy (non-configurable sensitive-capability override), R2 S3 client, corpus writer
  • src/evals — offline evaluation adapter: JSONL sources, judges (pinned dt-eval-lib via AI Gateway + deterministic stub), runner with fail-closed retries, gates, JUnit, R2 run store
  • src/cli — the scribe command (eval validate, eval run, doctor, verify-recipes, install-recipe, gen-collector)
  • fixtures/ — fixture records and eval configs (stub + gateway)
• collector/ — self-hosted OpenTelemetry Collector; otel-collector.yaml is generated from the scrub catalog
• skills/ — agent-operations skills (diagnose a turn by trace ID, compare eval runs)
• apps/web — landing page (Vite, React, Kumo)

The command recipe

node packages/kit/src/cli/scribe.ts eval validate --input records.jsonl --json
node packages/kit/src/cli/scribe.ts eval run --input records.jsonl \
  --config packages/kit/fixtures/eval.stub.yaml --json --junit junit.xml
node packages/kit/src/cli/scribe.ts doctor --json
node packages/kit/src/cli/scribe.ts install-recipe kit-core --to ../ruth
node packages/kit/src/cli/scribe.ts verify-recipes --dir ../ruth
node packages/kit/src/cli/scribe.ts gen-collector

Exit codes: 0 ok, 1 unexpected, 2 schema, 3 config/policy, 4 gate, 5 recipe drift. Commands are non-interactive under --json.

Evaluation tiers

1. Deterministic (PR gate) — stub judge driven by fixture attributes; no live backends, no credentials. Runs in CI on every PR.
2. Judge-backed (post-merge / nightly) — pinned @dynatrace-oss/dt-eval-lib reaching the Vercel AI Gateway (openai provider + baseUrl), default judge anthropic/claude-haiku-4.5. Hard-fails PII/prompt-injection breaches and sub-0.50 relevance/faithfulness; soft-alerts on a 0.10 mean decline vs the pinned baseline.

Only dataClass: "synthetic" or "fixture" records are ever judged. Real-user records go to the corpus and nothing else.

Development

Requires Node 24 and pnpm 11.

pnpm install
pnpm check    # typecheck everything
pnpm test     # kit test suite (scrub catalog guarantees, gates, CLI exit codes)
pnpm build    # landing page build
pnpm dev      # landing page on http://localhost:5173

License

Apache-2.0.