Skip to content

Telemetry: report how workspaces build their physical tables - #209

Merged
liam-machine merged 1 commit into
developfrom
feat/telemetry-physical-tooling
Oct 23, 2026
Merged

liam-machine merged 1 commit into
developfrom
feat/telemetry-physical-tooling

Conversation

@liam-machine

Copy link
Copy Markdown
Owner

Why

The Physical stage only compares against dbt today. To decide which other sources to support next (a DBML file, an information_schema snapshot, SQL projects, …), we need to know how ERD Studio users actually build their tables and on which platform.

What

Two optional heartbeat fields, each a subset of a fixed allowlist (contract stays additive, HEARTBEAT_VERSION stays 1):

Field Values
physicalTools dbt, sqlmesh, dataform, databricksBundle, fabric, powerBi, sqlProject, flyway, liquibase, alembic, prisma, django, rails, notebooks, ddlScripts, dbml
platforms databricks, snowflake, bigquery, postgres, redshift, fabric, synapse, sqlserver, duckdb, spark, trino, athena, mysql, oracle, clickhouse, other

Absent = not looked that day (stored NULL); [] = looked, found nothing.

  • src/services/physicalTooling.ts: pure, vscode-free, every file access through an injected probe. Detection uses marker file names plus a fixed pattern in the head of a few small files: CREATE TABLE, Databricks notebook header, SQLMesh config keys, a SQL project's <DSP>, dbt-<adapter> packages, and .dbml files that aren't our own export. The dbt warehouse is read from the first 8 KB of manifest.json (metadata.adapter_type). profiles.yml is never read.
  • extension.ts: recordPhysicalTooling(context) runs 15 s after both activation outcomes, once per workspace per UTC day, and not at all while telemetry is off. Not carried over by nextDay, because it describes workspaces, not the machine.
  • Worker: allowlists, validation, upsert, D1 migration 0003_physical_tooling.sql (applied by deploy.yml before the Worker deploy), tests.
  • telemetry.json, telemetry/README.md (contract + two ready queries) and CHANGELOG updated.

Privacy

Only allowlisted values leave the machine: never a path, name, count or content. test/unit/physicalTooling.test.ts runs detection over an in-memory workspace through the real include/exclude globs, including one full of secret-looking names, and asserts nothing else comes back. The payload privacy test feeds junk through recordPhysicalTooling too.

Testing

  • npm run compile, npm test (192 files, 4162 tests), npm run build: green
  • telemetry/: npm test (160) + npm run check: green
  • Ran the bundled detector over real local repos (Databricks bundle, Power BI PBIP, Alembic app, dbt fixture): correct flags; about 1.2 s of background ripgrep I/O on a 10.5k-file repo, 6–70 ms on normal ones
  • Migrations 0001–0003 plus the README queries run against a throwaway SQLite DB

Note

ERD Studio activates only on workspaceContains:**/dbt_project.yml, so non-dbt repos appear only when someone opens ERD Studio there. The data will skew towards dbt.

🤖 Generated with Claude Code

Adds two optional heartbeat fields so we can see which non-dbt sources the
Physical stage should learn to read first:

- physicalTools: dbt, sqlmesh, dataform, databricksBundle, fabric, powerBi,
  sqlProject, flyway, liquibase, alembic, prisma, django, rails, notebooks,
  ddlScripts, dbml
- platforms: databricks, snowflake, bigquery, postgres, redshift, fabric,
  synapse, sqlserver, duckdb, spark, trino, athena, mysql, oracle,
  clickhouse, other

Detection (src/services/physicalTooling.ts) is pure and returns only
allowlisted values: marker file names plus a fixed pattern in the head of a
few small files. The dbt warehouse comes from manifest.json's adapter_type;
profiles.yml is never read. It runs 15 s after activation, once per
workspace per UTC day, and not at all while telemetry is off. Absent means
not looked; [] means looked and found nothing.

Worker: allowlists, validation, D1 migration 0003 (physical_tools,
platforms), tests. telemetry.json, telemetry/README.md and CHANGELOG
updated.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@liam-machine
liam-machine merged commit 0e20871 into develop Oct 23, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant