Repository navigation
Telemetry: report how workspaces build their physical tables - #209
Merged
Merged
Conversation
Adds two optional heartbeat fields so we can see which non-dbt sources the Physical stage should learn to read first: - physicalTools: dbt, sqlmesh, dataform, databricksBundle, fabric, powerBi, sqlProject, flyway, liquibase, alembic, prisma, django, rails, notebooks, ddlScripts, dbml - platforms: databricks, snowflake, bigquery, postgres, redshift, fabric, synapse, sqlserver, duckdb, spark, trino, athena, mysql, oracle, clickhouse, other Detection (src/services/physicalTooling.ts) is pure and returns only allowlisted values: marker file names plus a fixed pattern in the head of a few small files. The dbt warehouse comes from manifest.json's adapter_type; profiles.yml is never read. It runs 15 s after activation, once per workspace per UTC day, and not at all while telemetry is off. Absent means not looked; [] means looked and found nothing. Worker: allowlists, validation, D1 migration 0003 (physical_tools, platforms), tests. telemetry.json, telemetry/README.md and CHANGELOG updated. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The Physical stage only compares against dbt today. To decide which other sources to support next (a DBML file, an
information_schemasnapshot, SQL projects, …), we need to know how ERD Studio users actually build their tables and on which platform.What
Two optional heartbeat fields, each a subset of a fixed allowlist (contract stays additive,
HEARTBEAT_VERSIONstays 1):physicalToolsplatformsAbsent = not looked that day (stored NULL);
[]= looked, found nothing.src/services/physicalTooling.ts: pure, vscode-free, every file access through an injected probe. Detection uses marker file names plus a fixed pattern in the head of a few small files:CREATE TABLE, Databricks notebook header, SQLMesh config keys, a SQL project's<DSP>,dbt-<adapter>packages, and.dbmlfiles that aren't our own export. The dbt warehouse is read from the first 8 KB ofmanifest.json(metadata.adapter_type).profiles.ymlis never read.extension.ts:recordPhysicalTooling(context)runs 15 s after both activation outcomes, once per workspace per UTC day, and not at all while telemetry is off. Not carried over bynextDay, because it describes workspaces, not the machine.0003_physical_tooling.sql(applied bydeploy.ymlbefore the Worker deploy), tests.telemetry.json,telemetry/README.md(contract + two ready queries) and CHANGELOG updated.Privacy
Only allowlisted values leave the machine: never a path, name, count or content.
test/unit/physicalTooling.test.tsruns detection over an in-memory workspace through the real include/exclude globs, including one full of secret-looking names, and asserts nothing else comes back. The payload privacy test feeds junk throughrecordPhysicalToolingtoo.Testing
npm run compile,npm test(192 files, 4162 tests),npm run build: greentelemetry/:npm test(160) +npm run check: greenNote
ERD Studio activates only on
workspaceContains:**/dbt_project.yml, so non-dbt repos appear only when someone opens ERD Studio there. The data will skew towards dbt.🤖 Generated with Claude Code