feat: SQL query-plan diagrams, and a reports/ folder for generated artifacts - #50
Merged
Conversation
Add scripts/sql_lineage.py, which parses a query with sqlglot and emits a Mermaid flowchart tracing each output column back to the source columns it reads, grouped into one subgraph per table. Exposed as `make sql-lineage` and driven by the /sql-lineage command. Lineage comes from the AST rather than model inference, so the edges are exactly what the query says. Columns that cannot be attributed to a table are grouped under "(unqualified)" instead of guessed at, and SELECT * is rejected outright since tracing it would need the table schemas. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Expand specs/tooling.md with install-layout and MCP-identity sections, consolidate the changelog discipline and branch rules into specs/workflow.md, and repoint CLAUDE.md at the new anchors. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Rework the sqlglot diagram tool from lineage-only into sql-diagram, whose default mode draws the query's execution steps: one node per scan, one per individual join with its side and keys, then filter, aggregate and sort. A CTE renders as its own sub-pipeline. sqlglot models a multi-table join as a single n-ary step, so build_plan splits it back into JOIN 1, JOIN 2, … in written order — which is what makes an under-constrained join predicate visible. Column lineage stays available as mode=lineage. Diagrams carry each join's full ON condition and, with --comments, the source table's Unity Catalog comment. Comment lookup is opt-in because it is the only part that touches the network, and it uses the dev profile: the MCP service principal lacks USE SCHEMA on system.billing. A table with no comment is left unannotated rather than described from guesswork. SVG is written by hand rather than shelled out to mmdc, which would pull a Node toolchain and a headless Chromium into a Python repo — the trade scripts/star_history.py already makes. Both renderers use one CSS class per box: a multi-class cascade is not resolved by every SVG renderer. Collect generated artifacts under reports/: coverage_reports/ becomes reports/coverage/ and cost_report/ becomes reports/cost/, with diagrams in reports/sql-diagram/. All three are gitignored, each keeping one committed example. Refreshes that example cost report to 2026-07-22. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…rministic sql-diagram now writes <name>.sql alongside the .mmd and .svg. A diagram without the query it came from cannot be checked, and the source is usually an f-string in a .py file that no longer reads the same once its placeholders are filled in; the emitted file is the query as parsed, so re-running the tool on it reproduces the diagram. Verifying that round trip exposed a real defect: planner.Step.dependencies is a set, so iteration order followed the process hash seed and node numbering changed on every run — three runs over one input gave three different files. That made the .mmd churn in git, defeating the diffable property it was documented to have. Dependencies are now walked in a stable (name, type) order, and the round trip reproduces byte-for-byte. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
andre-salvati
added a commit
that referenced
this pull request
Jul 22, 2026
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
andre-salvati
force-pushed
the
feat/sql-diagram
branch
from
July 22, 2026 16:48
bf624d2 to
b046b71
Compare
…tests Each fix gets a test named after the defect, so a regression says what it broke. scripts/ has had no test coverage until now, which is how these survived. - Plan mode no longer inherits lineage's constraints. main computed the lineage graph unconditionally, so a SELECT * query the plan renders fine died in the tracer that never ran. Each mode now builds only what it renders, which also drops four redundant parses of the same SQL. - Join predicates are split on the AST, not on generated text. The old regex deleted any "TRUE AND" — including one written inside a real predicate — and text splitting tore apart string literals containing AND and miscounted parens around a literal ")". An OR conjunct keeps its parentheses: prefixed with "and", a bare OR reads as the opposite grouping. - A join with no keys and no predicate is labelled CROSS, not INNER. Calling a cross product an inner join hides exactly the unconstrained join this diagram exists to expose. - Unity Catalog comments are matched to a scan by exact table name. The substring match gave system.billing.usage_extra the comment belonging to system.billing.usage. - The driving table is read from the FROM clause. Deducing it by elimination lost every join edge when it projected no columns of its own, and it is now drawn even then. Note that sqlglot v30 renames the arg to `from_`, so the previous key lookup would have returned None silently. - CTAS and INSERT … SELECT are unwrapped and diagrammed; unparseable or SELECT-less input exits with one line instead of a traceback. - Lineage refuses a query projecting one name twice rather than tracing both to the first match. The review suggested keying the graph by index, but sqlglot resolves lineage by name, so that keeps both nodes while attributing one to the wrong table — a confident wrong diagram is worse than a refusal. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The eight review fixes in scripts/sql_diagram.py stay; only their regression coverage is dropped. Recoverable from ce0fe54 if wanted back. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What?
Adds
make sql-diagram//sql-diagram: asqlglot-based tool that draws a SQL query either as its execution steps (default) or as column-level lineage, writing the analysed.sql, a Mermaid.mmdand a standalone.svg. Also collects generated artifacts under a newreports/folder —coverage_reports/becomesreports/coverage/,cost_report/becomesreports/cost/, diagrams land inreports/sql-diagram/— and folds in the pending doc consolidation inspecs/tooling.md/specs/workflow.md.Why?
Reading a non-trivial query in this repo means reconstructing its shape by hand, and the queries that matter most (the
system.billingcost queries) live interpolated inside f-strings, so the SQL that actually runs exists nowhere in the tree. Existing Claude skills for this either analyse anEXPLAINyou paste in or draw generic Mermaid from model inference; none derives the diagram from the query. Parsing the AST gives edges that are what the query says rather than what a model guessed.How?
sqlglotmodels a multi-table join as one n-ary step, sobuild_plansplits it back intoJOIN 1,JOIN 2, … in written order — that is what makes an under-constrained join predicate visible, which is exactly the class of bug that caused the 3.5× fan-out fixed in #47. Each join carries its side and fullONcondition;--commentsannotates tables with their Unity Catalog comment via thedevprofile, since the MCP service principal lacksUSE SCHEMAonsystem.billing. It is opt-in as the only network call, and a table without a comment is left unannotated rather than described from guesswork.SVG is written by hand rather than shelled out to
mmdc, which would pull a Node toolchain and headless Chromium into a Python repo — the same tradescripts/star_history.pyalready makes. One CSS class per box, because a multi-class cascade is not resolved by every SVG renderer.Validation?
uv run pytestpasses with coverage writing to the new path; ruff clean. Both modes exercised on the two cost queries, and the SVG rasterised and inspected at each step — which caught black-on-black boxes twice and mid-sentence truncation. Round-tripping the emitted.sqlexposed non-determinism:planner.Step.dependenciesis aset, so node numbering followed the hash seed and three runs over one input produced three different files, making the.mmdchurn in git. Dependencies are now walked in a stable order and the round trip reproduces byte-for-byte across five runs in both modes.Impact in prod
No medallion table, job, or pipeline is touched. The one operational change is the CI coverage artifact path in
onpush.yml, updated toreports/coverage/in the same commit as thepyproject.tomlsetting that produces it.