feat(sql): extract table-to-table lineage from INSERT/CTAS/MERGE/UPDATE (#1572)#1777
Open
rithyKabir wants to merge 1 commit into
Open
feat(sql): extract table-to-table lineage from INSERT/CTAS/MERGE/UPDATE (#1572)#1777rithyKabir wants to merge 1 commit into
rithyKabir wants to merge 1 commit into
Conversation
…TE (Graphify-Labs#1572) Dataflow edges were only walked inside create_view/create_function/ create_procedure, so ETL corpora (mostly INSERT INTO ... SELECT bodies) produced schema structure but essentially zero source->target lineage. extract_sql now emits <target> --reads_from--> <source> edges, reusing the existing _walk_from_refs traversal, for: - INSERT INTO tgt ... SELECT ... FROM/JOIN src (the target node is materialized, so plain VALUES inserts now at least produce a node) - CREATE TABLE tgt AS SELECT ... (CTAS; the SELECT lives in a create_query child) - MERGE INTO tgt USING src (tree-sitter-sql has no merge node - the keywords sit directly under statement, so it is handled there; USING (SELECT ...) subqueries are covered via the nested from walk) - UPDATE tgt SET ... FROM src _walk_from_refs now skips self-loops (INSERT INTO t SELECT ... FROM t backfill patterns read their own target). Also registers .ddl as a SQL extension (dispatch, CODE_EXTENSIONS, case-insensitive set, and the Graphify-Labs#1745 extras map), per the issue's nice-to-have. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1572.
Problem
Dataflow edges were only walked inside
create_view/create_function/create_procedure, so ETL corpora — mostlyINSERT INTO … SELECTbodies — produced schema structure but essentially zero source→target lineage (the issue measured ~16% of 700 files yielding any node and no lineage edges).What this adds
extract_sqlnow emits<target> --reads_from--> <source>edges, reusing the existing_walk_from_refstraversal, for all four requested statement types:INSERT INTO tgt … SELECT … FROM/JOIN src— the target node is materialized (so plainVALUESinserts now at least produce a node, per the issue's "no target node" complaint), and sources come from the nestedfrom/joinwalk.VALUESinserts have nofromnode, so they can't create false lineage.CREATE TABLE tgt AS SELECT …(CTAS) — the SELECT lives in acreate_querychild ofcreate_table; walked exactly likecreate_viewalready is.MERGE INTO tgt USING src— tree-sitter-sql has no dedicated merge node: the keywords sit directly understatement, so this is handled at the statement loop. Only direct children are scanned for the target/sourceobject_references (nested ON-clause fields never surface there), andUSING (SELECT … FROM src)subqueries are covered by the nestedfromwalk.UPDATE tgt SET … FROM src— the target is therelationdirectly underupdate, so_walk_from_refs(which only looks underfrom/join) never sees the target itself._walk_from_refsnow skips self-loops:INSERT INTO t SELECT … FROM t(dedup/backfill patterns) reads its own target and previously would have emitted at → tedge.Also includes the issue's nice-to-have:
.ddlis registered as a SQL extension — dispatch table,CODE_EXTENSIONS, the case-insensitive-identifier set, and the #1745 missing-extra warning map.Tests
8 new tests over a realistic ETL fixture (
tests/fixtures/sample_dml_lineage.sql) covering each statement type plus the two negative cases (no false lineage fromVALUES, no self-loop from self-referential backfill) and a no-dangling-edges check.tests/test_multilang.py56/56;test_extract.py+test_pg_introspect.pygreen; ruff clean. Full-suite failures on my machine (4×test_ollama_retry_cap, 1 order-flaky labeling test) reproduce identically on cleanv8— pre-existing and unrelated.The remaining nice-to-have from the issue (normalizing templated identifiers like
{project}.{dataset}.tbl) is not included — it needs a design decision about placeholder handling and feels like its own change.🤖 Generated with Claude Code