diff --git a/examples/README.md b/examples/README.md index f5b7076100..e32373bc31 100644 --- a/examples/README.md +++ b/examples/README.md @@ -13,6 +13,7 @@ Tutorials to get started with generic ingestion tasks in VDK: Tutorials to get started with generic processing tasks in VDK: * [Sqlite Processing](sqlite-processing-example/) +* [VDK Lineage](vdk-lineage-example/) ## Advanced Examples Tutorials with a complete pipeline from ingestion to final publication: diff --git a/examples/vdk-lineage-example/10_drop_raw_orders.sql b/examples/vdk-lineage-example/10_drop_raw_orders.sql new file mode 100644 index 0000000000..4e49f21115 --- /dev/null +++ b/examples/vdk-lineage-example/10_drop_raw_orders.sql @@ -0,0 +1 @@ +DROP TABLE IF EXISTS raw_orders; diff --git a/examples/vdk-lineage-example/20_create_raw_orders.sql b/examples/vdk-lineage-example/20_create_raw_orders.sql new file mode 100644 index 0000000000..2c43eba101 --- /dev/null +++ b/examples/vdk-lineage-example/20_create_raw_orders.sql @@ -0,0 +1,8 @@ +CREATE TABLE raw_orders ( + order_id INTEGER, + order_date TEXT, + customer_id INTEGER, + quantity INTEGER, + unit_price REAL, + status TEXT +); diff --git a/examples/vdk-lineage-example/30_insert_raw_orders.sql b/examples/vdk-lineage-example/30_insert_raw_orders.sql new file mode 100644 index 0000000000..877e5ebf4b --- /dev/null +++ b/examples/vdk-lineage-example/30_insert_raw_orders.sql @@ -0,0 +1,5 @@ +INSERT INTO raw_orders VALUES + (1, '2026-01-01', 101, 2, 19.99, 'PAID'), + (2, '2026-01-01', 102, 1, 49.00, 'PAID'), + (3, '2026-01-02', 101, 3, 15.50, 'CANCELLED'), + (4, '2026-01-02', 103, 4, 12.25, 'PAID'); diff --git a/examples/vdk-lineage-example/40_drop_daily_revenue.sql b/examples/vdk-lineage-example/40_drop_daily_revenue.sql new file mode 100644 index 0000000000..05609223e0 --- /dev/null +++ b/examples/vdk-lineage-example/40_drop_daily_revenue.sql @@ -0,0 +1 @@ +DROP TABLE IF EXISTS daily_revenue; diff --git a/examples/vdk-lineage-example/50_create_daily_revenue.sql b/examples/vdk-lineage-example/50_create_daily_revenue.sql new file mode 100644 index 0000000000..0efeff3ba6 --- /dev/null +++ b/examples/vdk-lineage-example/50_create_daily_revenue.sql @@ -0,0 +1,7 @@ +CREATE TABLE daily_revenue AS +SELECT + order_date, + SUM(quantity * unit_price) AS revenue +FROM raw_orders +WHERE status = 'PAID' +GROUP BY order_date; diff --git a/examples/vdk-lineage-example/60_read_daily_revenue.sql b/examples/vdk-lineage-example/60_read_daily_revenue.sql new file mode 100644 index 0000000000..ba7dd08836 --- /dev/null +++ b/examples/vdk-lineage-example/60_read_daily_revenue.sql @@ -0,0 +1 @@ +SELECT * FROM daily_revenue; diff --git a/examples/vdk-lineage-example/README.md b/examples/vdk-lineage-example/README.md new file mode 100644 index 0000000000..88c2aa3007 --- /dev/null +++ b/examples/vdk-lineage-example/README.md @@ -0,0 +1,155 @@ +# VDK Lineage Example + +This example shows how to collect data lineage from a local VDK data job and view it in Marquez. + +The data flow is: + +```text +raw_orders + | + | CREATE TABLE daily_revenue AS SELECT ... + v +daily_revenue +``` + +The `vdk-lineage` plugin sends OpenLineage events for: + +* The data job run start. +* SQL queries executed by the job, including input and output tables when they can be parsed. +* The data job run completion or failure. + +## Install + +Install VDK, the SQLite plugin, and the lineage plugin: + +```bash +pip install quickstart-vdk vdk-sqlite vdk-lineage +``` + +## Configure + +The data job configuration is in [config.ini](config.ini): + +```ini +[vdk] +db_default_type=SQLITE +sqlite_file=lineage-example.db +``` + +`db_default_type=SQLITE` tells VDK to execute SQL files against SQLite. `sqlite_file=lineage-example.db` names the local database file used by the job. + +Lineage export is configured with the `OPENLINEAGE_URL` VDK option. The easiest local setup is Marquez: + +```bash +vdk marquez-server --start +``` + +Then point `vdk-lineage` to the Marquez OpenLineage endpoint: + +```bash +export VDK_OPENLINEAGE_URL=http://localhost:5002 +``` + +On Windows PowerShell, use: + +```powershell +$env:VDK_OPENLINEAGE_URL = "http://localhost:5002" +``` + +## Run the Job + +From the repository `examples` directory, run: + +```bash +vdk run vdk-lineage-example +``` + +Because `sqlite_file` is a relative path, VDK creates `lineage-example.db` in the directory where you run the command. With the command above, that is the repository `examples` directory. + +The job performs these steps: + +* [10_drop_raw_orders.sql](10_drop_raw_orders.sql) removes the old source table if it exists. +* [20_create_raw_orders.sql](20_create_raw_orders.sql) creates the `raw_orders` source table. +* [30_insert_raw_orders.sql](30_insert_raw_orders.sql) loads sample orders. +* [40_drop_daily_revenue.sql](40_drop_daily_revenue.sql) removes the old output table if it exists. +* [50_create_daily_revenue.sql](50_create_daily_revenue.sql) reads `raw_orders` and creates the `daily_revenue` output table. +* [60_read_daily_revenue.sql](60_read_daily_revenue.sql) reads the result. + +You can verify the output table with: + +```bash +export VDK_SQLITE_FILE=lineage-example.db +vdk sqlite-query -q "SELECT * FROM daily_revenue" +``` + +On Windows PowerShell, use: + +```powershell +$env:VDK_SQLITE_FILE = "lineage-example.db" +vdk sqlite-query -q "SELECT * FROM daily_revenue" +``` + +## Show Lineage + +Open the Marquez UI: + +```text +http://localhost:3000 +``` + +Look for the `vdk-lineage-example` job under the `versatile_data_kit` namespace. + +The most useful lineage event in this example is generated by [50_create_daily_revenue.sql](50_create_daily_revenue.sql): + +```sql +CREATE TABLE daily_revenue AS +SELECT + order_date, + SUM(quantity * unit_price) AS revenue +FROM raw_orders +WHERE status = 'PAID' +GROUP BY order_date +``` + +That event shows: + +* Input dataset: `raw_orders` +* Job: `vdk-lineage-example` +* Output dataset: `daily_revenue` +* SQL facet: the query that created the output table + +## Debug With Lineage + +To see how lineage helps during debugging, introduce a typo in [50_create_daily_revenue.sql](50_create_daily_revenue.sql): + +```sql +FROM raw_order +``` + +Run the job again: + +```bash +vdk run vdk-lineage-example +``` + +The job fails because `raw_order` does not exist. In Marquez, the job run is marked as failed, and the SQL facet for the emitted query shows the statement that attempted to read `raw_order`. + +Use that information together with the VDK console error to identify the broken transformation step: + +* The failed job is `vdk-lineage-example`. +* The failing transformation is the `CREATE TABLE daily_revenue AS SELECT ...` query. +* The bad input table reference is `raw_order`; the expected source table is `raw_orders`. + +After testing the failure path, change `raw_order` back to `raw_orders`. + +For a code regression in a deployed job, use the failed lineage run to identify the affected job and version, then redeploy a known-good version with `vdk deploy --job-version`. For a data drift issue, use the lineage graph to find which upstream dataset or job feeds the broken output, then update the job that owns that upstream change. + +## Stop Marquez + +When finished, stop the local Marquez server: + +```bash +vdk marquez-server --stop +``` + +Stopping the server removes the local Marquez containers and deletes the recorded lineage data. diff --git a/examples/vdk-lineage-example/config.ini b/examples/vdk-lineage-example/config.ini new file mode 100644 index 0000000000..50797e1676 --- /dev/null +++ b/examples/vdk-lineage-example/config.ini @@ -0,0 +1,9 @@ +[owner] +team = examples + +[job] +schedule_cron = 0 0 * * * + +[vdk] +db_default_type=SQLITE +sqlite_file=lineage-example.db diff --git a/projects/vdk-plugins/vdk-lineage/README.md b/projects/vdk-plugins/vdk-lineage/README.md index e497a73309..10f05193a4 100644 --- a/projects/vdk-plugins/vdk-lineage/README.md +++ b/projects/vdk-plugins/vdk-lineage/README.md @@ -42,6 +42,9 @@ vdk run some-job vdk marquez-server --stop ``` +For a complete runnable scenario with a SQLite data job, Marquez setup, lineage inspection, and a debugging walkthrough, +see the [VDK Lineage example](../../../examples/vdk-lineage-example/). + ## Build and testing In order to build and test a plugin go to the plugin directory and use `../build-plugin.sh` script to build it