examples: Add vdk-lineage usage example

examples/README.md

@@ -13,6 +13,7 @@ Tutorials to get started with generic ingestion tasks in VDK:
 
 Tutorials to get started with generic processing tasks in VDK:
 * [Sqlite Processing](sqlite-processing-example/)
+* [VDK Lineage](vdk-lineage-example/)
 
 ## Advanced Examples
 Tutorials with a complete pipeline from ingestion to final publication:

examples/vdk-lineage-example/10_drop_raw_orders.sql

@@ -0,0 +1 @@
+DROP TABLE IF EXISTS raw_orders;

examples/vdk-lineage-example/20_create_raw_orders.sql

@@ -0,0 +1,8 @@
+CREATE TABLE raw_orders (
+    order_id INTEGER,
+    order_date TEXT,
+    customer_id INTEGER,
+    quantity INTEGER,
+    unit_price REAL,
+    status TEXT
+);

examples/vdk-lineage-example/30_insert_raw_orders.sql

@@ -0,0 +1,5 @@
+INSERT INTO raw_orders VALUES
+    (1, '2026-01-01', 101, 2, 19.99, 'PAID'),
+    (2, '2026-01-01', 102, 1, 49.00, 'PAID'),
+    (3, '2026-01-02', 101, 3, 15.50, 'CANCELLED'),
+    (4, '2026-01-02', 103, 4, 12.25, 'PAID');

examples/vdk-lineage-example/40_drop_daily_revenue.sql

@@ -0,0 +1 @@
+DROP TABLE IF EXISTS daily_revenue;

examples/vdk-lineage-example/50_create_daily_revenue.sql

@@ -0,0 +1,7 @@
+CREATE TABLE daily_revenue AS
+SELECT
+    order_date,
+    SUM(quantity * unit_price) AS revenue
+FROM raw_orders
+WHERE status = 'PAID'
+GROUP BY order_date;

examples/vdk-lineage-example/60_read_daily_revenue.sql

@@ -0,0 +1 @@
+SELECT * FROM daily_revenue;

examples/vdk-lineage-example/README.md

@@ -0,0 +1,155 @@
+# VDK Lineage Example
+
+This example shows how to collect data lineage from a local VDK data job and view it in Marquez.
+
+The data flow is:
+
+```text
+raw_orders
+    |
+    | CREATE TABLE daily_revenue AS SELECT ...
+    v
+daily_revenue
+```
+
+The `vdk-lineage` plugin sends OpenLineage events for:
+
+* The data job run start.
+* SQL queries executed by the job, including input and output tables when they can be parsed.
+* The data job run completion or failure.
+
+## Install
+
+Install VDK, the SQLite plugin, and the lineage plugin:
+
+```bash
+pip install quickstart-vdk vdk-sqlite vdk-lineage
+```
+
+## Configure
+
+The data job configuration is in [config.ini](config.ini):
+
+```ini
+[vdk]
+db_default_type=SQLITE
+sqlite_file=lineage-example.db
+```
+
+`db_default_type=SQLITE` tells VDK to execute SQL files against SQLite. `sqlite_file=lineage-example.db` names the local database file used by the job.
+
+Lineage export is configured with the `OPENLINEAGE_URL` VDK option. The easiest local setup is Marquez:
+
+```bash
+vdk marquez-server --start
+```
+
+Then point `vdk-lineage` to the Marquez OpenLineage endpoint:
+
+```bash
+export VDK_OPENLINEAGE_URL=http://localhost:5002
+```
+
+On Windows PowerShell, use:
+
+```powershell
+$env:VDK_OPENLINEAGE_URL = "http://localhost:5002"
+```
+
+## Run the Job
+
+From the repository `examples` directory, run:
+
+```bash
+vdk run vdk-lineage-example
+```
+
+Because `sqlite_file` is a relative path, VDK creates `lineage-example.db` in the directory where you run the command. With the command above, that is the repository `examples` directory.
+
+The job performs these steps:
+
+* [10_drop_raw_orders.sql](10_drop_raw_orders.sql) removes the old source table if it exists.
+* [20_create_raw_orders.sql](20_create_raw_orders.sql) creates the `raw_orders` source table.
+* [30_insert_raw_orders.sql](30_insert_raw_orders.sql) loads sample orders.
+* [40_drop_daily_revenue.sql](40_drop_daily_revenue.sql) removes the old output table if it exists.
+* [50_create_daily_revenue.sql](50_create_daily_revenue.sql) reads `raw_orders` and creates the `daily_revenue` output table.
+* [60_read_daily_revenue.sql](60_read_daily_revenue.sql) reads the result.
+
+You can verify the output table with:
+
+```bash
+export VDK_SQLITE_FILE=lineage-example.db
+vdk sqlite-query -q "SELECT * FROM daily_revenue"
+```
+
+On Windows PowerShell, use:
+
+```powershell
+$env:VDK_SQLITE_FILE = "lineage-example.db"
+vdk sqlite-query -q "SELECT * FROM daily_revenue"
+```
+
+## Show Lineage
+
+Open the Marquez UI:
+
+```text
+http://localhost:3000
+```
+
+Look for the `vdk-lineage-example` job under the `versatile_data_kit` namespace.
+
+The most useful lineage event in this example is generated by [50_create_daily_revenue.sql](50_create_daily_revenue.sql):
+
+```sql
+CREATE TABLE daily_revenue AS
+SELECT
+    order_date,
+    SUM(quantity * unit_price) AS revenue
+FROM raw_orders
+WHERE status = 'PAID'
+GROUP BY order_date
+```
+
+That event shows:
+
+* Input dataset: `raw_orders`
+* Job: `vdk-lineage-example`
+* Output dataset: `daily_revenue`
+* SQL facet: the query that created the output table
+
+## Debug With Lineage
+
+To see how lineage helps during debugging, introduce a typo in [50_create_daily_revenue.sql](50_create_daily_revenue.sql):
+
+```sql
+FROM raw_order
+```
+
+Run the job again:
+
+```bash
+vdk run vdk-lineage-example
+```
+
+The job fails because `raw_order` does not exist. In Marquez, the job run is marked as failed, and the SQL facet for the emitted query shows the statement that attempted to read `raw_order`.
+
+Use that information together with the VDK console error to identify the broken transformation step:
+
+* The failed job is `vdk-lineage-example`.
+* The failing transformation is the `CREATE TABLE daily_revenue AS SELECT ...` query.
+* The bad input table reference is `raw_order`; the expected source table is `raw_orders`.
+
+After testing the failure path, change `raw_order` back to `raw_orders`.
+
+For a code regression in a deployed job, use the failed lineage run to identify the affected job and version, then redeploy a known-good version with `vdk deploy --job-version`. For a data drift issue, use the lineage graph to find which upstream dataset or job feeds the broken output, then update the job that owns that upstream change.
+
+## Stop Marquez
+
+When finished, stop the local Marquez server:
+
+```bash
+vdk marquez-server --stop
+```
+
+Stopping the server removes the local Marquez containers and deletes the recorded lineage data.

examples/vdk-lineage-example/config.ini

@@ -0,0 +1,9 @@
+[owner]
+team = examples
+
+[job]
+schedule_cron = 0 0 * * *
+
+[vdk]
+db_default_type=SQLITE
+sqlite_file=lineage-example.db

projects/vdk-plugins/vdk-lineage/README.md

@@ -42,6 +42,9 @@ vdk run some-job
 vdk marquez-server --stop
 ```
 
+For a complete runnable scenario with a SQLite data job, Marquez setup, lineage inspection, and a debugging walkthrough,
+see the [VDK Lineage example](../../../examples/vdk-lineage-example/).
+
 ## Build and testing
 
 In order to build and test a plugin go to the plugin directory and use `../build-plugin.sh` script to build it