Skip to content

user-guide.md

Latest commit

 

History

History
453 lines (324 loc) · 18 KB

user-guide.md

File metadata and controls

453 lines (324 loc) · 18 KB
Raw
doc_type guide
title DEKC User Guide
slug user-guide
wiki_key guide/user-guide
truth_state current

DEKC user guide

Data Engineering Knowledge Capture turns data platforms into a durable, Git-native OKF knowledge graph, with multi-agent walk loops designed using AGER (OKF Agent Graph Engineering Runtime).

Plugin release 0.5.3. Storage format is OKF 0.2. Agent loops follow AGER 0.3 roles (orchestrator / worker / judge / synthesizer) even when you run DEKC skills without a separate AGER bundle. Search and pack use a disposable retrieval ladder (SQLite index → ripgrep → scan); Git + Markdown stays source of truth.

Existing second brains: noun-ownership migration (Workflow jobs → IngestionJob; diagrams stay SAC).

Who this is for

  • Analytics / platform engineers reverse-engineering a lakehouse
  • Agents that must answer “what is this table?” with business meaning
  • Teams that already use PKC for decisions and want the same graph for data

Prerequisites

Dependency Why
okf-plugin (okf-graph-eng) Validate links, impact, packs, visualization
okf-agent-graph (recommended) Author explicit LoopPolicy / FanOut graphs for walks
project-knowledge-capture (recommended) Meetings, ADRs, features linked to data decisions
Python 3.10+ Deterministic DEKC scripts (stdlib only)

Install

Claude Code

claude plugin marketplace add SpillwaveSolutions/okf-plugin
claude plugin install okf-graph-eng@okf-plugin-marketplace

claude plugin marketplace add SpillwaveSolutions/okf-agent-graph
claude plugin install okf-agent-graph@okf-agent-graph-marketplace

claude plugin marketplace add SpillwaveSolutions/project-knowledge-capture
claude plugin install project-knowledge-capture@pkc-plugin-marketplace

claude plugin marketplace add SpillwaveSolutions/data-engineering-knowledge-capture
claude plugin install data-engineering-knowledge-capture@dekc-plugin-marketplace

Start a new session. DEKC exposes /dekc-* commands; AGER exposes /ager-*.

Grok Build

Load the same Claude marketplaces — zero-config Claude plugin compatibility.

Codex

codex plugin marketplace add SpillwaveSolutions/data-engineering-knowledge-capture
codex
# /plugins → install data-engineering-knowledge-capture

Skills are also invokable as $dekc-init, $dekc-walk, etc.

OpenCode

Use skills under skills/ plus AGENTS.md. Register permissions via .opencode/opencode.json.

Diagrams, wireframes, lakes, catalogs, DQ

Wireframes (reports/dashboards)

python3 scripts/dekc_diagram.py wireframe \
  --name "Exec Revenue Wireframe" \
  --subject /dashboards/executive-revenue.md \
  --language plantuml

Architecture / ERD / job diagrams (Mermaid or PlantUML)

python3 scripts/dekc_diagram.py capture --name "Lake arch" --kind architecture --language mermaid --subject /lakes/...
python3 scripts/dekc_diagram.py capture --name "Orders ERD" --kind erd --language mermaid --subject /tables/...
python3 scripts/dekc_diagram.py job-pack --workflow /workflows/... --language mermaid

Kinds: wireframe, architecture, component, activity, state, class, erd, sequence, deployment, flowchart, c4.

Platform concepts

python3 scripts/dekc_platform.py lake|mart|catalog|domain|product|stream|storage|dq-rule|ingestion ...

# Landing / ingestion jobs (stream, file, CDC, API)
python3 scripts/dekc_platform.py ingestion \
  --name orders-stream-landing --mode streaming \
  --streams orders-events --lands-as bronze-orders-raw
python3 scripts/dekc_diagram.py ingestion-pack --job /ingestion/orders-stream-landing.md

See ingestion-jobs.md.

Full guide: diagrams.md.

Second brain for design work

The knowledge bundle is an indexed, schema-typed second brain. Use it before designing reports or landing data so agents reuse existing gold tables, metrics, glossary terms, sources, and jobs.

Standard OKF schemas

Concept frontmatter must match schemas/okf-concepts/. List and validate:

python3 scripts/dekc_schemas.py list
python3 scripts/dekc_schemas.py intents
python3 scripts/dekc_schemas.py validate --repo . --bundle knowledge

Intent queries

python3 scripts/dekc_brain.py "executive revenue" --intent design-report --repo . --bundle knowledge --write
python3 scripts/dekc_brain.py "orders event hub" --intent land-data --repo . --bundle knowledge --write
python3 scripts/dekc_brain.py "GMV" --intent design-metric --repo . --bundle knowledge
python3 scripts/dekc_brain.py "silver.orders" --intent impact --repo . --bundle knowledge

Each intent returns: design checklist, ranked schema-typed concepts, progressive disclosure pack, lineage snippet.

For Q&A, spawn data-retriever (/dekc-retrieve) instead of dumping a brain pack into the parent.

Skills: dekc-plan, dekc-retrieve, dekc-second-brain, dekc-design-report, dekc-land-data.

Patterns: patterns/design-report-from-gold.md, patterns/land-stream-to-bronze.md.

First 15 minutes

1. Scaffold a knowledge bundle

python3 "${CLAUDE_PLUGIN_ROOT}/scripts/dekc_common.py" init-bundle \
  --repo . --bundle knowledge --title "Platform Knowledge"

Or invoke /dekc-init.

Creates catalogs (tables, layers, lineage, glossary, …), seeds bronze/silver/gold Layer concepts, and writes index.md + log.md.

Optional config:

cp "${CLAUDE_PLUGIN_ROOT}/.dekc/config.example.yml" .dekc/config.yml
# edit knowledge_root, walk.max_files, promote.default_layer

2. Plan, then walk a filesystem export of your platform

Point the planner at SQL models, dbt, notebooks export, or a local mirror of lake paths. Review the ranked areas, then walk (or --from-plan --area):

python3 "${CLAUDE_PLUGIN_ROOT}/scripts/dekc_orchestrate.py" \
  --repo . --system "Retail Lake" --scan-root ./lake-mirror --plan-only
# review knowledge/.dekc/re-plan.md — spawn only listed specialists
python3 "${CLAUDE_PLUGIN_ROOT}/scripts/dekc_walk.py" ./lake-mirror \
  --repo . --bundle knowledge --source-name retail-lake

Or /dekc-plan then /dekc-walk with path arguments. The walker:

  1. Registers a SourceSystem
  2. Discovers *.sql, *.dax, parquet directories, medallion folder names
  3. Captures Query / Table / View concepts
  4. Infers bronze→silver→gold Transformation promotions when basenames align
  5. Writes an AgentNode walk receipt

3. Materialize lineage and business meaning

python3 "${CLAUDE_PLUGIN_ROOT}/scripts/dekc_lineage.py" --repo . --bundle knowledge materialize
python3 "${CLAUDE_PLUGIN_ROOT}/scripts/dekc_business.py" --repo . --bundle knowledge \
  promote-layer --layer gold

Promote a single asset:

python3 "${CLAUDE_PLUGIN_ROOT}/scripts/dekc_business.py" --repo . --bundle knowledge promote \
  tables/gold-order-daily.md \
  --name "Daily Order Summary" \
  --definition "Business-facing daily GMV and order volume by currency." \
  --owner "Analytics Engineering"

4. Index the second brain

Git + Markdown is the source of truth. Search and pack refresh a disposable SQLite index themselves (mtime+size). You do not need to run this first.

python3 "${CLAUDE_PLUGIN_ROOT}/scripts/dekc_index.py" status --repo . --bundle knowledge
python3 "${CLAUDE_PLUGIN_ROOT}/scripts/dekc_search.py" "GMV" --repo . --bundle knowledge
python3 "${CLAUDE_PLUGIN_ROOT}/scripts/dekc_doctor.py" --repo . --bundle knowledge

Index layout:

knowledge/.dekc/index.sqlite   # gitignored; delete anytime

build is an alias for refresh --force. --no-index / DEKC_NO_INDEX=1 fall through to ripgrep then a full scan. --engine fts opts into FTS5 MATCH (not score-identical). See retrieval ladder.

5. Progressive disclosure packs

When an agent works on one table:

python3 "${CLAUDE_PLUGIN_ROOT}/scripts/dekc_pack.py" tables/gold-order-daily.md \
  --repo . --bundle knowledge --hops 2
# tiny pack for chat focus
python3 "${CLAUDE_PLUGIN_ROOT}/scripts/dekc_pack.py" tables/gold-order-daily.md \
  --repo . --bundle knowledge --tiny
# card-friendly summary (bodies off) — prefer this from data-retriever
python3 "${CLAUDE_PLUGIN_ROOT}/scripts/dekc_pack.py" tables/gold-order-daily.md \
  --repo . --bundle knowledge --tiny --summary

If okf-graph-eng is available, prefer its pack/impact for the same paths.

Skills & commands

Skill / command Purpose
dekc-init Scaffold knowledge catalogs + medallion layers
dekc-plan Breadth-first RE plan + checklists (pause before fan-out)
dekc-walk Filesystem / SQL root discovery
dekc-capture-table Manual table + columns
dekc-capture-query SQL or DAX query
dekc-lineage Graph, mermaid, materialize paths
dekc-business-object Promote technical → business + glossary
dekc-glossary Author glossary terms
dekc-semantic Semantic models, metrics, dashboards
dekc-retrieve Query-time retrieval — spawn data-retriever; card only
dekc-context Context packs (retriever internals / known seed)
dekc-search Second-brain search (raw hits; not parent Q&A)
dekc-index Rebuild index
dekc-grade Rubric grade + adversarial RE protocol
dekc-doctor Health: coverage, orphans, validation

Agents (how to invoke)

Agent Use when
data-retriever Query-time Q&A — spawn via /dekc-retrieve; parent keeps a card only
data-lake-walker Full reverse-engineer loop (orchestrator): plan → fan-out → judge
schema-scout Only structure (schemas/tables/columns)
lineage-tracer Only SQL/job lineage and promotions
semantic-mapper Business objects, glossary, metrics
report-cataloger Dashboards, reports, DAX
airflow-scout / glue-job-scout / fabric-pipeline-scout Signal-gated orchestration (plan must list them)
dbt-elt-scout / duckdb-scout / notebook-scout / dq-scout Signal-gated ELT / DQ
layer-auditor Judge medallion health and orphans

In Claude/Grok, ask for the agent by name or describe the walk (“walk this lake and promote gold tables”). In AGER terms, the walker is an OrchestratorAgent; scouts are WorkerAgents; the auditor is a JudgeAgent.

See the design doc for the full loop graph, LoopPolicy, ScratchPad keys, and cloud playbooks.

Adversarial grading (reverse engineering quality)

After producers walk a lake or cloud mirror, do not claim complete until graded.

Automated baseline

python3 "${CLAUDE_PLUGIN_ROOT}/scripts/dekc_grade.py" --repo . --bundle knowledge
python3 "${CLAUDE_PLUGIN_ROOT}/scripts/dekc_grade.py" --repo . --bundle knowledge --json --write

Exit code 0 ≈ automated pass (score ≥ 0.75, no hard fails). Still run LLM skeptics for lineage evidence and definition quality.

Adversarial agents

Agent Rubric
lineage-skeptic lineage integrity (0.80)
business-skeptic business fidelity (0.72)
stream-job-skeptic stream/job landing (0.70)
coverage-skeptic structural coverage
re-adversary-judge aggregate reverse-engineering (0.75)

Orchestrators: data-lake-walker, reverse-engineering-orchestrator.

On fail: fix evidence or retract edges — never invent data to pass. See design doc § adversarial.

Capturing streams and jobs

Landing is not only batch SQL. Capture continuous and scheduled producers explicitly:

Streams (landing)

# Conceptual capture today: source + transformation + table + lineage path
python3 scripts/dekc_capture.py --repo . --bundle knowledge source \
  --name "orders-eventhub" --kind stream \
  --uri "endpoints://eventhubs/.../orders" \
  --description "Commerce order events stream"

python3 scripts/dekc_capture.py --repo . --bundle knowledge table \
  --name orders_raw --layer bronze --schema bronze \
  --source orders-eventhub \
  --description "Landing table for order stream"

python3 scripts/dekc_capture.py --repo . --bundle knowledge lineage \
  --name stream-orders-landing \
  --nodes sources/orders-eventhub tables/bronze-orders-raw \
  --description "Stream lands as bronze table"

Typical stream sources by cloud:

Cloud Stream services Lands as
Azure Event Hubs, IoT Hub, Fabric Eventstream Bronze Lakehouse / Eventhouse tables
AWS Kinesis Data Streams, MSK (Kafka), Firehose S3 raw prefixes, Iceberg/Glue tables
GCP Pub/Sub, Dataflow streaming BigQuery landing, GCS raw, Bigtable

Jobs (batch & micro-batch)

python3 scripts/dekc_capture.py --repo . --bundle knowledge workflow \
  --name "nightly-silver-orders" --orchestrator airflow \
  --description "Promotes bronze.orders_raw → silver.orders" \
  --steps "Extract bronze" "Cleanse" "Write silver" "DQ checks"

python3 scripts/dekc_capture.py --repo . --bundle knowledge transformation \
  --name cleanse-orders --from-layer bronze --to-layer silver \
  --inputs bronze-orders-raw --outputs silver-orders \
  --description "Nightly cleanse job body"
Cloud Job / pipeline services
Azure Fabric Data Pipelines, Data Factory, Synapse pipelines, Fabric notebooks
AWS Glue jobs, EMR, Step Functions, MWAA (Airflow), Lambda micro-batch
GCP Dataflow batch, Cloud Composer, Dataproc, Cloud Scheduler + Functions

Tag concepts with tags: [stream] or tags: [job] and link with feeds / writes_to / triggered_by (AGER Trigger when using an agent-graph bundle).

Multi-cloud reverse engineering (practical recipes)

Full architecture and agent graph: design doc § cloud reverse engineering.

Azure Fabric (typical)

  1. Export or mirror: Lakehouse tables, Warehouse SQL, Pipelines JSON, Notebooks, Semantic models (.bim / Power BI), Eventstreams definitions.
  2. /dekc-walk on the mirror root (prefer folders named bronze/silver/gold or lh_*).
  3. Map Eventstream / Event Hub → SourceSystem kind: stream → bronze tables.
  4. Map Data Pipeline / Notebook jobs → Workflow + Transformation.
  5. Map Semantic model + reports → SemanticModel / Metric / Dashboard (DAX via dekc-capture-query --dialect dax).
  6. Promote gold Lakehouse / Warehouse tables → business objects.

AWS (typical)

  1. Mirror: Glue Data Catalog export (or Athena DDL), S3 prefixes (s3://bucket/bronze|silver|gold/), Glue job scripts, MWAA DAGs, Kinesis/Firehose configs, QuickSight assets if available.
  2. Walk SQL + job scripts; register S3/Glue as SourceSystem.
  3. Firehose/Kinesis → bronze landing prefixes as stream lineage.
  4. Glue/EMR/Step Functions → Workflow + Transformation edges.
  5. Athena/Redshift views → View/Query; promote curated marts.

GCP (typical)

  1. Mirror: BigQuery dataset DDL, Dataform/dbt SQL, Dataflow templates, Pub/Sub topic configs, Composer DAGs, Looker/Looker Studio exports if present.
  2. Walk SQL roots; register project.dataset as schemas.
  3. Pub/Sub → Dataflow streaming → BQ landing as stream lineage.
  4. Scheduled queries / Dataform assertions → Workflow + DQ-oriented Transformations.
  5. Promote mart datasets (*_mart, gold) to business objects.

Linking to PKC decisions

When a data design decision is made in a meeting:

  1. Capture with PKC (/pkc-capture-meeting or /pkc-capture-decision).
  2. Link the DecisionRecord to a DEKC table or business object:
python3 scripts/dekc_link.py decisions/use-delta-lake.md \
  /tables/silver-orders.md --rel decides --repo .

(If the decision lives in a PKC bundle, federate paths or colocate catalogs under one knowledge root.)

Validation & doctor

python3 scripts/dekc_validate.py --bundle knowledge
python3 scripts/dekc_doctor.py --bundle knowledge --json

Doctor reports concept counts, lineage edge count, business coverage (tables→BO ratio), orphan technical assets, and whether the second-brain index exists.

When okf-graph-eng is installed:

python3 path/to/okf-plugin/scripts/okf-graph.py validate knowledge --strict
python3 path/to/okf-plugin/scripts/okf-graph.py impact knowledge tables/gold-order-daily.md

Sample bundle

python3 tests/test_dekc.py
python3 scripts/dekc_validate.py --bundle sample-knowledge
python3 scripts/dekc_search.py revenue --repo . --bundle sample-knowledge

Explorer UI (this repo): npm run dev → open the live preview catalog tabs.

CLI cheat sheet

Script Role
dekc_common.py init-bundle Scaffold
dekc_walk.py <path> Discover
dekc_capture.py <subcommand> source/table/view/query/lineage/…
dekc_lineage.py graph / upstream / mermaid / materialize
dekc_business.py promote / promote-layer
dekc_index.py status / refresh / drop / build
dekc_pack.py context packs
dekc_search.py full-text + index
dekc_validate.py / dekc_doctor.py quality
dekc_link.py typed edges

Global flags (parent before subcommand): --repo, --bundle, often --json.

Safety

  • Capture scripts scrub common secrets and PII patterns.
  • Never commit credentials; use AGER SecretRef patterns for runtime tools.
  • Prefer readonly federation when linking remote knowledge roots.

Next reading