TFF: Transformation Fitness Functions

Configurable fitness functions engine and linter for transformation projects.

TFF allows you to enforce architectural layout boundaries, layer structure policies, schema contracts, and code formatting rules across data pipelines. It ships with dedicated plugins for SQLMesh, dbt, and Google Cloud Dataform, supports custom adapters and proprietary rules via entry points and plugins, and outputs clean, color-coded lint reports to the terminal.

More screenshots

tff lint

tff info

CTE fingerprinting demo

---

Documentation

Setup and usage details differ depending on your pipeline engine. Refer to the corresponding guide:

• 📐 SQLMesh Integration: See docs/sqlmesh.md
• ⚡ dbt Integration: See docs/dbt.md
• ☁️ Dataform Integration: See docs/dataform.md
• 🔌 Custom Plugins & Adapters: See docs/rules_and_checks.md#3-custom-plugins--extensions
• 💻 CLI Reference Guide: See docs/cli.md
• 🤖 CI/CD & GitHub Actions Guide: See docs/ci_cd.md
• 🔍 Rules & Checks Reference: See docs/rules_and_checks.md
• 📊 Case Study: GitLab dbt Audit (2,200+ models): See docs/case_study_gitlab.md
• 🏗️ Architecture & Contributor Guide: See docs/contributing.md

---

Quick Installation

Install the package with the adapter matching your pipeline tool:

📐 For SQLMesh projects:

# With uv:
uv add "tff-core[sqlmesh]"

# Or pip:
pip install "tff-core[sqlmesh]"

⚡ For dbt projects:

# With uv:
uv add "tff-core[dbt]"

# Or pip:
pip install "tff-core[dbt]"

☁️ For Dataform projects:

# With uv:
uv add "tff-core[dataform]"
# (or simply: uv add tff-core)

# Or pip:
pip install "tff-core[dataform]"

---

CLI Usage Guide

Once installed, use the unified tff CLI to run linting, calculate health scores, and enforce architectural quality gates:

tff [command] [options]

Commands Overview

Command	Description	Quick Example
lint	Run architectural fitness checks and output lint reports	tff lint --fix
health	Calculate overall project fitness health score (0–100)	tff health --fail-under 80
action	Run official GitHub Action pipeline (score, diff, PR comments)	tff action --only-changed
docs	Generate standalone interactive HTML dashboard with lineage graphs	tff docs --output docs/index.html
init	Scaffold an annotated starter fitness_functions.yaml file	tff init
stats	View historical fitness check execution trends and logs	tff stats --days 30
info	Inspect project environment, config, and adapter versions	tff info
help	Show detailed help and options for any command	tff help lint

Quick Start Examples

# Lint current project (zero-config out of the box)
tff lint

# Automatically fix simple linting violations
tff lint --fix

# Require an 80% health score to pass CI
tff health --fail-under 80

# Restrict health scoring to a specific domain
tff health --scope models/marts/marketing

# Generate interactive HTML dashboard
tff docs

# Export SARIF for GitHub Code Scanning
tff lint --format sarif > results.sarif

# Accelerate large DAGs with parallel worker pool and persistent AST caching
tff lint --workers 8

# Run with verbose debug logging to inspect internal operations
tff lint --debug

👉 For the complete CLI reference, detailed option tables for every command, output formats, and cookbooks, see the CLI Reference Guide.

---

CI/CD & Automated Quality Gates

TFF integrates seamlessly into modern data engineering CI/CD pipelines to enforce architectural fitness functions, calculate health scores, and gate pull requests.

Official GitHub Action (tjirab/tff@v1)

Run TFF on pull requests with zero virtualenv setup. The action automatically installs the required engine adapter, gates merges based on health thresholds, emits inline annotations on modified lines, and posts interactive summary comments:

name: TFF Architectural Fitness Functions

on:
  pull_request:
    branches: [ main ]

jobs:
  tff-check:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pull-requests: write  # Required for posting/updating PR summary comments
    steps:
      - name: Checkout Code
        uses: actions/checkout@v4
        with:
          fetch-depth: 0  # Required to compute health score diff vs base branch

      - name: Run TFF Action
        uses: tjirab/tff@v1
        with:
          provider: "auto"       # auto, dbt, sqlmesh, or dataform
          fail-under: "80.0"     # Minimum health score to pass (0-100)
          fail-level: "error"    # Failure severity level (error, warning)
          only-changed: "true"   # 👈 Only gate models modified in this PR
          comment-pr: "true"     # Post/update PR health summary comment

👉 For the complete guide, full input/output reference tables, GitLab CI, and SARIF exports, see the CI/CD & GitHub Actions Guide.

---

Pre-commit Integration

TFF includes native pre-commit hooks to validate or auto-fix violations locally before commits are created:

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/tjirab/tff
    rev: v0.12.1
    hooks:
      - id: tff-lint
      # Or automatically fix simple violations:
      # - id: tff-lint-fix

For SQLMesh projects or advanced pre-commit dependency configurations, refer to the CI/CD Documentation.

---

Core Features

TFF runs two categories of quality guardrails (for full configuration details, see the Rules & Checks Reference):

1. Architectural Checks

• Layer integrity: Prevent models in upstream layers (e.g. marts) from depending on downstream/raw layers.
• Custom exclusions: Enforce custom domain isolation boundaries (e.g., prevent marts/finance from depending on marts/marketing).
• Schema contracts: Ensure matching structures between model schemas (e.g., source tables and target core columns).
• Dependency graph: Track DAG metrics and fail if model fan-in or fan-out exceeds defined thresholds.
• Materialization depth: Prevent deep nesting of views that degrades query performance.
• Duplicate CTEs: Detect duplicate complex transformation logic in CTEs across different models (Connascence of Algorithm).
• Connascence of Value: Identify duplicated domain-meaning literal values (strings, numbers) across multiple models (Connascence of Value).

2. Linter Rules

• Ban SELECT *: Require explicit columns to reduce upstream coupling.
• No positional GROUP BY/ORDER BY: Prevent using ordinal indexes (e.g., GROUP BY 1, 2) in queries.
• Classification macros: Require using standardized macros instead of inline CASE statements for classification fields.
• Sql complexity: Limits CTE count, join count, decision points, and line count in SQL.
• Mart naming: Ensure model filenames match their subfolder namespaces.
• Column names: Avoid deprecated or forbidden patterns in column names.
• Column types: Enforce expected types for matching column name patterns.
• Metadata checks: Enforce owners, descriptions, grains, unique assertions, and non-null constraints on models.
• Filename equals model name: Flags model name mismatch.
• Environment agnostic references: Ban hardcoded environment/catalog prefixes in queries.

---

Shared Configuration

All adapters use a shared fitness_functions.yaml config file located in the root of your project:

# Schema contracts and custom exclusions can be configured directly in YAML
# (or loaded from external JSON files via contract_groups_path / exclusions_path)
exclusions:
  - source_layer: core
    target_layer: derived

contract_groups:
  column_parity_groups:
    - reference: models/core/dim_customer_ref.sql
      members: [models/core/dim_customer_replica.sql]

layers:
  order: [staging, core, marts]  # Configured bottom-to-top hierarchy

checks:
  layer_integrity: { enabled: true }
  custom_exclusions: { enabled: true }
  schema_contracts: { enabled: true }
  dependency_graph:
    enabled: true
    fan_out_warn: 15
    fan_out_fail: 25
    fan_in_warn: 10
  duplicate_ctes:
    enabled: true
    severity: warning
    min_ast_nodes: 12
  connascence_of_value:
    enabled: true
    severity: warning
    min_occurrences: 2

rules:
  ban_select_star:
    enabled: true
  no_positional_group_by_or_order_by:
    enabled: true
  environment_agnostic_references:
    enabled: true
    banned_environments: [prod, dev, staging, uat, qa]
  classification_macros:
    enabled: true
    skip_layers: [staging]
    columns:
      product_type: "@product_type\\b"
  sql_complexity:
    enabled: true
    thresholds:
      decision_points: [15, 25]
      cte_count: [8, 12]
      join_count: [8, 12]
      line_count: [250, 400]
  mart_naming:
    enabled: true
    layer_name: marts
    rule: prefix_with_subdirectory
  column_names:
    enabled: true
    replacements:
      api_request: api_call
  column_types:
    enabled: true
    rules:
      - name: id_is_text
        pattern: "_id$"
        data_type: text
  metadata:
    owner: true
    description: true
    grain: true
    unique_values: true
    not_null: true
  filename_equals_modelname:
    enabled: true

# Configurable health scoring weights and failure penalties
health:
  weights:
    layer_integrity: 3.0
    schema_contracts: 2.0
    column_names: 0.5
  penalties:
    error: 1.0
    warning: 0.5
    project_error: 100.0
    project_warning: 50.0

---

Further Reading & Learning Resources

To learn more about the architectural concepts behind fitness functions and connascence, check out these resources:

• Connascence.io — A guide to software coupling metrics (connascence of name, type, meaning, algorithm, etc.), which inspired the classification and structure of the linter report findings.
• Evolutionary Architecture — The homepage for Building Evolutionary Architectures, which introduces the concept of architectural fitness functions to guide design changes over time.