TestQL — Interface Query Language for Testing

AI Cost Tracking

🤖 LLM usage: $7.5000 (113 commits)
👤 Human dev: ~$6230 (62.3h @ $100/h, 30min dedup)

Generated on 2026-06-09 using openrouter/qwen/qwen3-coder-next

TestQL is a declarative DSL (Domain Specific Language) for testing GUI, REST API, and hardware encoder interfaces. It provides a simple, readable syntax for writing automated tests without programming overhead.

What's new in 1.2.x

Native Desktop E2E: DESKTOP_* commands with vdisplay mirror→Xvfb capture, img2nl + imgl OCR on Linux (no Wayland portal)
Artifact Discovery & Topology: Complete codebase inspection with manifest generation
Web Inspection: Live URL scanning with Playwright browser automation
Multi-DSL Platform: TestTOON, NL, SQL, Proto, GraphQL adapters with Unified IR
Autoloop Integration: MCP-driven development with autonomous test loops
Refactor Plans: Structured refactoring plans with NLP summaries
Production deployment in c2004: 50-assertion realtime-health scenario running every 60s as part of Prometheus + planfile self-healing pipeline

Recent Improvements (1.2.48)

c2004 Healing Pipeline: testql-watchdog container, Prometheus metrics, LLM-ready ticket generation on failure
MCP Server: testql/mcp/server.py — resources for topology, tools for discover/run/explain
Internal IR: testql/ir/ module with steps, assertions, captures, fixtures, plan
TestTOON Adapter: full round-tripping with validation layer
Configuration Management: refactored config system
Code Analysis Engine: refactored analysis pipeline
Markdown Output: --output markdown for human-readable reports
Artifact Discovery: testql discover with topology graph generation
Web Inspection: --scan-network with headless browser capture, console errors, network logs
Multi-Format Output: JSON, YAML, TOON for all results and topology
Browser Testing: Playwright integration for JS-rendered DOM capture
API Testing: 7 OpenAPI endpoints with FastAPI integration
CLI/Shell Execution: SHELL, EXEC, RUN commands with assertions
GUI Execution: Playwright/Selenium support with GUI_START, GUI_CLICK, GUI_INPUT
Unit Test Execution: UNIT_PYTEST, UNIT_ASSERT, UNIT_IMPORT integration
Test Coverage: 65% with pyqual.yaml quality pipeline

Installation

# Install from source
pip install -e .

# Install with development dependencies
pip install -e ".[dev]"

Requirements

Python 3.10+
HTTPX for API testing
Playwright for browser GUI testing (optional)
Native desktop E2E (optional): vdisplay, img2nl, imgl — see Desktop GUI E2E

Optional: native desktop + vision

pip install -e ".[desktop,vision]" \
  -e ~/github/wronai/vdisplay[pillow] \
  -e ~/github/wronai/img2nl[analyze,similarity,opencv,scan] \
  -e ~/github/semcod/imgl

# System: xvfb x11-apps xdotool xrandr tesseract-ocr

Quick Start

# Run a test scenario
testql scenarios/tests/test-api.testql.toon.yaml --url http://localhost:8101

# Dry-run (parse + validate only)
testql scenarios/views/connect-id-barcode.testql.toon.yaml --dry-run

# JSON output for CI integration
testql scenarios/tests/test-api.testql.toon.yaml --output json

# Run with verbose logging
testql scenarios/tests/test-api.testql.toon.yaml --verbose

Artifact Discovery, Topology, and Web Inspection

TestQL can now inspect codebases, manifests, and opt-in live URLs, then write structured topology and result artifacts for humans, CI, and LLM workflows.

# Discover local artifact types
python3 -m testql.cli discover ./testql --format json

# Build a topology graph from discovered artifacts
python3 -m testql.cli topology ./testql --format toon

# Inspect a live web page and write all data/metadata to .testql
python3 -m testql.cli inspect https://tom.sapletta.com/ \
  --scan-network \
  --out-dir .testql

The .testql/ artifact bundle contains:

metadata.json
topology.{json,yaml,toon.yaml}
result.{json,yaml,toon.yaml}
refactor-plan.{json,yaml,toon.yaml}
inspection.{json,yaml,toon.yaml}
summary.md

Live URL inspection currently extracts:

HTTP metadata: status code, final URL, content type.
Page schema: title, links, assets, forms.
Topology graph: page, link, asset, form, interface, evidence nodes.
Structured results: web checks for status, title, links, assets, forms.
Reports: TOON/YAML/JSON plus NLP summary.

Example:

examples/web-inspection-dot-testql/run.sh https://tom.sapletta.com/

Current capabilities:

Asset classification: script, stylesheet, image, icon, preload, link.
Bounded link validation: HEAD checks for all internal links (up to 100).
Bounded asset validation: HEAD checks for all extracted assets.
Broken resource detection: assets or links returning error status are flagged as findings.
Bounded sitemap crawl: fetches up to 10 internal subpages, extracts titles and link counts, adds subpage nodes to the topology.
Sitemap checks: crawl coverage, broken subpage detection, duplicate title warnings.
Playwright browser inspection (--browser): renders the page in a headless browser, captures console errors, network calls (REST/GraphQL/WebSocket), and JS-rendered DOM.
Browser checks: render detection, console error count, network call capture, title extraction, link/asset/form enumeration.

Current limitations:

Per-resource validation uses HEAD requests only; full page content is not fetched for linked pages.
Screenshots, performance metrics, accessibility checks, and auth flows are planned next.

API Endpoint Detection

TestQL includes advanced endpoint detection for multiple frameworks:

# List all detected endpoints in a project
testql endpoints ./my-project
testql endpoints ./my-project --format json
testql endpoints ./my-project --framework fastapi

# Analyze project structure and generate tests
testql analyze ./my-project
testql generate ./my-project

Supported Frameworks

FastAPI — Detects routers, decorators, include_router patterns
Flask — Blueprints, MethodView, route decorators
Django — URL patterns from urls.py
Express.js — JavaScript/TypeScript route definitions
OpenAPI/Swagger — Spec file parsing (JSON/YAML)
GraphQL — Schema and resolver detection
WebSocket — WS endpoint detection

Endpoint Detection Features

Framework Detection: Automatically identifies the web framework
Handler Names: Extracts function/method names for documentation
Parameters: Detects path/query/body parameters
Docstrings: Uses function docstrings for test descriptions
Test Inference: Analyzes existing tests to discover endpoints

OpenAPI Generation

Generate OpenAPI 3.0 specifications from your code:

# Generate OpenAPI spec (YAML format)
testql openapi ./my-project
testql openapi ./my-project --format json

# Generate with contract tests
testql openapi ./my-project --contract-tests

# Custom output file
testql openapi ./my-project -o ./docs/api-spec.yaml --title "My API"

Contract Testing

TestQL generates contract tests from OpenAPI specs:

# Auto-generated from openapi.yaml
API[25]{method, endpoint, expected_status}:
  GET, /api/v1/users, 200
  POST, /api/v1/users, 201
  ...

ASSERT[3]{field, operator, expected}:
  content_type, ==, application/json
  schema_valid, ==, true
  status, <, 500

Project Echo (AI Context)

Generate AI-friendly project metadata by combining TESTQL scenarios with DOQL system models:

# Generate unified project context
testql echo --toon-path testql-scenarios/ --doql-path app.doql.less

# JSON output for LLM consumption
testql echo --toon-path ./tests --doql-path ./app.doql.less --format json -o context.json

# Text output (human-readable)
testql echo --doql-path ./app.doql.less --format text

Echo Layers

Layer	Source	Content
API Contract	`*.testql.toon.yaml`	Endpoints, methods, assertions
System Model	`*.doql.less`	Entities, workflows, interfaces
Unified Context	Combined	Complete project metadata for AI

Example Output

📦 Project: weboql (0.1.2)

🧠 Type:
  • API (fastapi)

🛠️ Workflows:
  • install: pip install -e .
  • test: pytest
  • run: HARDWARE_MODE=mock weboql-server

🌐 API scenarios:
  • API Health Check (api) - 4 endpoint(s)

💡 LLM suggestions:
  • Run tests: task test
  • Start server: task run

Language Reference

Variables

SET api_url "http://localhost:8101"
SET timeout 5000
GET api_url              # Prints variable value

Variables support ${var} and $var interpolation:

SET base_url "http://localhost:8101"
API GET "${base_url}/api/devices"

Logging

LOG "Starting test suite"
LOG "Current value: ${var}"

API Commands

# HTTP methods
API GET "/api/v3/data/devices"
API POST "/api/v3/scenarios" {"id": "ts1", "name": "Test"}
API PUT "/api/v3/devices/123" {"status": "active"}
API DELETE "/api/v3/scenarios/old"

Assertions

# Status code assertions
ASSERT_STATUS 200
ASSERT_OK                  # Shorthand for 2xx status

# Content assertions
ASSERT_CONTAINS "device"
ASSERT_CONTAINS "status": "ok"

# JSON path assertions
ASSERT_JSON data.length > 0
ASSERT_JSON devices[0].id == "dev-001"
ASSERT_JSON status != "error"

Operators: ==, !=, >, >=, <, <=

GUI Navigation (Playwright)

Browser-based GUI tests via Playwright. For native OS desktop (monitors, windows, OCR, mirror capture on Wayland), see Desktop GUI E2E.

# Navigation
NAVIGATE "/connect-workshop"
NAVIGATE "${base_url}/devices"

# Interaction
WAIT 500
CLICK "[data-action='search']"
INPUT "#search-input" "drager"
CLICK "button[type='submit']"

# Assertions
ASSERT_VISIBLE "[data-testid='results']"
ASSERT_TEXT "#status" "Connected"

Native Desktop (Linux)

Real OS desktop E2E via vdisplay mirror→Xvfb (no Wayland portal). Full guide: Desktop GUI E2E.

DESKTOP_MONITORS
DESKTOP_CAPTURE "shot.png" primary
DESKTOP_ASSERT_WINDOW "Toolbox"
DESKTOP_ASSERT_ELEMENTS 1 "shot.png"
DESKTOP_ANALYZE "shot.png" "layout.json"
DESKTOP_FOCUS "Toolbox"

DISPLAY=:0 testql run examples/desktop/gui-e2e-inspect.oql
DISPLAY=:0 bash examples/desktop/run-all.sh

Hardware Encoder Commands

ENCODER_ON                    # Activate encoder mode
ENCODER_FOCUS column1         # Focus specific column
ENCODER_SCROLL 3              # Scroll 3 steps
ENCODER_CLICK                 # Confirm (single click)
ENCODER_DBLCLICK              # Cancel (double click)
ENCODER_PAGE_NEXT             # Next page
ENCODER_PAGE_PREV             # Previous page
ENCODER_STATUS                # Print current encoder state
ENCODER_OFF                   # Deactivate encoder mode

Script Composition

# Include common setup
INCLUDE "common-setup.testql.toon.yaml"

# Include relative paths
INCLUDE "../helpers/auth.testql.toon.yaml"

Control Flow (Planned)

IF status == "ready"
  LOG "System ready"
ELSE ERROR "System not ready"

LABEL start
REPEAT 3 {
  API GET "/api/ping"
}
GOTO start

Project Structure

testql/
├── testql/
│   ├── cli.py           # Command-line interface
│   ├── base.py          # Base test runner
│   ├── commands/        # Command implementations
│   ├── runners/         # Test execution runners
│   └── reporters/       # Output formatters
├── scenarios/           # Sample test scenarios
│   ├── tests/          # Automated test suites
│   ├── views/          # GUI view tests
│   ├── diagnostics/    # System diagnostic scripts
│   ├── examples/       # Usage examples
│   └── recordings/     # Recorded test sessions
├── docs/
│   ├── testql-spec.md  # Full language specification
│   └── recipes/        # Common testing patterns
└── tests/              # Unit tests

Scenario Organization

TestQL scenarios use two formats:

*.testql.toon.yaml — TestTOON tabular format for test sequences (preserves order)
*.testql.less — LESS format for shared test configuration (variables, mixins)

Directory	Purpose	Example
`scenarios/tests/`	API/integration tests	`test-api.testql.toon.yaml`
`scenarios/views/`	GUI view tests	`connect-id-barcode.testql.toon.yaml`
`scenarios/diagnostics/`	System checks	`health-check.testql.toon.yaml`
`scenarios/examples/`	Learning samples	`device-identification.testql.toon.yaml`
`scenarios/recordings/`	Recorded sessions	`session-recording.testql.toon.yaml`

TestTOON Format

Test files use the tabular TestTOON format where each section declares its schema:

# SCENARIO: API Smoke Test
# TYPE: api
# VERSION: 1.0

API[3]{method, endpoint, status}:
  GET,  /api/v3/health,   200
  GET,  /api/v3/devices,  200
  POST, /api/v3/start,    201

ASSERT[2]{field, op, expected}:
  status,  ==, ok
  count,   >,  0

LESS Configuration

Shared test configuration uses LESS for variables and mixins:

// config.testql.less
@api_url: http://localhost:8101;
@timeout_ms: 30000;

.api-test(@method, @endpoint) {
  method: @method;
  endpoint: @endpoint;
  expect_status: 200;
}

CLI Options

testql <file> [options]

Options:
  --url <url>         Base URL for API tests (default: http://localhost:8101)
  --dry-run           Parse and validate only, don't execute
  --output <format>   Output format: text (default), json, junit
  --verbose           Enable verbose logging
  --timeout <ms>      Default timeout for operations

Generate from Topology

Generate executable scenarios from discovered topology paths:

# Generate TestTOON from the first topology trace
testql generate-topology ./project

# Generate IR JSON from a specific trace
testql generate-topology ./project --trace-id trace.001 --format ir-json

# Write to file with live network scanning
testql generate-topology ./project --scan-network -o scenario.testql.toon.yaml

Testing

# Run all tests
pytest

# Run with coverage
pytest --cov=testql

# Run specific test file
pytest tests/test_runner.py -v

Example Scenario

# scenarios/tests/test-api.testql.toon.yaml
# SCENARIO: Device API Test
# TYPE: api
# VERSION: 1.0

CONFIG[1]{key, value}:
  base_url,  http://localhost:8101

API[2]{method, endpoint, status}:
  GET,   /api/v3/data/devices,   200
  POST,  /api/v3/scenarios,      201

ASSERT[2]{field, op, expected}:
  data.length,  >,   0
  data.id,      !=,  null

Real-world deployment — c2004 healing pipeline

Since May 2026 TestQL runs as the active-probing component of the c2004 fleet management monorepo's self-healing pipeline. It's the canonical, production-grade reference for how to wire testql run into Prometheus + Alertmanager + an LLM-agnostic ticket layer.

What c2004 uses TestQL for

Component	Role
`realtime-health.testql.toon.yaml`	50-assertion scenario covering 20 endpoints across backend, frontend, connect-* services, hw-firmware, and modules
`testql-watchdog` (Docker container)	Runs the scenario every 60s, exposes `/metrics` to Prometheus, POSTs failures to a healing webhook
TOON output format	30-60% smaller than JSON in LLM context — critical for fitting failures into a prompt window

Pipeline shape

testql-watchdog (loop)
    │ runs `testql run realtime-health.testql.toon.yaml --output json`
    ▼
50 assertions execute → exposes testql_scenario_pass_total / fail_total
    │ on failure: POST http://healing-webhook:8810/probe-failure
    ▼
healing-webhook → planfile ticket create --label llm-ready
    │
    ▼
LLM agent (Windsurf/Cursor/Claude Code/aider) reads
  `planfile ticket show PLF-XXX` and proposes a fix

Operator flow

Note: The observability stack (including testql-watchdog) does not start automatically with make dev / make run. Those commands only start the application processes (Node.js + Python). You must launch the monitoring stack separately.

# 1. Terminal A — start the application (no monitoring)
make dev                        # starts frontend + backend + encoder

# 2. Terminal B — start observability + self-healing (separate step)
task monitor:up                 # brings up testql-watchdog + 10 other containers

# 3. Verify watchdog is running
task monitor:probe              # ad-hoc run of the scenario (50 assertions)
docker logs c2004-testql-watchdog | tail
# → cycle done exit=0 pass=50 fail=0

# 4. View Prometheus metrics
open http://localhost:9101/metrics   # testql_scenario_pass_total / fail_total

Key commands:

task monitor:up — start full observability stack
task monitor:core — start core stack (lighter, without Loki/Blackbox)
task monitor:down — stop all observability containers
task monitor:logs:watchdog — tail testql-watchdog logs

Why TestQL was chosen for this role

Black-box + business-logic — pinging /health is not enough; TestQL exercises real endpoints (/api/v3/devices, /api/v3/menu/configurations, etc.) with full assertion blocks.
TOON format — the scenario file is human-editable and ~40% smaller than the equivalent JSON when sent to an LLM for diagnosis.
Single binary — pip install testql and the watchdog container is ~80 MB.
JSON output mode — --output json makes Prometheus metric extraction a one-liner in watchdog.py.

Examples

Example	Description
API Testing	REST API testing with `API GET/POST` and assertions
Artifact Bundle	Generate `.testql/` bundles via CLI or Python script
Browser Inspection	Headless browser inspection with Playwright
Discovery	Discover artifacts and build project topology
Desktop E2E	Native Linux desktop: vdisplay mirror, img2nl, imgl OCR
GUI Testing	Playwright-based GUI navigation and assertions
Project Echo	Generate AI context from TestQL + DOQL models
Shell Testing	Run shell commands and assert exit codes/output
TestTOON Basics	Tabular TestTOON format walkthrough
Topology	Generate topology graphs from codebases
Web Inspection	Inspect live URLs and generate structured reports
Web Inspection + Bundle	Full web inspection writing `.testql` artifact bundle

Documentation

TestQL Specification — Complete language reference
Desktop GUI E2E — Native Linux desktop testing (vdisplay, img2nl, imgl)
Desktop examples — Scenario index and run-all script
Recipes — Common testing patterns and examples

License

Licensed under Apache-2.0.

Name		Name	Last commit message	Last commit date
Latest commit History 129 Commits
.aider		.aider
.github/workflows		.github/workflows
.testql		.testql
.windsurf		.windsurf
TODO		TODO
articles		articles
code2llm_output		code2llm_output
docs		docs
examples		examples
packages		packages
project		project
scenarios		scenarios
scripts		scripts
testql-scenarios		testql-scenarios
testql		testql
tests		tests
.gitignore		.gitignore
CHANGELOG.md		CHANGELOG.md
DSL.md		DSL.md
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
REFACTORING_PLAN.md		REFACTORING_PLAN.md
SUMD.md		SUMD.md
SUMR.md		SUMR.md
TODO.md		TODO.md
Taskfile.testql.yml		Taskfile.testql.yml
Taskfile.yml		Taskfile.yml
VERSION		VERSION
app.doql.less		app.doql.less
coverage.json		coverage.json
goal.yaml		goal.yaml
openapi.yaml		openapi.yaml
planfile.yaml		planfile.yaml
prefact.yaml		prefact.yaml
project.sh		project.sh
pyproject.toml		pyproject.toml
pyqual.yaml		pyqual.yaml
sumd.json		sumd.json
test_autoloop_api.py		test_autoloop_api.py
test_autoloop_mcp.py		test_autoloop_mcp.py
test_manifest_and_generators.py		test_manifest_and_generators.py
topology.toon		topology.toon
topology.toon.yaml		topology.toon.yaml
tree.sh		tree.sh
tree.txt		tree.txt

Folders and files

Latest commit

History

Repository files navigation

TestQL — Interface Query Language for Testing

AI Cost Tracking

What's new in 1.2.x

Recent Improvements (1.2.48)

Installation

Requirements

Optional: native desktop + vision

Quick Start

Artifact Discovery, Topology, and Web Inspection

API Endpoint Detection

Supported Frameworks

Endpoint Detection Features

OpenAPI Generation

Contract Testing

Project Echo (AI Context)

Echo Layers

Example Output

Language Reference

Variables

Logging

API Commands

Assertions

GUI Navigation (Playwright)

Native Desktop (Linux)

Hardware Encoder Commands

Script Composition

Control Flow (Planned)

Project Structure

Scenario Organization

TestTOON Format

LESS Configuration

CLI Options

Generate from Topology

Testing

Example Scenario

Real-world deployment — c2004 healing pipeline

What c2004 uses TestQL for

Pipeline shape

Operator flow

Why TestQL was chosen for this role

See also

Examples

Documentation

License

About

Topics

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

Packages