gillm

GUI Control Plugin with NLP & Intent Contracts

Sterowanie GUI przez DSL, NLP i wiele adapterów wejścia (CLI, REST, MCP, URI). Domena (focus, inject, capture, orchestrator) żyje w src/gillm/; warstwa kontroli w packages/*2gillm.

AI Cost Tracking

• 🤖 LLM usage: $2.6263 (14 commits)
• 👤 Human dev: ~$994 (9.9h @ $100/h, 30min dedup)

Generated on 2026-06-10 using openrouter/qwen/qwen3-coder-next

---

Graphical interface for LLM automation in the semcod/coru ecosystem. Pair with tillm for text / shell control (aider, claude, codex, …).

Package	Interface	Role
gillm	graphical	GUI domain + *2gillm adapters (REST, MCP, URI, NLP)
tillm	text / shell	vendor CLI drive, NLP → shell DSL

Contents

• Architecture
• Control layer packages
• Installation
• Quick start
• DSL verbs
• CLI
• REST API
• Environment variables
• Testing
• Project layout
• Documentation
• License

Architecture

SUMD (description) → DOQL/source (code) → taskfile (automation) → testql (verification)

Sources: SUMD.md · app.doql.less · Makefile

flowchart TB
  subgraph adapters [Input adapters — packages]
    NL[nlp2gillm]
    URI[uri2gillm]
    CLI[cli2gillm]
    MCP[mcp2gillm]
    REST[rest2gillm]
  end

  subgraph control [Control layer]
    TXT[DSL text line]
    DICT[JSON dict]
    PBIN[protobuf bytes]
    SCH[JSON Schema validate]
    DSL[dsl2gillm.dispatch]
    Q[QueryHandler]
    C[CommandHandler]
    ES[(EventStore)]
  end

  subgraph domain [Domain — src/gillm]
    ORCH[orchestrator.drive]
    FOCUS[focus / injection]
    CAPTURE[capture]
    NLP[nlp_bridge]
  end

  NL --> TXT
  URI --> TXT
  CLI --> TXT
  MCP --> TXT
  MCP --> DICT
  MCP --> PBIN
  REST --> TXT
  REST --> DICT
  REST --> PBIN

  TXT --> SCH
  DICT --> SCH
  PBIN --> SCH
  SCH --> DSL
  DSL --> Q
  DSL --> C
  C --> ORCH
  C --> ES
  Q --> ORCH
  Q --> FOCUS
  Q --> CAPTURE
  Q --> NLP

Loading

Jedyny punkt mutacji w warstwie kontroli: dsl2gillm.dispatch().

Control layer packages

Package	Role	Docs
dsl2gillm	DSL grammar, JSON Schema, Protobuf, CQRS bus, EventStore	README
uri2gillm	gillm:// URI → DSL line → dispatch()	README
nlp2gillm	Natural language → DSL (to-dsl); apply = dispatch	README
cli2gillm	Shell REPL / exec / run	README
mcp2gillm	MCP stdio (gillm_run_command, gillm_run_command_pb, …)	README
rest2gillm	FastAPI /v1/dsl, default port 8220	README

Overview: packages/README.md.

Installation

Runtime (core GUI domain only):

pip install -e .
# or: make install

Development (control layer + test tooling):

bash packages/install-dev.sh
# or: pip install -e ".[dev]"
# or: make dev-install

Optional control extras without full dev stack:

pip install -e ".[control]"

Requires Python ≥ 3.10.

Quick start

# Schema + dry-run workflow (no real GUI)
dsl2gillm validate-schema
gillm run fixtures/workflow.json --dry-run
gillm run fixtures/workflow-dry.json --dry-run

# NLP → DSL
nlp2gillm to-dsl "check health"

# REST smoke test (separate terminal)
rest2gillm serve --port 8220
curl http://127.0.0.1:8220/health
curl -X POST http://127.0.0.1:8220/v1/dsl -d 'HEALTH'

DSL verbs

Type	Verbs
Query	HEALTH, ORIENT, PARSE, ACTIONS, VALIDATE, RESOLVE, CAPTURE
Command	EXECUTE, SIMULATE, FOCUS, INJECT

Example lines:

HEALTH
ORIENT
ACTIONS
PARSE "focus vscode and type hello"
VALIDATE STEPS [{"action":"wait","config":{"seconds":0.01}}]
RESOLVE "capture screen"
CAPTURE SCALE 0.2
EXECUTE FILE workflow.json
SIMULATE FILE workflow.json
FOCUS HINTS vscode,cursor
INJECT "hello" IDE default

Generate Pydantic models from JSON schemas:

python -m dsl2gillm.codegen
# or: dsl2gillm codegen

CLI

Entry point	Purpose	Docs
gillm	Legacy CLI: run, nlp, capture (delegates to dsl2gillm.dispatch)	README
dsl2gillm	DSL bus: -c 'HEALTH', validate-schema, codegen, protobuf codecs	README
nlp2gillm	NL → DSL translation	README
uri2gillm	gillm:// URI resolution	README
cli2gillm	Interactive shell	README
mcp2gillm	MCP server	README
rest2gillm	FastAPI server (serve --port 8220)	README

gillm run fixtures/workflow-dry.json --dry-run
gillm nlp "focus vscode and type hello" --dry-run
gillm capture --scale 0.2
dsl2gillm -c 'HEALTH'

REST API

GET / returns an endpoint map. Default base URL: http://127.0.0.1:8220/.

rest2gillm serve --port 8220
curl http://127.0.0.1:8220/
curl http://127.0.0.1:8220/health
curl -X POST http://127.0.0.1:8220/v1/dsl -d 'HEALTH'

Environment variables

Copy .env.example to .env for local development.

Variable	Default	Description
OPENROUTER_API_KEY	(not set)	OpenRouter API key (keys)
LLM_MODEL	openrouter/qwen/qwen3-coder-next	LLM model for NLP / pfix
PFIX_AUTO_APPLY	true	Apply fixes without asking
PFIX_AUTO_INSTALL_DEPS	true	Auto pip/uv install
PFIX_AUTO_RESTART	false	Restart process after fix
PFIX_MAX_RETRIES	3	Max self-heal retries
PFIX_DRY_RUN	false	Dry-run mode
PFIX_ENABLED	true	Enable pfix
PFIX_GIT_COMMIT	false	Auto-commit fixes
PFIX_GIT_PREFIX	pfix:	Commit message prefix
PFIX_CREATE_BACKUPS	false	Disable .pfix_backups/

GUI injection (Wayland/X11) also reads KORU_* variables — see SUMD.md for the full list (KORU_OS_INJECTOR, KORU_INJECTOR_BACKEND, WAYLAND_DISPLAY, DISPLAY, …).

Testing

Core domain (src/gillm):

python -m pytest tests/ -q
# or: make test

Main modules:

• tests/test_injector.py — gillm.injection.injector.Injector (keyboard backends: xdotool/ydotool/wtype)
• tests/test_os_injector.py — calibrated profiles under ~/.koru/ide-os-injector.json
• tests/test_gui_driver.py, tests/test_drive_backend.py, tests/test_recovery.py

Control layer (packages/*2gillm):

python -m pytest packages/dsl2gillm/tests packages/uri2gillm/tests packages/nlp2gillm/tests \
       packages/cli2gillm/tests packages/mcp2gillm/tests packages/rest2gillm/tests -q

Makefile shortcuts: make test-fast, make test-unit, make test-integration, make test-slow.

Koru keeps integration tests only (daemon wire protocol, IDE shims, CLI). Do not re-add duplicate injector unit tests under koru/tests/.

Project layout

gillm/
├── src/gillm/          # GUI domain: focus, injection, capture, orchestrator, recovery
├── packages/           # Control adapters: dsl2gillm, uri2gillm, nlp2gillm, cli2gillm, mcp2gillm, rest2gillm
├── tests/              # Domain unit tests
├── fixtures/           # Workflow JSON for dry-run smoke tests
├── app.doql.less       # DOQL application declaration
├── SUMD.md             # Structured Unified Markdown Descriptor (full project spec)
└── pyproject.toml

Documentation

Document	Purpose
README.md	User guide (this file)
SUMD.md	Full Structured Unified Markdown Descriptor — DOQL, interfaces, env vars, call graph, test contracts
SUMR.md	Compact SUMD for refactoring (call graph, duplication, evolution)
packages/README.md	Control layer architecture, smoke tests, DSL verbs
app.doql.less	DOQL application declaration (source of truth for entities/workflows)
CHANGELOG.md	Release history
project/README.md	Generated code analysis artifacts (*.toon.yaml, Mermaid)
project/context.md	LLM-ready architecture narrative
packages/CONTROL_LAYER_PROMPT.template.md	Template for extending *2gillm adapters

Per-package docs: dsl2gillm · uri2gillm · nlp2gillm · cli2gillm · mcp2gillm · rest2gillm

License

Licensed under Apache-2.0.