Choose the visual model that best fits your images, JSON Schema, and production constraints—not the one that tops a public leaderboard.
中文文档 · API · Methodology · Architecture · Roadmap
VLMForge is a local-first decision workbench for teams building structured extraction with cloud vision models. It evaluates each provider route, structured-output strategy, parameter set, latency, and cost against the same frozen prompt, schema, preprocessing, and private image contract—not just an abstract model name.
It then rejects candidates that violate hard constraints, exposes the quality/cost/P95 Pareto frontier, and explains why a candidate was recommended or eliminated. Schema and critical-rule failures can never be hidden by an LLM judge or an attractive aggregate score.
Public benchmarks are useful for discovering models. They do not tell you whether a pinned provider route can extract your private catalog attributes, honor your cross-field rules, meet your P95 target, and stay within your per-item budget.
VLMForge freezes that real production contract into a reproducible run:
- private image datasets with Draft 2020-12 JSON Schema and safe JSONLogic rules;
- deterministic JSON, schema, rule, field F1, repeat-consistency, and bootstrap-CI metrics;
- complete cost provenance:
actual,estimated, orunknown—never silently zero; - seeded randomized attempt planning, warm-up separation, bounded concurrency, and failure taxonomy;
- hard gates before a three-dimensional quality/cost/P95 Pareto decision;
- anonymous human review through a durable LangGraph
interrupt()/Command(resume=...)flow; - replayable SSE events and HTML, Markdown, JSON, and CSV exports;
- a no-key Replay adapter for demos, CI, and bug reports.
Requirements: Docker Desktop or Docker Engine with Compose v2. No model key is required.
git clone https://github.com/To3akaRin/VLMForge.git
cd VLMForge
cp .env.example .env
docker compose up --build -dWait until all required dependencies are ready, then open http://localhost:4173 and select Start Replay demo.
curl --fail http://localhost:8000/health/ready
curl --fail -X POST http://localhost:8000/api/v1/demoThe deterministic demo creates 20 attempts: two candidates, three cases, three scored repeats, and one warm-up per candidate. The cheaper invalid candidate fails the schema gate; the eligible candidate is recommended and exported with its failure evidence and cost split.
CLI users can drive the same API—there is no second execution engine:
uv sync --all-packages
uv run --package vlmforge vlmforge demo
uv run --package vlmforge vlmforge validate examples/retail-product-demo
uv run --package vlmforge vlmforge status RUN_ID --watch
uv run --package vlmforge vlmforge report RUN_ID --format markdown --output report.md| Decision surface | Failure evidence | Durable orchestration |
|---|---|---|
 |
 |
 |
The synthetic retail product fixture includes four project-bound images, expected JSON, a strict schema, cross-field rules, and asset provenance. It is deliberately small enough to inspect in a code review; it is not presented as a public benchmark.
flowchart LR
A[Freeze run snapshot] --> B[Capability preflight]
B --> C[Plan seeded matrix]
C --> D{Send warm-ups}
D --> E[Attempt subgraph]
E --> F[Warm-up fan-in barrier]
F --> S{Send scored matrix}
S --> T[Attempt subgraph]
T --> U[Deterministic aggregation]
U --> G{Human review enabled?}
G -->|yes| H[interrupt anonymous review]
H --> I[Command resume]
G -->|no| J[Apply hard gates]
I --> J
J --> K[Compute Pareto frontier]
K --> L[Publish explainable report]
Each attempt is its own subgraph:
prepare input → invoke provider → parse JSON → validate schema
→ evaluate rules → optional judge → persist idempotently
The LangGraph checkpoint contains only IDs, status, and small summaries. Images, complete provider responses, and report artifacts stay in MinIO; PostgreSQL remains the business source of truth. A paid request is never advertised as exactly-once: if a worker disappears after dispatch and the provider outcome is unknowable, recovery marks the attempt uncertain instead of charging again.
See Architecture and the evaluation methodology for the invariants.
A CandidateSpec pins the model-side variables below; the run snapshot separately pins the shared dataset, prompt, schema, rules, and preprocessing contract:
{
"id": "openrouter-gemini-prod-route",
"name": "Gemini on pinned production route",
"adapter": "openrouter",
"model": "google/gemini-2.5-flash",
"provider_order": ["Google AI Studio"],
"parameters": {"temperature": 0, "max_tokens": 800, "seed": 42},
"structured_output": "native_only",
"data_collection": "deny",
"disable_cache": true,
"allow_fallbacks": false,
"timeout_seconds": 60,
"concurrency": 2,
"api_key_env": "OPENROUTER_API_KEY"
}Preflight reports unsupported schema keywords and parameters. OpenRouter requests pin provider order, disable fallbacks and caching, deny data collection, require native parameters when requested, and record actual routing metadata. Images are sent as base64 data URLs rather than public URLs.
| Project | Primary focus | Why VLMForge is different |
|---|---|---|
| Promptfoo | General LLM evaluation, red teaming, CLI, and CI | VLMForge narrows the workflow to private vision extraction and an explainable deployment decision. |
| Langfuse | AI observability, traces, prompt management, datasets, and evaluations | VLMForge is a model-selection workbench rather than an observability platform. |
| Phoenix | AI observability and evaluation | VLMForge freezes image/schema/rule/provider candidates and applies hard operational gates. |
| VLMEvalKit | Broad VLM support across public academic benchmarks | VLMForge optimizes for a team's private business images and structured-output contract. |
These projects are complementary. VLMForge intentionally does not become another public leaderboard, tracing backend, model host, or red-team suite.
- Web: Vue 3, TypeScript, Vite, Tailwind, ECharts, TanStack Query, Pinia.
- API: FastAPI, Pydantic v2, SQLAlchemy, Alembic, httpx, Loguru.
- Orchestration: LangGraph with
Send,AsyncPostgresSaver,interrupt, and resume. - Jobs and state: ARQ, Redis, PostgreSQL, MinIO.
- Tooling: UV, pnpm, Pytest, Ruff, strict Mypy, Vitest, Playwright, Docker Compose.
VLMForge/
├── backend/package/vlmforge/ # typed evaluation kernel, adapters, CLI
├── backend/server/ # FastAPI, LangGraph, worker, persistence
├── web/ # Vue decision workbench
├── examples/retail-product-demo/
├── docs/ # specs, ADRs, architecture, methodology
├── docker/ # static web image and reverse proxy
└── compose.yaml # the only supported deployment topology in v0.1
Python 3.12 and Node.js 22 are the supported local toolchains.
uv sync --all-packages
uv run pytest
uv run ruff check backend
uv run mypy backend/package/vlmforge/src
PYTHONPATH=backend/server uv run mypy backend/server/app
cd web
corepack enable
pnpm install --frozen-lockfile
pnpm lint
pnpm test
pnpm build
pnpm test:e2eEnvironment variables, migrations, readiness behavior, and an operator checklist are documented in Deployment. The REST/SSE contract and curl examples are in API.md.
- v0.1 is a trusted local single-workspace tool with no login. Do not expose it directly to the public internet.
- Dataset ZIPs reject path traversal, symlinks, oversized expansion, malformed images, and unexpected contract files.
- Business rules run in a bounded JSONLogic allowlist; arbitrary Python and shell execution are impossible.
- Provider keys are referenced by environment-variable name and are not returned in events or logs.
- SSE events contain IDs, status, progress, and safe summaries—not images, prompts, tokens, or complete model output.
.envis ignored..env.examplecontains placeholders and Docker Compose deliberately does not use Docker Secrets for this local-only release.
Please report vulnerabilities privately as described in SECURITY.md.
Not included: SaaS/multi-tenancy, authentication, public leaderboards, training/fine-tuning, GPU scheduling, model hosting, video/audio/PDF/RAG, agent chat, red teaming, or automatic prompt optimization.
The project currently supports Docker Compose only. Real paid-provider smoke tests are opt-in and never run in default CI.
Start with CONTRIBUTING.md, the implementation specification, and issues labeled good first issue. Changes to evaluation semantics require tests and a methodology/ADR update in the same pull request.
MIT licensed. See CHANGELOG.md for release history.