EnterpriseClawBench

A benchmark for evaluating coding-agent systems on realistic enterprise workflows.

   

📖 Overview • 📊 Results • 🔧 Installation • 🚀 Quick Start • 📚 Documentation • 🎈 Citation

📖Overview

EnterpriseClawBench is a benchmark and evaluation protocol for coding-agent systems on realistic enterprise workflows. It focuses on complete agent systems, including both the harness and the model, and evaluates whether agents can operate in workspaces, understand files, use tools, and deliver usable business artifacts.

Enterprise agents increasingly work inside persistent workspaces: they inspect uploaded files, call tools, create documents, update spreadsheets, generate web pages, and return multimodal deliverables. EnterpriseClawBench is designed for this setting.

The benchmark is built from real enterprise agent sessions through a privacy-preserving construction pipeline. Since the original sessions contain proprietary information, this repository releases:

• a sanitized public raw-session example;
• the benchmark construction pipeline;
• the local evaluation pipeline;
• the sandbox run-directory protocol;
• a small reference run for evaluation smoke tests;
• aggregate leaderboard data and public visualizations.

The full private benchmark data are not released.

📊Results

Main Leaderboard

The public leaderboard compares harness-model systems rather than base models alone. Each bar is one agent configuration.

Score-Cost Tradeoff

Enterprise agent quality is coupled with cost and runtime. The score-cost plot summarizes the deployment tradeoff across harness-model configurations.

Benchmark Construction Pipeline

EnterpriseClawBench converts raw session traces into benchmark tasks through filtering, fixture recovery, self-contained rewriting, taxonomy assignment, rule generation, and rubric generation.

Task and Artifact Analysis

More figures

Pipeline Overview

Skill Transfer

Judge Ablation

🔧Installation

Install the public modules once from the cloned repository root:

git clone https://github.com/FrontisAI/EnterpriseClawBench
cd EnterpriseClawBench
python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip setuptools wheel
python -m pip install -e './[render]'
python -m playwright install chromium

🚀Quick Start

Verify the construction package:

enterprise-clawbench-construction list-stages
python -m unittest discover \
  -s construction/tests \
  -t construction \
  -v

Run an offline construction smoke test against the bundled public raw session:

cd construction
enterprise-clawbench-construction run \
  --config configs/smoke_raw_examples.yaml \
  --force \
  --to-stage 04_mechanical_join

Run the full construction pipeline from the bundled public raw session. LLM-backed stages require an OpenAI-compatible API key:

cd construction
export DMX_API_KEY='...'
enterprise-clawbench-construction run \
  --config configs/public_session_full.yaml \
  --force

Without --force, construction resumes automatically from the first incomplete stage.

Agent Run Protocol

After construction, each row in eval_tasks.from_task_pack.jsonl is a task to run with an agent. Users may use any sandbox, agent harness, or manual script. The evaluation module only requires one directory per task:

evaluation_runs/<run_name>/<task_id>/
  task.json          # recommended copy of the task row
  prompt.txt         # prompt actually sent to the agent
  response.txt       # final natural-language response
  artifacts/         # files generated by the agent
  result.json        # optional status/time/model metadata
  token_usage.json   # optional usage metadata
  history.jsonl      # optional audit trace

The prompt to send is normally agent_prompt from the task row. If the task has fixtures, upload or mount them for the agent and record the actual file paths in prompt.txt or task.json.

The optional sandbox/ package can produce this layout in environments with access to the required sandbox SDK/service:

enterprise-clawbench-sandbox run \
  --tasks construction_runs/public_session_full_000000000000000000/15_export/eval_tasks.from_task_pack.jsonl \
  --agent deepagents \
  --model claude-sonnet-4-6 \
  --workers 4

External users can bypass sandbox/ and produce the run directory with their own agent runtime.

Evaluation Quickstart

The repository includes one sanitized reference run:

example_runs/public_session_deepagents_sonnet/

Judge an agent run:

cd evaluation
enterprise-clawbench-eval judge-run \
  --tasks ../construction_runs/public_session_full_000000000000000000/15_export/eval_tasks.from_task_pack.jsonl \
  --run-dir ../example_runs/public_session_deepagents_sonnet \
  --out-dir ../outputs/example_public_session_judge \
  --base-url https://your-openai-compatible-endpoint/v1 \
  --text-judge-model claude-sonnet-4-6 \
  --visual-judge-model claude-sonnet-4-6

Without --force, judge-run skips cases that already have enterprise_clawbench_judge_result.json in the sandbox run directory. Add --force to recompute judge results.

📦Repository Contents

construction/            Benchmark construction package
sandbox/                 Agent sandbox execution package
evaluation/              Artifact and rubric evaluation package
docs/                    Setup, protocol, and reproduction notes
raw_session_example/     Sanitized public raw-session snapshot
example_runs/            Sanitized reference agent-run result
assets/figures/          Public figure assets

Generated outputs are intentionally ignored:

construction_runs/
evaluation_runs/
outputs/

📚Documentation

Document	Contents
docs/evaluation_protocol.md	Expected run layout and judge routing behavior.
docs/environment_setup.md	Python, Playwright, system-tool, API-key, and optional sandbox-SDK setup.
docs/release_reproduction_guide.md	End-to-end release workflow.

🎈Citation

If you use EnterpriseClawBench in your research, please cite our work:

@misc{enterpriseclawbench2026,
  title         = {EnterpriseClawBench: Benchmarking Agents from Real Workplace Sessions},
  author        = {Jincheng Zhong and Weizhi Wang and Che Jiang and Kai Tian and Zhenzhao Yuan and Junlin Yang and Dianqiao Lei and Kaiyan Zhang},
  year          = {2026},
  eprint        = {2606.23654},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2606.23654}
}

📝Notes

• Secrets are never stored in configs. Provide judge/construction keys through environment variables.
• The public repository tracks only one sanitized raw-session example: raw_session_example/000000000000000000.
• The construction pipeline is self-contained and does not import from the private research workspace.
• The sandbox package is a thin local wrapper around the installed sandbox_sdk; generated run evidence stays under ignored evaluation_runs/.

📬Contact

Please open a GitHub issue for questions or feedback.