Skip to content

Add docs/: setup, concepts, methodology, FAQ, and codex support - #16

Merged
shekharprateek merged 1 commit into
mainfrom
docs/harness-docs
Sep 15, 2026
Merged

shekharprateek merged 1 commit into
mainfrom
docs/harness-docs

Conversation

@shekharprateek

@shekharprateek shekharprateek commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

The problem

Every link in the README's documentation map was a 404docs/ did not exist on main. A new reader hit a dead end before reaching a first run. Separately, codex is fully supported in the harness but was invisible in the docs and omitted from the --agent list.

What this adds

16 doc pages — setup and explanation, not published results:

getting-started · hosting-paths · how-a-run-works · benchmark-your-own-repo · repository-structure · why-this-exists · vision · diagram-ascii · cost-per-task-methodology · serving-optimization-notes · omp-setup · kiro-cli-setup · codex-setup · faq/wiring-claude-code-to-models · faq/wiring-codex-to-models · faq/wiring-omp-to-models

codex-setup.md and docs/faq/ landed upstream after the first import, so they were missing here.

--agent list corrected in two places

Both omitted codex, which the harness has supported all along — 68 references in run-swe-headless.py, and run-e2e-benchmark.sh validation at line 167 already accepted all five agents.

  • README.md: --agent claude|pi|omp|kiroclaude|pi|omp|kiro|codex
  • run-e2e-benchmark.sh:84: the inline comment still read "claude (Claude Code) or pi (pi agent)" — it predated omp, kiro and codex entirely.

Deliberately left out (README rows removed where they existed)

Doc Why
gpu-selection-h200-vs-l40s.md 10 links into self-hosted/vllm/benchmark-output/ — committed throughput dashboards this repo doesn't ship. It's a report on that data.
release-notes/0.1.0.md, 0.2.0.md Both source their figures from docs/metrics/pareto-frontier-*.json and quote per-model scores and per-tier breakdowns.
harness-{pi,omp,claude-code,kiro-cli}-swe*.md (6) Not onboarding guides. harness-pi-swe2.md opens # Results: pi harness (swe2) and is a per-model table of scores, costs and token counts. Onboarding for these harnesses is covered by omp-setup.md, kiro-cli-setup.md, codex-setup.md, docs/faq/ and benchmarks/docs/path-*.md.
results-*, agentic-coding-swe-comparison-*, best-harness-selection, model-selection-by-complexity, swe-router-evaluation* Published model rankings.
metrics/ (12), images/ (54), slides/ (5) The frontier JSON and rendered charts behind those rankings.

Edits to the imported pages

All to remove references to content this repo doesn't carry:

  • benchmark-your-own-repo.md — its "use the frontier we already published" path pointed at an excluded comparison doc. Repointed at the reference frontier that actually ships, vend/swe-router/models.json, noting that file's own single_repository_warning.
  • why-this-exists.md, cost-per-task-methodology.md, omp-setup.md, diagram-ascii.md — dropped links to excluded comparison/results docs, the slides/ directory, and two raw run-data files.
  • cost-per-task-methodology.md — two sentences referred to a README leaderboard that no longer exists.
  • vision.md, kiro-cli-setup.md — dropped three issue links into the aarora79 personal repo; those issue numbers don't exist here.
  • repository-structure.md — the tree still said claude-code-multi-model/, omitted docs/ and vend/, and claimed a committed Hello-World example under swe-benchmark-data/ that isn't there (that path is gitignored run output).

Verified

  • Every relative link in README.md resolves
  • Every internal link across docs/ including docs/faq/ resolves — 28 broken ones found and fixed
  • No aarora79 refs, no references to excluded results docs or docs/images
  • bash -n clean on the edited script
  • Pre-commit conditions clean: no trailing whitespace, no missing final newlines, nothing over the 1536 KB ceiling

Every link in the README's documentation map was a 404 -- docs/ did not exist.
This adds the 12 pages that are setup and explanation rather than published
results, so a new reader can get from the landing page to a first run.

Imported: getting-started, hosting-paths, how-a-run-works, benchmark-your-own-repo,
repository-structure, why-this-exists, vision, diagram-ascii, cost-per-task-methodology,
serving-optimization-notes, omp-setup, kiro-cli-setup.

Not imported, and their README rows removed:
- gpu-selection-h200-vs-l40s.md -- 10 of its links point into
  self-hosted/vllm/benchmark-output/, the committed throughput dashboards this
  repo does not ship. The doc is a report on that data, so it cannot stand
  without it.
- release-notes/ -- 0.1.0.md states that every score in it comes from
  docs/metrics/pareto-frontier-omp-swe3.json. It is a results release note.

Edits to the imported pages, all to remove references to content this repo does
not carry:
- benchmark-your-own-repo.md: the "use the frontier we already published" path
  pointed at agentic-coding-swe-comparison-swe3.md. Repointed at the reference
  frontier that actually ships, vend/swe-router/models.json, with its own
  single-repository caveat noted.
- why-this-exists.md, cost-per-task-methodology.md, omp-setup.md,
  diagram-ascii.md: dropped links to excluded comparison/results docs, the
  slides directory, and two raw run-data files.
- cost-per-task-methodology.md: two sentences referred to a README leaderboard
  that no longer exists.
- vision.md, kiro-cli-setup.md: dropped three issue links into the aarora79
  personal repo; those issue numbers do not exist here.
- repository-structure.md: the tree still said claude-code-multi-model/ and
  omitted docs/ and vend/. It also claimed a committed Hello-World example under
  swe-benchmark-data/, which is not present -- that path is gitignored run output.

Verified: every relative link in README.md resolves, and every internal link
across docs/ resolves.
@shekharprateek
shekharprateek merged commit ad652e9 into main Sep 15, 2026
8 checks passed
@shekharprateek shekharprateek changed the title Add docs/: setup, concepts, and methodology pages Add docs/: setup, concepts, methodology, FAQ, and codex support Sep 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant