Add docs/: setup, concepts, methodology, FAQ, and codex support - #16
Merged
Merged
Conversation
Every link in the README's documentation map was a 404 -- docs/ did not exist. This adds the 12 pages that are setup and explanation rather than published results, so a new reader can get from the landing page to a first run. Imported: getting-started, hosting-paths, how-a-run-works, benchmark-your-own-repo, repository-structure, why-this-exists, vision, diagram-ascii, cost-per-task-methodology, serving-optimization-notes, omp-setup, kiro-cli-setup. Not imported, and their README rows removed: - gpu-selection-h200-vs-l40s.md -- 10 of its links point into self-hosted/vllm/benchmark-output/, the committed throughput dashboards this repo does not ship. The doc is a report on that data, so it cannot stand without it. - release-notes/ -- 0.1.0.md states that every score in it comes from docs/metrics/pareto-frontier-omp-swe3.json. It is a results release note. Edits to the imported pages, all to remove references to content this repo does not carry: - benchmark-your-own-repo.md: the "use the frontier we already published" path pointed at agentic-coding-swe-comparison-swe3.md. Repointed at the reference frontier that actually ships, vend/swe-router/models.json, with its own single-repository caveat noted. - why-this-exists.md, cost-per-task-methodology.md, omp-setup.md, diagram-ascii.md: dropped links to excluded comparison/results docs, the slides directory, and two raw run-data files. - cost-per-task-methodology.md: two sentences referred to a README leaderboard that no longer exists. - vision.md, kiro-cli-setup.md: dropped three issue links into the aarora79 personal repo; those issue numbers do not exist here. - repository-structure.md: the tree still said claude-code-multi-model/ and omitted docs/ and vend/. It also claimed a committed Hello-World example under swe-benchmark-data/, which is not present -- that path is gitignored run output. Verified: every relative link in README.md resolves, and every internal link across docs/ resolves.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
Every link in the README's documentation map was a 404 —
docs/did not exist onmain. A new reader hit a dead end before reaching a first run. Separately,codexis fully supported in the harness but was invisible in the docs and omitted from the--agentlist.What this adds
16 doc pages — setup and explanation, not published results:
getting-started·hosting-paths·how-a-run-works·benchmark-your-own-repo·repository-structure·why-this-exists·vision·diagram-ascii·cost-per-task-methodology·serving-optimization-notes·omp-setup·kiro-cli-setup·codex-setup·faq/wiring-claude-code-to-models·faq/wiring-codex-to-models·faq/wiring-omp-to-modelscodex-setup.mdanddocs/faq/landed upstream after the first import, so they were missing here.--agentlist corrected in two placesBoth omitted
codex, which the harness has supported all along — 68 references inrun-swe-headless.py, andrun-e2e-benchmark.shvalidation at line 167 already accepted all five agents.README.md:--agent claude|pi|omp|kiro→claude|pi|omp|kiro|codexrun-e2e-benchmark.sh:84: the inline comment still read "claude (Claude Code) or pi (pi agent)" — it predatedomp,kiroandcodexentirely.Deliberately left out (README rows removed where they existed)
gpu-selection-h200-vs-l40s.mdself-hosted/vllm/benchmark-output/— committed throughput dashboards this repo doesn't ship. It's a report on that data.release-notes/0.1.0.md,0.2.0.mddocs/metrics/pareto-frontier-*.jsonand quote per-model scores and per-tier breakdowns.harness-{pi,omp,claude-code,kiro-cli}-swe*.md(6)harness-pi-swe2.mdopens# Results: pi harness (swe2)and is a per-model table of scores, costs and token counts. Onboarding for these harnesses is covered byomp-setup.md,kiro-cli-setup.md,codex-setup.md,docs/faq/andbenchmarks/docs/path-*.md.results-*,agentic-coding-swe-comparison-*,best-harness-selection,model-selection-by-complexity,swe-router-evaluation*metrics/(12),images/(54),slides/(5)Edits to the imported pages
All to remove references to content this repo doesn't carry:
benchmark-your-own-repo.md— its "use the frontier we already published" path pointed at an excluded comparison doc. Repointed at the reference frontier that actually ships,vend/swe-router/models.json, noting that file's ownsingle_repository_warning.why-this-exists.md,cost-per-task-methodology.md,omp-setup.md,diagram-ascii.md— dropped links to excluded comparison/results docs, theslides/directory, and two raw run-data files.cost-per-task-methodology.md— two sentences referred to a README leaderboard that no longer exists.vision.md,kiro-cli-setup.md— dropped three issue links into theaarora79personal repo; those issue numbers don't exist here.repository-structure.md— the tree still saidclaude-code-multi-model/, omitteddocs/andvend/, and claimed a committedHello-Worldexample underswe-benchmark-data/that isn't there (that path is gitignored run output).Verified
README.mdresolvesdocs/includingdocs/faq/resolves — 28 broken ones found and fixedaarora79refs, no references to excluded results docs ordocs/imagesbash -nclean on the edited script