Repository navigation
TypedPython benchmarks: harness for CPython / Node / Java / Rust - #28
Merged
Merged
Conversation
… Rust) Five programs (nbody, spectral-norm, fannkuch-redux, binary-trees, and a dict-heavy word-frequency count), each in gate-clean typed Python, Node, Java and Rust, all printing identical results. run.py verify checks agreement; run.py measure refuses to run above load 4.0 and writes results only with --write. The @compiled column reports n/a until the backend exists. No results are committed: the machine was at load 40+ throughout. Refs #24
Median of 5, wall time, measured alone at load 1.60/1.55/1.47 (a first run at load 3.1 agreed within ~3%): | benchmark | CPython | Node | Java | Rust | |---|---|---|---|---| | nbody 1e6 | 4.143 s | 0.130 s | 0.187 s | 0.061 s | | spectral-norm 800 | 3.441 s | 0.076 s | 0.116 s | 0.027 s | | fannkuch-redux 10 | 6.196 s | 0.271 s | 0.269 s | 0.261 s | | binary-trees 16 | 3.938 s | 0.172 s | 0.163 s | 0.266 s | | wordfreq 3e6 | 2.509 s | 0.972 s | 0.343 s | 0.164 s | Java includes JVM startup, excludes javac. @compiled column: not built yet. Closes #24
With --typedpython <dir> (and --typedpython-python for the compile step), each benchmark is copied with the module marker, compiled by typedpython.pipeline, and its main() runs from the built extension. verify includes the column in the agreement check; benchmarks with nothing compiled report n/a. verify: all 5 agree. Refs #24, #41
This was referenced Oct 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #24 — the baseline is now measured on an idle machine (see the last commit and results/2026-10-03-*.json).
Feature added
benchmarks/typedpython/contains five programs, each implemented in typed Python, Node, Java and Rust:All implementations print byte-identical results. A sixth column,
TypedPython @compiled, printsn/a (not built)until M2 delivers the compiler.The runner has two commands:
python3 run.py verifychecks that all implementations agree at small sizes.python3 run.py measure [--repeat N] [--write]reports the median of N runs. It printsuptimefirst, refuses to run above load 4.0 unless--forceis given, and writesresults/<date>-<host>.jsononly with--write.Test added / verification
run.py verifyexits 0 and all 5 benchmarks agree. Re-run by the coordinating session.--mode compiled: exit 0, no diagnostics.Not done
No results are committed. The load average stayed between 11 and 48 for the whole session, so the numbers would not mean anything (AGENTS.md §8).