Repository navigation
README: 0.8.5 benchmarks - #21
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a7810afb46
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| Pi-Bolt 0.8.5 against Pi 1.1.0 as released, on stock Bun 1.4.2 and on Node, and against two other fast Pi builds: | ||
| [PiG](https://github.com/MichaelKinsy/PiG) 0.4.1, the Go port of Pi (based on Pi 1.0.3), and | ||
| [oh-my-pi](https://github.com/can1357/oh-my-pi) 18.8.7, a fork of Pi on Bun. All five run interleaved on the same machine; | ||
| medians, lower is better. |
There was a problem hiding this comment.
The claim that all five builds ran interleaved no longer describes these tables: this change replaces the Pi-Bolt values across macOS and Windows and several Linux rows while retaining the prior 0.8.0 comparator values; only four Linux scenarios were rerun alongside comparators. Since interleaving is what controls for changes in machine conditions, either rerun every compared build or state which values come from the earlier run.
Useful? React with 👍 / 👎.
| | CPU, one prompt (`pi -p`) | **80 ms** | 261 ms | 1,752 ms | 323 ms | 618 ms | | ||
| | Peak memory, one prompt (`pi -p`) | 124 MB | **55 MB** | 468 MB | 133 MB | 135 MB | | ||
| | Ready to type | 45 ms | **38 ms** | 988 ms | 128 ms | 316 ms | | ||
| | Writing a 200 KB file through a tool call | **0.5 s** | 92 s | 3.7 s | 26.9 s | 37.2 s | |
There was a problem hiding this comment.
The new 0.8.5 values cannot be audited through the README's advertised method/raw-data link: docs/BENCHMARKS.md still documents only 0.8.0, and a repository-wide check of bench/results finds no 0.8.5 result directory. Add the result files and update the methodology document, or avoid presenting these numbers as covered by that link.
Useful? React with 👍 / 👎.
Pi-Bolt's columns in the README's Linux, macOS and Windows tables are 0.8.5's, measured on the same machines as before. On Linux, Pi on Bun, Pi on Node and PiG were measured again alongside it for the streaming, session CPU, one-prompt and ready-to-type rows.