Repository navigation
Add parser benchmark suite - #37
Merged
Merged
Conversation
Replace heap-allocated containers in the tokenizer with stack-buffered alternatives and eliminate the overhead of building temporary maps for option lookup. Use bit-packed prefix representations and smaller integer types in tokens to reduce memory footprint. - Use SmallVector for tokens to avoid heap usage in typical cases. - Replace map-based lookup with on-the-fly field iteration. - Refactor TokenizerConfig to use constexpr and std::string_view. - Update Description to use std::string_view instead of std::string. - Configure Nix shell with venvShellHook for Python environments.
Invoke benchmark scripts using explicit interpreters and switch google-benchmark to libcxxStdenv in the Nix flake for better C++ standard support and environment consistency.
gen740
marked this pull request as ready for review
May 12, 2026 16:25
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces a standalone benchmark suite (runtime, size, and compile-time) for comparing cli20 against several other C++ CLI libraries, and wires the generated results into the documentation site. To support more meaningful allocation/size measurements, it also refactors parts of the parser/tokenizer and removes std::format usage from several core headers.
Changes:
- Add a
bench/CMake project plus scripts to run runtime/size/compile-time benchmarks and summarize/export results. - Add documentation UI (React component + CSS + MDX page) to display benchmark results from a generated JSON dataset.
- Refactor tokenizer/config/token storage and
parse_or_exitoutput handling to reduce dependencies (notablystd::format) and support benchmarking.
Reviewed changes
Copilot reviewed 74 out of 75 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_tokenize.cc | Updates tokenize test harness to new lookup-based tokenize API and prefix accessor helper. |
| tests/compile_pass/token_stack_capacity.cc | Adds compile-pass coverage for CLI20_TOKEN_STACK_CAPACITY override. |
| tests/CMakeLists.txt | Adds try_compile-based compile-pass tests. |
| include/cli/parser.hh | Refactors tokenizer API, token prefix representation, small-vector token storage, and parse_or_exit output behavior. |
| include/cli/parser_stdio.hh | Adds parse_or_exit overload using FILE* destinations. |
| include/cli/parser_iostream.hh | Adds parse_or_exit overload using iostream destinations. |
| include/cli/help.hh | Removes std::format usage in help text generation. |
| include/cli/cli.hh | Simplifies umbrella include to parser.hh only. |
| include/cli/argument.hh | Changes Description representation (now string_view) to reduce dependency/size. |
| include/cli/action.hh | Replaces std::format-based formatting with to_chars/stream-based formatting for error subjects/messages. |
| flake.nix | Adds Nix packaging for benchmark dependencies and a bench build target/app. |
| examples/custom_action.cc | Removes std::formatter example and adjusts output formatting accordingly. |
| docs/src/data/benchmarks/results.json | Adds benchmark result dataset consumed by docs UI. |
| docs/src/css/custom.css | Adds styling for benchmark charts/table UI. |
| docs/src/components/BenchmarkCharts.jsx | Adds React component to render benchmark charts and a detailed results table. |
| docs/sidebars.ts | Adds “benchmarks” page to docs sidebar. |
| docs/docs/benchmarks.mdx | Adds benchmarks documentation page embedding the new charts component. |
| docs/docs/actions.mdx | Updates docs example strings to remove std::format usage. |
| bench/size/subcommand/cxxopts.cc | Adds size benchmark case implementation for cxxopts (subcommand). |
| bench/size/subcommand/cli20.cc | Adds size benchmark case implementation for cli20 (subcommand). |
| bench/size/subcommand/cli11.cc | Adds size benchmark case implementation for CLI11 (subcommand). |
| bench/size/subcommand/boost.cc | Adds size benchmark case implementation for Boost.Program_options (subcommand). |
| bench/size/subcommand/argparse.cc | Adds size benchmark case implementation for argparse (subcommand). |
| bench/size/simple/cxxopts.cc | Adds size benchmark case implementation for cxxopts (simple). |
| bench/size/simple/cli20.cc | Adds size benchmark case implementation for cli20 (simple). |
| bench/size/simple/cli11.cc | Adds size benchmark case implementation for CLI11 (simple). |
| bench/size/simple/boost.cc | Adds size benchmark case implementation for Boost.Program_options (simple). |
| bench/size/simple/argparse.cc | Adds size benchmark case implementation for argparse (simple). |
| bench/size/many_options/cxxopts.cc | Adds size benchmark case implementation for cxxopts (many_options). |
| bench/size/many_options/cli20.cc | Adds size benchmark case implementation for cli20 (many_options). |
| bench/size/many_options/cli11.cc | Adds size benchmark case implementation for CLI11 (many_options). |
| bench/size/many_options/boost.cc | Adds size benchmark case implementation for Boost.Program_options (many_options). |
| bench/size/many_options/argparse.cc | Adds size benchmark case implementation for argparse (many_options). |
| bench/size/CMakeLists.txt | Adds CMake targets for building size benchmark executables across cases/libraries. |
| bench/scripts/summarize.py | Adds benchmark report generator producing Markdown tables from result artifacts. |
| bench/scripts/run-size.sh | Adds script to build size targets, strip binaries, and emit TSV results. |
| bench/scripts/run-runtime.sh | Adds script to build and run Google Benchmark runtime suite and emit JSON results. |
| bench/scripts/export-doc-data.py | Adds script to normalize runtime/size/compile-time results into docs JSON schema. |
| bench/runtime/subcommand.cc | Adds runtime benchmarks (and allocation measurement) for subcommand schema. |
| bench/runtime/simple.cc | Adds runtime benchmarks (and allocation measurement) for simple schema. |
| bench/runtime/many_options.cc | Adds runtime benchmarks (and allocation measurement) for many_options schema. |
| bench/runtime/CMakeLists.txt | Adds runtime benchmark executable target linking required libs. |
| bench/runtime/allocation_count.cc | Overrides global new/delete to count allocations during benchmarks. |
| bench/results/.gitkeep | Keeps results directory present in the repo. |
| bench/results/.gitignore | Ignores generated benchmark artifacts under bench/results/. |
| bench/include/bench/many_options.hh | Adds shared types/helpers for many_options benchmark schema. |
| bench/include/bench/argv.hh | Adds shared argv storage/helper for benchmarks. |
| bench/include/bench/allocation_counter.hh | Adds allocation counting utilities integrated with Google Benchmark counters. |
| bench/compile_time/scripts/run.sh | Adds hyperfine-driven compile-time benchmark runner producing TSV/JSON outputs. |
| bench/compile_time/scripts/compile-one.sh | Adds script to compile one case/library TU into an object for timing. |
| bench/compile_time/cases/subcommand/cxxopts.cc | Reuses size benchmark source for compile-time measurements (subcommand/cxxopts). |
| bench/compile_time/cases/subcommand/cli20.cc | Reuses size benchmark source for compile-time measurements (subcommand/cli20). |
| bench/compile_time/cases/subcommand/cli11.cc | Reuses size benchmark source for compile-time measurements (subcommand/cli11). |
| bench/compile_time/cases/subcommand/boost.cc | Reuses size benchmark source for compile-time measurements (subcommand/boost). |
| bench/compile_time/cases/subcommand/argparse.cc | Reuses size benchmark source for compile-time measurements (subcommand/argparse). |
| bench/compile_time/cases/simple/cxxopts.cc | Reuses size benchmark source for compile-time measurements (simple/cxxopts). |
| bench/compile_time/cases/simple/cli20.cc | Reuses size benchmark source for compile-time measurements (simple/cli20). |
| bench/compile_time/cases/simple/cli11.cc | Reuses size benchmark source for compile-time measurements (simple/cli11). |
| bench/compile_time/cases/simple/boost.cc | Reuses size benchmark source for compile-time measurements (simple/boost). |
| bench/compile_time/cases/simple/argparse.cc | Reuses size benchmark source for compile-time measurements (simple/argparse). |
| bench/compile_time/cases/many_options/cxxopts.cc | Reuses size benchmark source for compile-time measurements (many_options/cxxopts). |
| bench/compile_time/cases/many_options/cli20.cc | Reuses size benchmark source for compile-time measurements (many_options/cli20). |
| bench/compile_time/cases/many_options/cli11.cc | Reuses size benchmark source for compile-time measurements (many_options/cli11). |
| bench/compile_time/cases/many_options/boost.cc | Reuses size benchmark source for compile-time measurements (many_options/boost). |
| bench/compile_time/cases/many_options/argparse.cc | Reuses size benchmark source for compile-time measurements (many_options/argparse). |
| bench/CMakeLists.txt | Adds top-level bench CMake project wiring runtime + size subdirs. |
| bench/cmake/BenchLibraries.cmake | Adds CMake helper libraries and dependency discovery for benchmark builds. |
| .gitignore | Adds build-bench/ to ignored build artifacts. |
| .github/workflows/windows-unicode.yml | Updates checkout action version used in CI workflow. |
| .github/workflows/release.yml | Updates checkout action version used in release workflow. |
| .github/workflows/release-docs.yml | Updates checkout action version used in docs release workflow. |
| .github/workflows/docs-preview.yml | Updates checkout action version used in docs preview workflow. |
| .github/workflows/deploy-nightly-docs.yml | Updates checkout action version used in nightly docs deploy workflow. |
| .github/workflows/ci.yml | Updates checkout action version used in main CI workflow. |
| .github/workflows/bump-version.yml | Updates checkout action version used in bump-version workflow. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 74 out of 75 changed files in this pull request and generated 9 comments.
Comments suppressed due to low confidence (1)
include/cli/cli.hh:4
cli/cli.hhno longer includescli/argument.hh, but the repository still has code (e.g.bench/size/many_options/cli20.cc) that includes<cli/cli.hh>and usescli::StringOption/cli::Arg, which will fail to compile. Either restore#include "cli/argument.hh"in this umbrella header, or update all consumers to includecli/argument.hhexplicitly (and consider documenting the change since it’s a breaking include-level API change).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
bench/CMake project comparing cli20, CLI11, argparse, cxxopts, and Boost.Program_options.nix build .#bench.many_optionsto include string, int, double, path, and enum-like conversions.User-Facing API Changes
Token::positionnow usesstd::size_tto avoid truncating argv positions.TokenizeResultkeeps tokenizer storage configurable throughCLI20_TOKEN_STACK_CAPACITY.tokenize()now acceptsoption_nargs/is_commandcallables instead of prebuilt spec maps.parse_or_exitoutput overloads are split into stdio and iostream headers soparser.hhavoids iostream output.TokenizerConfigremains non-owning, but prefix/separator customization now usesStringLiteraltemplate arguments to avoid dangling views.Descriptionremains non-owning and now accepts string literals only; runtime-owned descriptions should be represented by a separate explicit owning API if needed.Validation
nix develop -c cmake --build build-benchBENCH_CXX_STANDARD=26 nix develop -c cmake --build build-bench/cxx26BENCH_CXX_STANDARD=26 nix develop -c bench/scripts/run-runtime.sh --benchmark_filter=Simple --benchmark_min_time=0.01snix build -L .#benchnix run .#build