Skip to content

Add parser benchmark suite - #37

Merged
gen740 merged 15 commits into
mainfrom
codex/benchmarks
May 12, 2026
Merged

gen740 merged 15 commits into
mainfrom
codex/benchmarks

Conversation

@gen740

@gen740 gen740 commented May 12, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • Add a standalone bench/ CMake project comparing cli20, CLI11, argparse, cxxopts, and Boost.Program_options.
  • Add runtime, binary size, stripped size, allocation count, and hyperfine compile-time benchmarks.
  • Run benchmark reports for both C++20 and C++26 via nix build .#bench.
  • Expand many_options to include string, int, double, path, and enum-like conversions.

User-Facing API Changes

  • Token::position now uses std::size_t to avoid truncating argv positions.
  • TokenizeResult keeps tokenizer storage configurable through CLI20_TOKEN_STACK_CAPACITY.
  • tokenize() now accepts option_nargs / is_command callables instead of prebuilt spec maps.
  • parse_or_exit output overloads are split into stdio and iostream headers so parser.hh avoids iostream output.
  • TokenizerConfig remains non-owning, but prefix/separator customization now uses StringLiteral template arguments to avoid dangling views.
  • Description remains non-owning and now accepts string literals only; runtime-owned descriptions should be represented by a separate explicit owning API if needed.

Validation

  • nix develop -c cmake --build build-bench
  • BENCH_CXX_STANDARD=26 nix develop -c cmake --build build-bench/cxx26
  • BENCH_CXX_STANDARD=26 nix develop -c bench/scripts/run-runtime.sh --benchmark_filter=Simple --benchmark_min_time=0.01s
  • nix build -L .#bench
  • nix run .#build

gen740 added 9 commits May 12, 2026 19:58
Replace heap-allocated containers in the tokenizer with stack-buffered
alternatives and eliminate the overhead of building temporary maps for
option lookup. Use bit-packed prefix representations and smaller integer
types in tokens to reduce memory footprint.

- Use SmallVector for tokens to avoid heap usage in typical cases.
- Replace map-based lookup with on-the-fly field iteration.
- Refactor TokenizerConfig to use constexpr and std::string_view.
- Update Description to use std::string_view instead of std::string.
- Configure Nix shell with venvShellHook for Python environments.
Invoke benchmark scripts using explicit interpreters and switch
google-benchmark to libcxxStdenv in the Nix flake for better C++
standard support and environment consistency.
@gen740
gen740 marked this pull request as ready for review May 12, 2026 16:25
@gen740 gen740 self-assigned this May 12, 2026
@gen740
gen740 requested a review from Copilot May 12, 2026 16:46

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces a standalone benchmark suite (runtime, size, and compile-time) for comparing cli20 against several other C++ CLI libraries, and wires the generated results into the documentation site. To support more meaningful allocation/size measurements, it also refactors parts of the parser/tokenizer and removes std::format usage from several core headers.

Changes:

  • Add a bench/ CMake project plus scripts to run runtime/size/compile-time benchmarks and summarize/export results.
  • Add documentation UI (React component + CSS + MDX page) to display benchmark results from a generated JSON dataset.
  • Refactor tokenizer/config/token storage and parse_or_exit output handling to reduce dependencies (notably std::format) and support benchmarking.

Reviewed changes

Copilot reviewed 74 out of 75 changed files in this pull request and generated 7 comments.

Show a summary per file
File Description
tests/test_tokenize.cc Updates tokenize test harness to new lookup-based tokenize API and prefix accessor helper.
tests/compile_pass/token_stack_capacity.cc Adds compile-pass coverage for CLI20_TOKEN_STACK_CAPACITY override.
tests/CMakeLists.txt Adds try_compile-based compile-pass tests.
include/cli/parser.hh Refactors tokenizer API, token prefix representation, small-vector token storage, and parse_or_exit output behavior.
include/cli/parser_stdio.hh Adds parse_or_exit overload using FILE* destinations.
include/cli/parser_iostream.hh Adds parse_or_exit overload using iostream destinations.
include/cli/help.hh Removes std::format usage in help text generation.
include/cli/cli.hh Simplifies umbrella include to parser.hh only.
include/cli/argument.hh Changes Description representation (now string_view) to reduce dependency/size.
include/cli/action.hh Replaces std::format-based formatting with to_chars/stream-based formatting for error subjects/messages.
flake.nix Adds Nix packaging for benchmark dependencies and a bench build target/app.
examples/custom_action.cc Removes std::formatter example and adjusts output formatting accordingly.
docs/src/data/benchmarks/results.json Adds benchmark result dataset consumed by docs UI.
docs/src/css/custom.css Adds styling for benchmark charts/table UI.
docs/src/components/BenchmarkCharts.jsx Adds React component to render benchmark charts and a detailed results table.
docs/sidebars.ts Adds “benchmarks” page to docs sidebar.
docs/docs/benchmarks.mdx Adds benchmarks documentation page embedding the new charts component.
docs/docs/actions.mdx Updates docs example strings to remove std::format usage.
bench/size/subcommand/cxxopts.cc Adds size benchmark case implementation for cxxopts (subcommand).
bench/size/subcommand/cli20.cc Adds size benchmark case implementation for cli20 (subcommand).
bench/size/subcommand/cli11.cc Adds size benchmark case implementation for CLI11 (subcommand).
bench/size/subcommand/boost.cc Adds size benchmark case implementation for Boost.Program_options (subcommand).
bench/size/subcommand/argparse.cc Adds size benchmark case implementation for argparse (subcommand).
bench/size/simple/cxxopts.cc Adds size benchmark case implementation for cxxopts (simple).
bench/size/simple/cli20.cc Adds size benchmark case implementation for cli20 (simple).
bench/size/simple/cli11.cc Adds size benchmark case implementation for CLI11 (simple).
bench/size/simple/boost.cc Adds size benchmark case implementation for Boost.Program_options (simple).
bench/size/simple/argparse.cc Adds size benchmark case implementation for argparse (simple).
bench/size/many_options/cxxopts.cc Adds size benchmark case implementation for cxxopts (many_options).
bench/size/many_options/cli20.cc Adds size benchmark case implementation for cli20 (many_options).
bench/size/many_options/cli11.cc Adds size benchmark case implementation for CLI11 (many_options).
bench/size/many_options/boost.cc Adds size benchmark case implementation for Boost.Program_options (many_options).
bench/size/many_options/argparse.cc Adds size benchmark case implementation for argparse (many_options).
bench/size/CMakeLists.txt Adds CMake targets for building size benchmark executables across cases/libraries.
bench/scripts/summarize.py Adds benchmark report generator producing Markdown tables from result artifacts.
bench/scripts/run-size.sh Adds script to build size targets, strip binaries, and emit TSV results.
bench/scripts/run-runtime.sh Adds script to build and run Google Benchmark runtime suite and emit JSON results.
bench/scripts/export-doc-data.py Adds script to normalize runtime/size/compile-time results into docs JSON schema.
bench/runtime/subcommand.cc Adds runtime benchmarks (and allocation measurement) for subcommand schema.
bench/runtime/simple.cc Adds runtime benchmarks (and allocation measurement) for simple schema.
bench/runtime/many_options.cc Adds runtime benchmarks (and allocation measurement) for many_options schema.
bench/runtime/CMakeLists.txt Adds runtime benchmark executable target linking required libs.
bench/runtime/allocation_count.cc Overrides global new/delete to count allocations during benchmarks.
bench/results/.gitkeep Keeps results directory present in the repo.
bench/results/.gitignore Ignores generated benchmark artifacts under bench/results/.
bench/include/bench/many_options.hh Adds shared types/helpers for many_options benchmark schema.
bench/include/bench/argv.hh Adds shared argv storage/helper for benchmarks.
bench/include/bench/allocation_counter.hh Adds allocation counting utilities integrated with Google Benchmark counters.
bench/compile_time/scripts/run.sh Adds hyperfine-driven compile-time benchmark runner producing TSV/JSON outputs.
bench/compile_time/scripts/compile-one.sh Adds script to compile one case/library TU into an object for timing.
bench/compile_time/cases/subcommand/cxxopts.cc Reuses size benchmark source for compile-time measurements (subcommand/cxxopts).
bench/compile_time/cases/subcommand/cli20.cc Reuses size benchmark source for compile-time measurements (subcommand/cli20).
bench/compile_time/cases/subcommand/cli11.cc Reuses size benchmark source for compile-time measurements (subcommand/cli11).
bench/compile_time/cases/subcommand/boost.cc Reuses size benchmark source for compile-time measurements (subcommand/boost).
bench/compile_time/cases/subcommand/argparse.cc Reuses size benchmark source for compile-time measurements (subcommand/argparse).
bench/compile_time/cases/simple/cxxopts.cc Reuses size benchmark source for compile-time measurements (simple/cxxopts).
bench/compile_time/cases/simple/cli20.cc Reuses size benchmark source for compile-time measurements (simple/cli20).
bench/compile_time/cases/simple/cli11.cc Reuses size benchmark source for compile-time measurements (simple/cli11).
bench/compile_time/cases/simple/boost.cc Reuses size benchmark source for compile-time measurements (simple/boost).
bench/compile_time/cases/simple/argparse.cc Reuses size benchmark source for compile-time measurements (simple/argparse).
bench/compile_time/cases/many_options/cxxopts.cc Reuses size benchmark source for compile-time measurements (many_options/cxxopts).
bench/compile_time/cases/many_options/cli20.cc Reuses size benchmark source for compile-time measurements (many_options/cli20).
bench/compile_time/cases/many_options/cli11.cc Reuses size benchmark source for compile-time measurements (many_options/cli11).
bench/compile_time/cases/many_options/boost.cc Reuses size benchmark source for compile-time measurements (many_options/boost).
bench/compile_time/cases/many_options/argparse.cc Reuses size benchmark source for compile-time measurements (many_options/argparse).
bench/CMakeLists.txt Adds top-level bench CMake project wiring runtime + size subdirs.
bench/cmake/BenchLibraries.cmake Adds CMake helper libraries and dependency discovery for benchmark builds.
.gitignore Adds build-bench/ to ignored build artifacts.
.github/workflows/windows-unicode.yml Updates checkout action version used in CI workflow.
.github/workflows/release.yml Updates checkout action version used in release workflow.
.github/workflows/release-docs.yml Updates checkout action version used in docs release workflow.
.github/workflows/docs-preview.yml Updates checkout action version used in docs preview workflow.
.github/workflows/deploy-nightly-docs.yml Updates checkout action version used in nightly docs deploy workflow.
.github/workflows/ci.yml Updates checkout action version used in main CI workflow.
.github/workflows/bump-version.yml Updates checkout action version used in bump-version workflow.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread include/cli/action.hh
Comment thread include/cli/parser.hh
Comment thread include/cli/parser.hh Outdated
Comment thread include/cli/parser.hh
Comment thread include/cli/parser.hh
Comment thread include/cli/argument.hh
Comment thread include/cli/parser.hh

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 74 out of 75 changed files in this pull request and generated 9 comments.

Comments suppressed due to low confidence (1)

include/cli/cli.hh:4

  • cli/cli.hh no longer includes cli/argument.hh, but the repository still has code (e.g. bench/size/many_options/cli20.cc) that includes <cli/cli.hh> and uses cli::StringOption/cli::Arg, which will fail to compile. Either restore #include "cli/argument.hh" in this umbrella header, or update all consumers to include cli/argument.hh explicitly (and consider documenting the change since it’s a breaking include-level API change).

Comment thread bench/size/many_options/cli20.cc
Comment thread include/cli/parser.hh
Comment thread include/cli/parser.hh
Comment thread include/cli/parser.hh
Comment thread include/cli/parser.hh
Comment thread include/cli/parser.hh Outdated
Comment thread include/cli/argument.hh
Comment thread include/cli/parser.hh
Comment thread include/cli/parser.hh
@gen740
gen740 force-pushed the codex/benchmarks branch from 7ea82b3 to 1e523bd Compare May 12, 2026 17:31
@gen740
gen740 enabled auto-merge (rebase) May 12, 2026 17:32
@gen740
gen740 merged commit 85c9cdd into main May 12, 2026
9 checks passed
@gen740
gen740 deleted the codex/benchmarks branch May 13, 2026 02:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants