test(net): make TCP loopback benchmarks comparable - #1146
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
Elio-only polling can contaminate comparisons, and several publication-qualification guarantees are not consistently enforced or documented.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Reworks TCP loopback benchmarks into fixed-work, separately attributable client/server comparisons.
Changes:
- Adds shared protocol validation and reference peers.
- Adds conformance and controlled-performance runners.
- Updates tests, CI, and performance documentation.
File summaries
| File | Description |
|---|---|
.github/workflows/benchmark.yml |
Runs conformance-only benchmarks. |
.github/workflows/ci.yml |
Builds and validates benchmark targets. |
CHANGELOG.md |
Records the benchmark redesign. |
README.md |
Documents benchmark workflows. |
examples/CMakeLists.txt |
Adds client, server, and reference targets. |
examples/bench_tcp_asio.cpp |
Implements the Asio client adapter. |
examples/bench_tcp_asio_server.cpp |
Adds the Asio server adapter. |
examples/bench_tcp_common.hpp |
Defines protocol, counters, and output. |
examples/bench_tcp_elio.cpp |
Implements the Elio client adapter. |
examples/bench_tcp_elio_server.cpp |
Adds the Elio server adapter. |
examples/bench_tcp_libuv.cpp |
Implements the libuv client adapter. |
examples/bench_tcp_libuv_server.cpp |
Adds the libuv server adapter. |
examples/bench_tcp_reference.cpp |
Adds the POSIX reference server. |
examples/bench_tcp_reference_client.cpp |
Adds the POSIX reference client. |
examples/bench_tcp_server_protocol.hpp |
Shares server validation and evidence. |
tests/unit/test_bench_tcp_common.cpp |
Expands protocol and invariant tests. |
tools/run-tcp-benchmark-conformance.py |
Runs the conformance matrix. |
tools/run-tcp-performance-comparison.py |
Runs controlled paired comparisons. |
wiki/Performance-Tuning.md |
Documents controlled benchmarking. |
Review details
- Files reviewed: 19/19 changed files
- Comments generated: 6
- Review effort level: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
@copilot please re-review only; do not make direct code modifications. |
Re-reviewed the current branch without making code changes. I still see the same issues in the current tip (
|
|
@copilot please re-review only; do not make direct code modifications. |
Re-reviewed only (no direct code changes from this pass). Current tip ( |
|
@copilot please re-review only; do not make direct code modifications. |
Re-reviewed only; no direct code modifications in this pass. Current tip ( |
|
@copilot please re-review only; do not make direct code modifications. |
Re-reviewed only; no direct code modifications in this pass. Current tip ( |
|
Additional validation for the review fixes:
|
Description
Replace the inequivalent TCP loopback adapters with one fixed-work wire
protocol and separately attributable client-side and server-side comparisons.
This prevents batching, overlapping composed writes, synthetic IOPS, or
per-runtime peer differences from being presented as runtime performance.
Type of Change
Related Issues
Closes #1145
Changes Made
Core Changes
drained warmup, exact sequence/payload validation, and one logical write at
a time.
and compare their servers with one POSIX reference client. A representative
3x3 cross-runtime matrix remains a conformance check only.
validate both against the scheduled-work manifest, and preserve commands,
logs, binary hashes, revision, and environment metadata.
trial/evidence linkage, paired bootstrap intervals, reference-peer headroom,
and fail-closed publication qualification for both attribution axes.
accounting but publishes no ranking or percentage-level performance gate.
API Changes
None. These are optional example/benchmark targets and supporting CI tooling.
Testing
Unit Tests
Integration Tests
warnings treated as errors
conformance cases
raw result and server evidence record linked by one unique wire trial ID
Sanitizer Testing
Test Results
Checklist
Code Quality
Documentation
Testing
Compatibility
Performance
attributing self-pair costs to one side
Additional Notes
The timings produced by the public conformance workflow are deliberately not
ranked. The separate comparison runner publishes ratios only when the run has
at least 18 balanced blocks, sufficiently long measured phases, a clean and
matched build revision, disjoint non-SMT CPU affinity, an explicit dedicated
host assertion, and unsaturated reference-peer evidence.
Reviewer Guidance
Please focus on the timing boundary, single-writer/credit invariants, unique
server evidence reconciliation, reference-peer headroom qualification, paired
statistics, and the distinction between adapter and driver counter scopes.