Skip to content

varnish: implement static-tls, baseline-h2c, upload - #1422

Draft
gquintard wants to merge 4 commits into
MDA2AV:mainfrom
gquintard:varnish-static-tls-h2c-upload
Draft

varnish: implement static-tls, baseline-h2c, upload#1422
gquintard wants to merge 4 commits into
MDA2AV:mainfrom
gquintard:varnish-static-tls-h2c-upload

Conversation

@gquintard

Copy link
Copy Markdown
Contributor
  • static-tls: second TLS frontend on :8081 reusing the existing vmod-fileserver /static/* route.
  • baseline-h2c: second cleartext listener on :8082; feature=+http2 already negotiates h2 prior-knowledge on it.
  • upload: new CountingWriter + httparena.upload_count() vmod function, reads and discards the body without buffering it.

Description


PR Commands — comment on this PR to trigger (requires collaborator approval):

Command Description
/benchmark -f <framework> Run every test the framework subscribes to
/benchmark -f <framework> -t <test> Run one test only
/benchmark -f <framework> --save Run and save results (updates the leaderboard on merge)
/benchmark -f <framework> -t <test> --save Run one test and save results
/benchmark -f <framework> --compare <other> Measure the deltas against another framework instead of this one
/benchmark-multiple -f <fw1>,<fw2>,... Benchmark several frameworks in one run — takes -t and --save too; saved results land in a single commit
/benchmark-multiple --save No -f needed: benchmark and save every framework the PR touches
/benchmark-test -t <test> Benchmark all enabled frameworks subscribed to <test> and save the results

For /benchmark, always specify -f <framework>; the flags combine in any order. Results come back as a comment with a per-profile table of RPS, p99, CPU and memory — one table per framework on multi runs. A new benchmark comment while a run is in flight queues behind it (one deep) instead of cancelling it. For multi-framework PRs (dependency bumps, same-language refactors) prefer /benchmark-multiple, which runs everything in a single job and commits all saved results together, so no run overwrites another. --compare works on single-framework runs only.

What the deltas are measured against. By default, this framework's own results published on main - answering "did this change help?". When you are tuning a variant or a successor entry, --compare re-bases them on another entry instead:

/benchmark -f genhttp --compare genhttp-kestrel

The reply states which baseline it used, and profiles the other framework does not run show n/a rather than a delta.


Run benchmarks locally

You can validate and benchmark your framework locally with the lite script — no CPU pinning, fixed connection counts, all load generators run in Docker.

./scripts/validate.sh <framework>
./scripts/benchmark-lite.sh <framework> baseline
./scripts/benchmark-lite.sh --load-threads 4 <framework>

Requirements: Docker Engine on Linux. Load generators (gcannon, h2load, h2load-h3, wrk) are built as self-contained Docker images on first run.

- static-tls: second TLS frontend on :8081 reusing the existing
  vmod-fileserver /static/* route.
- baseline-h2c: second cleartext listener on :8082; feature=+http2
  already negotiates h2 prior-knowledge on it.
- upload: new CountingWriter + httparena.upload_count() vmod function,
  reads and discards the body without buffering it.
@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

👋 Heads up! This PR modifies the following frameworks:

upload was unscored and replaced by 8gbit upstream (MDA2AV#1373, MDA2AV#1382) before
this branch's meta.json change landed, so CI correctly rejected it as an
unknown profile. Removes the vmod's CountingWriter/upload_count and the
/upload VCL route along with it.

latency-10k already exists in the shared profile registry and drives the
same GET /baseline11 baseline already validates, so subscribing costs no
new code, same as latency-1m.
default.vcl had no vcl_backend_response, so fetched objects held Varnish's
default TTL (~120s) — well past the 30s staleness window infrastructure
entries get. static-tls's fileserver.root() backend is otherwise identical
to static-h2's, but only static-tls's suite runs the staleness probe, so
this was the first place it surfaced.

5s keeps this a real cache (repeat requests still served from memory) while
revalidating against the mounted directory inside the probe's window.
@gquintard

Copy link
Copy Markdown
Contributor Author

/benchmark -f varnish --save

@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

👋 Benchmark request received. A collaborator will review and approve the run.

@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results

Framework: varnish | Test: all tests

Test Conn RPS Rate CPU/req p99 p99.9 CPU Mem Δ RPS Δ Mem
baseline 4096 199,035 6256.6% 920MiB -2.8% +8.9%
pipelined 4096 279,036 6411.5% 830MiB -1.6% -37.7%
limited-conn 4096 183,208 6517.2% 798MiB -3.4% +1.7%
static-tls 1024 308,685 6291.3% 709MiB NEW NEW
baseline-h2 256 217,317 6466.0% 4.5GiB +1.6% ~0%
baseline-h2 1024 202,194 6508.8% 4.9GiB -1.9% +2.1%
static-h2 256 150,998 6566.5% 2.4GiB -0.4% +9.1%
static-h2 1024 139,986 6596.1% 4.6GiB -0.5% -2.1%
baseline-h2c 256 221,962 6405.4% 4.5GiB NEW NEW
baseline-h2c 1024 213,225 6405.1% 4.9GiB NEW NEW
baseline-h2c 4096 169,108 6318.2% 5.2GiB NEW NEW
latency-10k 1024 9,983 0.9983 77.52us 140.0us 250.0us 74.7% 656MiB NEW NEW
Full log
Process Request Failure:11363
Process Request Failure:10908
Process Request Failure:12669
Process Request Failure:11637
Process Request Failure:12147
Process Request Failure:11029
Process Request Failure:11407
Process Request Failure:11684
Process Request Failure:10853
Process Request Failure:12018
Process Request Failure:12324
Process Request Failure:12425
Process Request Failure:11105
Process Request Failure:11204
Process Request Failure:11415
Process Request Failure:11551
Process Request Failure:11464
Process Request Failure:10946
Process Request Failure:12434
Process Request Failure:11310
Process Request Failure:12124
Process Request Failure:11617
Process Request Failure:11378
Process Request Failure:12244
Process Request Failure:11484
Process Request Failure:12830
Process Request Failure:11685
Process Request Failure:11148
Process Request Failure:10448
Process Request Failure:11588
Process Request Failure:11249
Process Request Failure:12398
Process Request Failure:12416
Process Request Failure:10549
Process Request Failure:11215
Process Request Failure:12072
Process Request Failure:11785
Process Request Failure:11394
Process Request Failure:11039
Process Request Failure:11525
Process Request Failure:12366
Process Request Failure:11914
Process Request Failure:12181
Process Request Failure:11368
Process Request Failure:12282
Process Request Failure:11742
Process Request Failure:11426
Process Request Failure:11354
Process Request Failure:10449
Process Request Failure:12829
Process Request Failure:11505
Process Request Failure:11912
Process Request Failure:11510
Process Request Failure:12015
18. Stopping all clients.. Stopping all clients.63

55
. Stopping all clients.
. Stopping all clients.
. Stopping all clients.. Stopping all clients.17Stopped all clients for thread #54
. Stopping all clients.
4

. Stopping all clients.39. Stopping all clients.
. Stopping all clients.
. Stopping all clients.Stopped all clients for thread #17


163Stopped all clients for thread #7Stopped all clients for thread #Stopped all clients for thread #Stopped all clients for thread #23
2
33Stopped all clients for thread #2




. Stopping all clients.

finished in 5.10s, 165049.60 req/s, 17.29MB/s
requests: 1398547 total, 1596496 started, 1398547 done, 825248 succeeded, 573299 failed, 573299 errored, 0 timeout
status codes: 862453 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 86.45MB (90647537) total, 66.53MB (69765399) headers (space savings 27.05%), 805.91KB (825248) data
                     min         max         mean         sd        +/- sd
time for request:    12.59ms    950.36ms    103.48ms     99.79ms    96.24%
time for connect:       18us       1.02s    232.40ms    417.62ms    77.81%
time to 1st byte:    35.28ms       1.48s    919.78ms    419.22ms    44.51%
req/s           :       0.00      928.32       48.72      181.45    93.85%
[info] CPU 6318.2% | Mem 5.2GiB

=== Best: 169108 req/s (CPU: 6318.2%, Mem: 5.2GiB) ===
[info] saved results/baseline-h2c/4096/varnish.json
httparena-bench-varnish
httparena-bench-varnish
[info] skip: varnish does not subscribe to json-h2c
[info] skip: varnish does not subscribe to baseline-h3
[info] skip: varnish does not subscribe to static-h3
[info] skip: varnish does not subscribe to gateway-64
[info] skip: varnish does not subscribe to gateway-h3
[info] skip: varnish does not subscribe to production-stack
[info] skip: varnish does not subscribe to unary-grpc
[info] skip: varnish does not subscribe to unary-grpc-tls
[info] skip: varnish does not subscribe to echo-ws
[info] skip: varnish does not subscribe to echo-ws-pipeline
[info] skip: varnish does not subscribe to echo-ws-limited
[info] skip: varnish does not subscribe to latency-1m

==============================================
=== varnish / latency-10k / 1024c (tool=zrk) ===
==============================================
[info] waiting for server...
[info] server ready

[run 1/3]
{
  "zrk_version": "2.3.0",
  "target": { "url": "http://localhost:8080/baseline11?a=1&b=2", "method": "GET" },
  "config": { "connections": 1024, "launched": 1024, "duration_s": 20.000, "closed": false, "target_rate": 10000, "timeout_ms": 2000, "deadline_ms": 0, "deadline_abort": false, "record_timeouts": true },
  "duration_s": 20.003,
  "requests": 199602,
  "bytes": 32671906,
  "achieved_rate": 9978.74,
  "target_rate": 10000,
  "target_rate_end": 10000,
  "rate_ratio": 0.9979,
  "bytes_per_sec": 1633372.95,
  "error_rate": 0.000000,
  "max_schedule_lag_us": 0,
  "latency_us": {
    "min": 54, "mean": 115.5, "stdev": 869.4, "max": 65663,
    "p50": 88, "p75": 95, "p90": 103, "p99": 133, "p99_9": 7218, "p99_99": 39920
  },
  "status_codes": { "1xx": 0, "2xx": 199602, "3xx": 0, "4xx": 0, "5xx": 0 },
  "errors": { "connect": 0, "read": 0, "write": 0, "timeout": 0, "deadline": 0, "non_2xx_3xx": 0 },
  "latency_histogram": "HISTFAAAAwd4nC1SS0iUQRz/5vfN996H67qu6yOrTVTEYhMrMZWsqOghJUEPMkMqyCTqkIeIUpbUCJHsEngIM7AOy+YhQuzSISIkgqKgJDpEh4jwUBch6DfjzjAz/8fv/57K7GTcMGTaWF1m4RWamRwxOpZXBf2AIY3isvW7jgyLvMhiBUtyzvkTGi95vvZW49T22b2jRxaO3el50DPdm+tdOffz4sLV5Wufs+LlmJgbFV+HxI/Be5fe9r06+a578cDtPRM7c+0fW35vzjc+qZ9Lz6/Jld8smy/5Flssykemw7/9Gf+Dl3Pzzi971lqSr+W4nJJZmTf/mMNm1nyGGUxgUXwW38UvnjciJ+bEXdF75dCFLSc6Ozu31bVvqNu4oby0pHxtZWVpcWlxyI+4xaGyUFFRJOLakZAbkX4Iru+GQj583/V96dqs0ZWGH7EhbdgSJgySBtSW0jZJsg9k1OItqVbLIZKcNJWUl3TD5C2QLojiyhcd2o4SpmjhKIBHpYcKihzlhRITgbYyUUsuyjeBJM+Alm3l6SawCUcRI11DdAvqyZuowh7SHXTeRV0GjJmmyKCTfrwQjBNjJCaVwWHqmH6M20EfqoEwsfcFn2ocRzNaaRRgn47ZSugp7McnQcskcVGeAGhTThNgTiSTdFzN8GcItui9SjaRGhLYgYdCBU2SGiSghv4yeCQYOIHTvAMkZZTqOAvO0CaJEcFqE8Q5rKmCigRrqsU6NKCRtTVwmzjJXFUTozTqkCmGCLMzMVymxmNOBykPw+PEasm1YhN20zqgrpnyqKo4pvptUBanH1WYGnOasZJ8YzzKu0eMQ86TFpHV7LRTmJcHPSo1BoOJxvScPe08RWAbizV0887qsAFnUcH7vWAGXcBfgfNkB3CdkDizGxO4wRQN3UsLXwTraSKR4lYZ/gNboBpRxe0RFeZr6SFGqZaqH+pHQXpMxNKttXT+pqrWlI7+qZQ4hbo9XQcpO1j9pZb+yZYuPdDtVd+RyiiFMqAX/SPiyl+t6l4GjwVGLDyV+A/IqIAA"
}
[info] CPU 82.4% | Mem 628MiB

[run 2/3]
{
  "zrk_version": "2.3.0",
  "target": { "url": "http://localhost:8080/baseline11?a=1&b=2", "method": "GET" },
  "config": { "connections": 1024, "launched": 1024, "duration_s": 20.000, "closed": false, "target_rate": 10000, "timeout_ms": 2000, "deadline_ms": 0, "deadline_abort": false, "record_timeouts": true },
  "duration_s": 20.002,
  "requests": 199682,
  "bytes": 32687489,
  "achieved_rate": 9982.92,
  "target_rate": 10000,
  "target_rate_end": 10000,
  "rate_ratio": 0.9983,
  "bytes_per_sec": 1634181.97,
  "error_rate": 0.000000,
  "max_schedule_lag_us": 0,
  "latency_us": {
    "min": 56, "mean": 100.1, "stdev": 357.1, "max": 32895,
    "p50": 89, "p75": 97, "p90": 108, "p99": 140, "p99_9": 250, "p99_99": 20328
  },
  "status_codes": { "1xx": 0, "2xx": 199682, "3xx": 0, "4xx": 0, "5xx": 0 },
  "errors": { "connect": 0, "read": 0, "write": 0, "timeout": 0, "deadline": 0, "non_2xx_3xx": 0 },
  "latency_histogram": "HISTFAAAAkh4nC2QW0hUURSGz/7PPvdxbBwHpxkvmchoJuKEmkxiESUkIQMh3ctCInoQgiSJyGowESlBGoRCKsKghxDxIaEsCIxExEQqhhKSiKGHoYeILjD0z7Gzzj77P2uv9a29VnFi1K8oeKisP+r/Xbg/ozeU1u/rjh6oprcw2Hiwd0Ksioz6UZuzMt6XwXR5oq5/x4e24fhIZ/rw/JEHx2+eXju72DPX238lc10MJsRCv0j1TZ0f7F49+uvAXPuXvWu7F2NLTWPRbM2Pqq/l02VD4ZHgUGDZN7thNn/cm/Vc86TsJWvS+mZOmUlj3nxsrVhJe8IecO7ZT61JM2s8M98ZP/WUfldf0ZZlWh2WY+oTdRoDGMcfsSCSYtC1iydPde7bub0xWhXduim0JRTcXFxcUBqyS/UCj+m1pUmzdd00Cyl006MrutT5ApAKF4UOSVOkotOhIrd4RK1QuZsfeSiBhjLXkY8ALDwSOIGrAs+BLrTjHNq4uvBW4DPwV2BG4pWKFxKjAn2o53EL87tgoJasWwJN6MZtgRjZv4kRrFKOCM4APhlgMYVlVARkJbWGPW7ZML0GHKp2FOGS1HAIZdLAMWxkAQOV3MPwEdlBWJw5BpEVtBL6FNwRTM2jVRBZgQvsI8Jb5XyW226HG+aAx1SadAjzoVpaRDtE16KBA4owWnOnEssNLD+XoKCOaz+79LNoK2pIxTaawzupTNzFbzgXXkRrISeA9yreAM3kNeAymQ559Ryrn0OTcYY1oxqfBJu9L6heC/wDL/J8qA=="
}
[info] CPU 74.7% | Mem 656MiB

[run 3/3]
{
  "zrk_version": "2.3.0",
  "target": { "url": "http://localhost:8080/baseline11?a=1&b=2", "method": "GET" },
  "config": { "connections": 1024, "launched": 1024, "duration_s": 20.000, "closed": false, "target_rate": 10000, "timeout_ms": 2000, "deadline_ms": 0, "deadline_abort": false, "record_timeouts": true },
  "duration_s": 20.002,
  "requests": 199673,
  "bytes": 32674439,
  "achieved_rate": 9982.45,
  "target_rate": 10000,
  "target_rate_end": 10000,
  "rate_ratio": 0.9982,
  "bytes_per_sec": 1633525.24,
  "error_rate": 0.000000,
  "max_schedule_lag_us": 0,
  "latency_us": {
    "min": 55, "mean": 104.1, "stdev": 462.5, "max": 31679,
    "p50": 89, "p75": 97, "p90": 108, "p99": 140, "p99_9": 408, "p99_99": 24344
  },
  "status_codes": { "1xx": 0, "2xx": 199673, "3xx": 0, "4xx": 0, "5xx": 0 },
  "errors": { "connect": 0, "read": 0, "write": 0, "timeout": 0, "deadline": 0, "non_2xx_3xx": 0 },
  "latency_histogram": "HISTFAAAAj14nC1QX0hTURy+57vn7u7ebc55dVNa2h9J1xwrUiqFbGCE9GAxaFkY7kko7EWIHoLGmAiy/qAJg6RApEh6ilExRHoIHNGDWBSMHiR6GlI+riHUd6+dc8/9nd93vu93vt8J5+YtRcEzZW+o/6NwkvkZZXBnD5gEFCmbD7T3nr2+JPLIqGX5WH/uq7RU27ciX/oKidpwJlm4vHP1x9jT9K+JzVtbd4r31qbFdk7Uc+LB3eLUxo2l8bnRT5fyF+pDuTPVU9nejXjp6GrnWkcpvNs6G1yxlpvq/o8Ni9533hWzajwyP7i/6zl3Wa/o23rBeG/UjZoxaxaNVfeuXtPr2mtX1jWnZbSK/Km+UpfVNygjj01REl/FosiK2+MXr/T3Dx3r6TxyKNzdHXa3hsM9weamYEOjl8NsNN0SLhdczk8qplQgoXAyBwxpEIAGVdoYNBnAOWAYAUQxAg98OIgphGQXJog8FATG8Ffgm8ATgfvAb4EFQMcLA2m81PBWRQqfJdoQQx8GMS1Y/Tg364KsCPVxdJCaxmkWjMHPXRcsHMY+8K44QvgjqDRwAqO0NEIrKWYjZLJSgGGAMUq5RVgn3+egGpfFa/0UhTjt0i08Z1dtbJQsXeqwp58GDC5bEYVK9X6eXiObhzH68FORYLQvGOAjeGhsQZCbIoSovEmRXVDl4udnokGGKOfzWY4njeY9LGOwqsq2LbYzaTdg2TZlih1GcZ5HJ+2SuuPF57AShD3MIkgyJkmZcd76H5KveBI="
}
[info] CPU 75.5% | Mem 666MiB
[info] run 2 wins on score

=== Best: 9983 req/s (CPU: 74.7%, Mem: 656MiB) ===
[info] exact CPU: 15.48 core-seconds | 77.524 us/req
=== Fixed-rate: cpu/req 77.52us | p99 140.0us | p99.9 250.0us | rate_ratio 0.9983 ===
[info] saved results/latency-10k/1024/varnish.json
httparena-bench-varnish
httparena-bench-varnish
[info] skip: varnish does not subscribe to async
[info] rebuilding site/data/*.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/frameworks.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/results/varnish.json - 12 new, 12 total
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/current.json
[info] done
[info] restoring loopback MTU to 65536

@gquintard
gquintard marked this pull request as draft September 2, 2026 05:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant