From 1919fb13459475beb3d8b473288b27265bdb73ef Mon Sep 17 00:00:00 2001 From: Vishal Rana Date: Wed, 14 Oct 2026 17:12:11 -0700 Subject: [PATCH 1/4] feat: benchmark Fanout v2026.10.1-alpha on throwaway Hetzner VMs Port the harness to the current Fanout release and make a short-lived Hetzner Cloud VM the standard publication host. - Start Fanout with its environment-only configuration: one FANOUT_ADDR listener, chat disabled, and a per-run FANOUT_METRICS_TOKEN for the private /metrics endpoint. Setup must return the ingest token. - Accept release versions with a maturity suffix, such as v2026.10.1-alpha. - Replace the removed observability reads with the operations the UI issues: a System overview refresh (batched panel queries plus annotations), an exemplar drill-down, and trace detail within the discovered trace's own window. Latency covers the whole operation, any failed request or panel fails it, and the sustained read gate fails when an operation kind never succeeds. - scripts/hetzner.sh creates one ccx33 VM, runs one or more `--`-separated benchmark groups on it, and always deletes it. Hetzner bills every started hour, so a publication run shares one VM between its ingest-only and read-under-write groups. - Reports record the host description and redact the SSH address from the reproduction command. --- .github/workflows/test.yml | 2 + README.md | 75 +-- cmd/fanout-bench/main.go | 15 +- cmd/fanout-bench/main_test.go | 23 + docs/metrics.md | 16 +- docs/publishing.md | 28 +- internal/bench/managed.go | 62 ++- internal/bench/managed_test.go | 120 +++++ internal/bench/query.go | 173 +++++-- internal/bench/query_test.go | 18 +- internal/bench/query_workload.go | 317 ++++++++++++ internal/bench/query_workload_test.go | 454 ++++++++++++++++++ internal/bench/remote.go | 27 +- internal/bench/remote_test.go | 104 +++- internal/bench/report.go | 27 +- internal/bench/report_test.go | 76 ++- internal/bench/runner.go | 44 +- internal/bench/runner_test.go | 2 +- .../testdata/fanout-v2026.10.1-alpha-env.txt | 46 ++ .../system-overview-v2026.10.1-alpha.json | 347 +++++++++++++ internal/bench/types.go | 4 + justfile | 4 + scripts/hetzner.sh | 180 +++++++ 23 files changed, 2026 insertions(+), 138 deletions(-) create mode 100644 cmd/fanout-bench/main_test.go create mode 100644 internal/bench/managed_test.go create mode 100644 internal/bench/query_workload.go create mode 100644 internal/bench/query_workload_test.go create mode 100644 internal/bench/testdata/fanout-v2026.10.1-alpha-env.txt create mode 100644 internal/bench/testdata/system-overview-v2026.10.1-alpha.json create mode 100755 scripts/hetzner.sh diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 497249b..b250f1c 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -17,3 +17,5 @@ jobs: go-version: "1.27.x" - run: go test ./... - run: go vet ./... + - name: ShellCheck + run: shellcheck scripts/*.sh diff --git a/README.md b/README.md index 8b08c09..633d7c3 100644 --- a/README.md +++ b/README.md @@ -22,31 +22,43 @@ With pinned telemetrygen v0.157, each generated trace exports one parent and one ## Quick start -Prerequisites: Go 1.27+, [just](https://github.com/casey/just), SSH, and a clean SSH-accessible Linux host. The host may be a cloud instance, dedicated machine, or bare metal. +Prerequisites: [hcloud](https://github.com/hetznercloud/cli) with an active context (or `HCLOUD_TOKEN`), a registered Hetzner SSH key available to your SSH client, Go 1.27+, [just](https://github.com/casey/just), and SSH/SCP. ```sh -just build +export FANOUT_BENCH_HCLOUD_SSH_KEY='benchmark-key' +just hetzner --fanout-version v2026.10.1-alpha --preset quick +``` + +The script runs `just build` before provisioning, creates a fresh Hetzner Cloud VM with a random name suffix, waits for SSH, runs the harness for each group, downloads the evidence, and attempts deletion on every exit, including failures and Ctrl-C. If deletion fails after three attempts, it prints the server name for manual deletion and exits nonzero. Hetzner bills every started hour, so grouping runs in one invocation shares that VM time. + +The defaults are `ccx33` (8 dedicated vCPUs, 32 GB RAM), `fsn1`, and `ubuntu-24.04`. Override them with `FANOUT_BENCH_HCLOUD_TYPE`, `FANOUT_BENCH_HCLOUD_LOCATION`, and `FANOUT_BENCH_HCLOUD_IMAGE`. `FANOUT_BENCH_HCLOUD_SSH_KEY` also accepts comma-separated key names. Each invocation shares one temporary known-hosts file across its groups, leaving your SSH known-hosts file untouched. Pass `--ssh-identity ~/.ssh/benchmark` if the registered key is not available through your SSH agent. +Separate complete sets of `fanout-bench run` flags with a standalone `--`. Groups must be nonempty and use matching SSH port and identity settings (defaults: port 22 and the SSH client's default identity). All groups run in order even if one fails, with a one-line summary per group at the end and a nonzero exit if any failed. They share one VM and host description; each harness run creates a fresh remote directory and Fanout data directory. The VM is deleted after the last group. + +Any clean SSH-accessible Linux host is also supported: + +```sh +just build ./bin/fanout-bench run \ - --host root@203.0.113.10 \ - --fanout-version v2026.8.11 \ - --preset standard \ - --query-rate 50 + --host 'root@' \ + --host-description 'Linux benchmark host' \ + --fanout-version v2026.10.1-alpha \ + --preset quick ``` -That one command: +Replace `` with the address of your host. The remote harness: 1. detects the host's Linux architecture over SSH; 2. downloads the matching Fanout release and verifies it against the release `SHA256SUMS`; 3. cross-builds pinned telemetrygen and the benchmark agent for the host; 4. uploads all three binaries to a private, isolated `/tmp/fanout-bench-*` directory; -5. starts a fresh Fanout data directory on loopback-only random ports; +5. starts a fresh Fanout data directory on one loopback-only random port; 6. creates one-time admin, ingest, and query credentials without printing them; 7. runs every phase and then `fanout repair verify`; 8. downloads `report.json`, `report.md`, and `samples.jsonl` locally; 9. stops Fanout and removes the remote run directory. -Pass `--ssh-identity ~/.ssh/bench` or `--ssh-port 2222` when needed. `--keep-remote` preserves the isolated remote directory for investigation; teardown is the default. +For an existing host, pass `--ssh-identity ~/.ssh/benchmark`, `--ssh-port 2222`, or `--ssh-known-hosts ` when needed. `--host-description ` records the provider or machine configuration beside the detected CPU model. Reports redact the SSH address and omit the run-scoped known-hosts path from reproduction commands. `--keep-remote` preserves the isolated directory on an existing host for investigation; the Hetzner script always deletes its VM. Nothing is installed system-wide. Existing Fanout services and data directories are not touched. The remote host needs no Go toolchain, Docker daemon, or preconfigured Fanout account. @@ -62,16 +74,17 @@ Nothing is installed system-wide. Existing Fanout services and data directories The default topology is 8 workers per process at 2,500 items/s per worker with 1,000-item OTLP batches. A mixed step launches the same topology independently for traces, logs, and metrics. Without read load, the sustained proof reruns the mixed topology with the highest **accepted** throughput. With `--query-rate`, it chooses the highest-throughput topology that also completed at least 95% of the requested reads with no errors or scheduler shedding. If no sweep point qualifies, it repeats the best read-completion point and fails the sustained-read quality gate rather than publishing a write-only peak as mixed capacity. -For a publication run, use at least three repetitions: +For publication runs, use the default ccx33 and three repetitions. Run ingest-only and read-under-write as two groups in one invocation on the same fresh VM; see [the publication commands](docs/publishing.md#publication-runs). For example: ```sh -./bin/fanout-bench run \ - --host root@bench-host \ - --fanout-version v2026.8.11 \ - --preset publish \ - --repeat 3 \ - --query-rate 50 \ - --output results/publish-2026-08-28 +scripts/hetzner.sh \ + --fanout-version v2026.10.1-alpha \ + --preset publish --repeat 3 \ + --output results/publish-ingest-2026-10-14 \ + -- \ + --fanout-version v2026.10.1-alpha \ + --preset publish --repeat 3 --query-rate 20 \ + --output results/publish-read-write-2026-10-14 ``` Provisioned runs deliberately place Fanout and generators on the same remote host, measuring the complete single-node product under load. The report labels this an observed same-host ceiling and warns that generator contention can make it lower than Fanout's isolated backend ceiling. @@ -81,21 +94,22 @@ Provisioned runs deliberately place Fanout and generators on the same remote hos The write-only suite answers “how quickly can Fanout ingest?” It does not answer “what happens to dashboard latency while ingesting?” Add automatically authenticated read load to mixed phases: ```sh -./bin/fanout-bench run \ - --host root@bench-host \ - --fanout-version v2026.8.11 \ +scripts/hetzner.sh \ + --fanout-version v2026.10.1-alpha \ --preset standard \ --query-rate 50 ``` -The harness uses a bounded open-loop scheduler and rotates across overview, topology, performance, trace, and log queries with 15-minute through 24-hour windows. It reports attempted and completed rates, phase-end cancellations, explicit scheduler shedding, failures, mean, p50, p95, p99, and maximum latency. The concurrency bound prevents a stalled server from creating unbounded requests. +The harness uses a bounded open-loop scheduler targeting user operations per second at 15-minute, 1-hour, 6-hour, 12-hour, and 24-hour windows. Operations rotate through dashboard refresh, service-map trace drill-down, and trace detail. A refresh sends the full System overview panel batch exactly as the UI's initial all-visible load does, then requests annotations over the returned time-panel bounds. Layout widths model a fixed 1664-pixel viewport and are rounded as in the UI. Drill-down uses `POST /api/panels/exemplars` with the selected service and the panel response's absolute window. Trace detail uses `GET /api/traces/:id` with a discovered ID, namespace, and that same window, retained separately for each range; an empty detail is a failure. Until a trace is discovered, detail slots perform drill-down; until a selectable panel is cached, those slots perform dashboard refresh instead. Each slot performs one user operation. The current System overview has no log drill or standalone log/trace search operation. Each scheduled read measures every request in its operation, and any request or panel error fails the operation. The report records per-operation counts and latency in its methodology alongside the dashboard definition hash; aggregate fields and quality gates retain their schema. + +It reports attempted and completed rates, phase-end cancellations, explicit scheduler shedding, failures, mean, p50, p95, p99, and maximum latency. The concurrency bound prevents a stalled server from creating unbounded requests. ## Output Every run produces one self-contained directory: ```text -results/20260828T190000Z/ +results/20261014T190000Z/ ├── report.json authoritative schema-versioned result ├── report.md readable, publication-ready report ├── samples.jsonl per-second Fanout resource samples @@ -109,10 +123,10 @@ The terminal output is optimized for decisions: ╭────────────────────────────────────────────────────────────────────╮ │ FANOUT BENCH • accepted-row performance, with evidence │ ╰────────────────────────────────────────────────────────────────────╯ - Target localhost:4317 (plaintext gRPC) + Target localhost:7520 (plaintext gRPC) Generator telemetrygen v0.157.0 same-host=true Suite standard repeat=1 18 phases ~5.2 min load time - Evidence Fanout counters @ /-/metrics samples every 1.0s + Evidence Fanout counters @ /metrics samples every 1.0s ◇ mixed-p4-r1 offered 240,000 items/s for 15s … accepted …/s ratio …% CPU … cores peak RSS … @@ -140,8 +154,8 @@ Regenerate Markdown without rerunning load: ```sh ./bin/fanout-bench render \ - --input results/publish-2026-08-28/report.json \ - --output results/publish-2026-08-28/report.md + --input results/publish-2026-10-14/report.json \ + --output results/publish-2026-10-14/report.md ``` ## Metrics captured @@ -189,20 +203,23 @@ The project uses only the Go standard library at runtime. The generator is a sep ## Existing local Fanout process -Remote provisioning is the publication path. For local development against an already-running Fanout, install telemetrygen, provide short-lived credentials through environment variables, and omit `--host`: +Fresh Hetzner Cloud VMs are the publication path. For local development against an already-running Fanout, install telemetrygen, provide short-lived credentials through environment variables, and omit `--host`: ```sh just telemetrygen export FANOUT_BENCH_INGEST_TOKEN='short-lived-benchmark-token' +export FANOUT_BENCH_METRICS_TOKEN='matching-Fanout-metrics-token' ./bin/fanout-bench doctor --telemetrygen ./bin/telemetrygen ./bin/fanout-bench run \ --preset quick \ --telemetrygen ./bin/telemetrygen \ - --endpoint localhost:4317 \ - --metrics-url http://localhost:7520/-/metrics \ + --endpoint localhost:7520 \ + --metrics-url http://localhost:7520/metrics \ --insecure \ --same-host ``` +Managed runs set `FANOUT_ADDR` to one loopback random port for OTLP/gRPC, OTLP/HTTP, the API, and `/metrics`. Metrics stay private: a per-run `FANOUT_METRICS_TOKEN` is sent as `Authorization: Bearer …` and never written to artifacts. MCP, alerts, and chat stay disabled. + Private local metrics endpoints use `FANOUT_BENCH_METRICS_TOKEN`. Local authenticated read load uses `FANOUT_BENCH_QUERY_COOKIE`. Secrets have no value flags and are never persisted. telemetrygen does expose its OTLP header in its process arguments while running, so use only a dedicated short-lived token and revoke it afterward. Managed remote runs create and destroy this token automatically. diff --git a/cmd/fanout-bench/main.go b/cmd/fanout-bench/main.go index 5faa73c..75f95a7 100644 --- a/cmd/fanout-bench/main.go +++ b/cmd/fanout-bench/main.go @@ -63,13 +63,15 @@ func commonFlags(name string, args []string) (bench.Config, error) { fs := flag.NewFlagSet(name, flag.ContinueOnError) cfg := bench.Config{} var tokenEnv, metricsTokenEnv, queryCookieEnv string - fs.StringVar(&cfg.Host, "host", "", "SSH target to provision (for example root@bench-host); empty runs locally") + fs.StringVar(&cfg.Host, "host", "", "SSH target to provision (for example root@example.com); empty runs locally") fs.StringVar(&cfg.SSHIdentity, "ssh-identity", "", "SSH private key path") + fs.StringVar(&cfg.SSHKnownHosts, "ssh-known-hosts", "", "run-scoped SSH known-hosts file") + fs.StringVar(&cfg.HostDescription, "host-description", "", "host description recorded alongside hardware evidence") fs.IntVar(&cfg.SSHPort, "ssh-port", 22, "SSH port") fs.BoolVar(&cfg.KeepRemote, "keep-remote", false, "preserve the isolated remote run directory after downloading artifacts") fs.StringVar(&cfg.Preset, "preset", "standard", "suite: quick, standard, or publish") - fs.StringVar(&cfg.Endpoint, "endpoint", "localhost:4317", "Fanout OTLP/gRPC endpoint") - fs.StringVar(&cfg.MetricsURL, "metrics-url", "http://localhost:7520/-/metrics", "Fanout Prometheus metrics URL (required evidence source)") + fs.StringVar(&cfg.Endpoint, "endpoint", "localhost:7520", "Fanout OTLP/gRPC endpoint") + fs.StringVar(&cfg.MetricsURL, "metrics-url", "http://localhost:7520/metrics", "Fanout Prometheus metrics URL (required evidence source)") fs.BoolVar(&cfg.Insecure, "insecure", false, "use plaintext OTLP/gRPC") fs.StringVar(&cfg.Telemetrygen, "telemetrygen", "./bin/telemetrygen", "path to pinned telemetrygen binary") fs.StringVar(&cfg.TelemetrygenVersion, "telemetrygen-version", "v0.157.0", "version recorded for the generator binary") @@ -87,7 +89,7 @@ func commonFlags(name string, args []string) (bench.Config, error) { fs.BoolVar(&cfg.SameHost, "same-host", false, "declare that Fanout and generators share a host") fs.IntVar(&cfg.BatchSize, "batch-size", 1000, "telemetrygen OTLP batch size") fs.StringVar(&cfg.QueryURL, "query-url", "", "Fanout HTTP base URL for authenticated read-under-write load") - fs.Float64Var(&cfg.QueryRate, "query-rate", 0, "aggregate dashboard queries/s during mixed phases") + fs.Float64Var(&cfg.QueryRate, "query-rate", 0, "aggregate UI operations/s during mixed phases") fs.StringVar(&queryCookieEnv, "query-cookie-env", "FANOUT_BENCH_QUERY_COOKIE", "environment variable containing the authenticated session Cookie header") fs.IntVar(&cfg.ProcessOverride, "processes", 0, "override sweep with one process count per signal") fs.IntVar(&cfg.WorkerOverride, "workers", 0, "override workers per telemetrygen process") @@ -149,12 +151,13 @@ func managed(ctx context.Context, args []string) error { fs.StringVar(&cfg.Preset, "preset", "standard", "suite preset") fs.IntVar(&cfg.Repeat, "repeat", 1, "sweep repetitions") fs.IntVar(&cfg.BatchSize, "batch-size", 1000, "OTLP batch size") - fs.Float64Var(&cfg.QueryRate, "query-rate", 0, "read-under-write queries/s") + fs.Float64Var(&cfg.QueryRate, "query-rate", 0, "read-under-write UI operations/s") fs.DurationVar(&cfg.SampleInterval, "sample-interval", time.Second, "resource sample interval") fs.DurationVar(&cfg.Cooldown, "cooldown", 15*time.Second, "post-load cooldown") fs.StringVar(&cfg.HarnessVersion, "harness-version", version, "harness version") fs.StringVar(&cfg.TelemetrygenVersion, "telemetrygen-version", "v0.157.0", "telemetrygen version") - fs.StringVar(&cfg.ProvisionedHost, "provisioned-host", "", "SSH target recorded in remote evidence") + fs.StringVar(&cfg.ProvisionedHost, "provisioned-host", "", "SSH target used to construct the redacted reproduction command") + fs.StringVar(&cfg.HostDescription, "host-description", "", "host description recorded alongside hardware evidence") fs.IntVar(&cfg.ProvisionedSSHPort, "provisioned-ssh-port", 22, "SSH port recorded in remote evidence") fs.StringVar(&cfg.ProvisionedSSHIdentity, "provisioned-ssh-identity", "", "SSH identity path recorded in remote evidence") fs.IntVar(&cfg.ProcessOverride, "processes", 0, "override process count per signal") diff --git a/cmd/fanout-bench/main_test.go b/cmd/fanout-bench/main_test.go new file mode 100644 index 0000000..3ff122d --- /dev/null +++ b/cmd/fanout-bench/main_test.go @@ -0,0 +1,23 @@ +package main + +import "testing" + +func TestLocalDefaultsUseSharedFanoutListener(t *testing.T) { + cfg, err := commonFlags("test", nil) + if err != nil { + t.Fatal(err) + } + if cfg.Endpoint != "localhost:7520" || cfg.MetricsURL != "http://localhost:7520/metrics" { + t.Fatalf("endpoint=%q metrics=%q", cfg.Endpoint, cfg.MetricsURL) + } +} + +func TestHostFlags(t *testing.T) { + cfg, err := commonFlags("test", []string{"--ssh-known-hosts", "/tmp/run scope/known_hosts", "--host-description", "Hetzner Cloud ccx33, fsn1, ubuntu-24.04"}) + if err != nil { + t.Fatal(err) + } + if cfg.SSHKnownHosts != "/tmp/run scope/known_hosts" || cfg.HostDescription != "Hetzner Cloud ccx33, fsn1, ubuntu-24.04" { + t.Fatalf("known-hosts=%q description=%q", cfg.SSHKnownHosts, cfg.HostDescription) + } +} diff --git a/docs/metrics.md b/docs/metrics.md index 446cf24..6216462 100644 --- a/docs/metrics.md +++ b/docs/metrics.md @@ -1,5 +1,7 @@ # Metric definitions +Metrics are scraped from `/metrics` using `Authorization: Bearer …` with `FANOUT_METRICS_TOKEN` (local harness input: `FANOUT_BENCH_METRICS_TOKEN`). Managed runs generate a private token in memory. + ## Throughput | Field | Definition | @@ -28,6 +30,16 @@ telemetrygen exports a parent and child span per generated trace, but its limite CPU is the Fanout process only. On a same-host run it does not include telemetrygen CPU, but telemetrygen still competes with Fanout for machine resources. RSS and heap are sampled during the phase; an end-only measurement would miss transient peaks. +The sampled series above, plus `fanout_rows_dropped_total`, `process_start_time_seconds`, `fanout_parquet_publish_waiters`, and `fanout_parquet_publish_timeouts_total`, come from Fanout v2026.10.1-alpha and its registered Go/process collectors. + +## Read workload + +Authenticated reads rotate equally across user operations `dashboard-refresh`, `drill-traces` (System overview service-map selection), and `trace-detail`, at 15m, 1h, 6h, 12h, and 24h. A refresh follows `page.tsx`, `use-panel-results.ts`, and `refresh.ts`: one sorted batch of all initially visible panel IDs with the full dashboard spec, range and refresh setting, service All, comparison per spec, and layout widths rounded to 100 pixels for a fixed 1664-pixel viewport. It then sends `POST /api/annotations` with the union of the returned time-panel bounds, when annotations are enabled and those bounds are present. + +Drill-down follows `drill-state.ts` and `drill.tsx`: select the first service-map node, then `POST /api/panels/exemplars` with the original dashboard, panel ID, kind, absolute panel window (`refresh: off`), full-window selection, service dimension, and variables. Trace detail is a separate UI operation using `GET /api/traces/:id` with the discovered namespace, original absolute window, and limit 200. References and selections are retained separately per range. Without a discovered trace, detail slots perform drill-down; without a cached selection, the slot performs dashboard refresh instead. A later slot can select the returned node, so each tick remains one user operation. Fallbacks are reported under the operation actually performed. The current System overview defines no log drill or standalone log/trace search, so synthetic search panels are omitted. + +One scheduled read is one operation. Its latency runs until every request completes; any failed request, panel error, missing panel result, or empty discovered trace detail fails the operation. Responses remain bounded at 32 MiB; responses exceeding this cap fail because their contents cannot be validated. The existing methodology string records the fetched dashboard spec SHA-256 and per-phase operation kinds, dispatched/completed counts, failures, cancellations, p95, and p99. Aggregate latency fields retain their schema. + ## Validity A report passes its evidence-quality gates when: @@ -37,9 +49,9 @@ A report passes its evidence-quality gates when: 3. the Fanout process start time does not change; 4. Fanout's accepted-row counter advances in every phase. -When read load is enabled, the sustained mixed phase must also complete at least 95% of the requested query rate with zero request failures and zero scheduler shedding. The report records requests canceled exactly at the phase boundary separately; they are not hidden as successful work. +When read load is enabled, the sustained mixed phase must also complete at least 95% of the requested operation rate with zero operation failures and zero scheduler shedding. The report records operations canceled exactly at the phase boundary separately; they are not hidden as successful work. -The read scheduler is open-loop and bounded. `query_attempts` counts scheduled requests, `query_latency_ms.count` counts valid completed responses, `query_shed` counts ticks rejected because the concurrency bound was full, and `query_canceled_at_phase_end` counts in-flight requests canceled when load stopped. +The read scheduler is open-loop and bounded. `query_attempts` counts scheduled operations, `query_latency_ms.count` counts valid completed operations, `query_shed` counts ticks rejected because the concurrency bound was full, and `query_canceled_at_phase_end` counts in-flight operations canceled when load stopped. Saturation is not an evidence failure. A valid phase can fail to sustain its offered target, which is the point of a capacity sweep. diff --git a/docs/publishing.md b/docs/publishing.md index b6c3714..33530c5 100644 --- a/docs/publishing.md +++ b/docs/publishing.md @@ -7,13 +7,13 @@ A performance number is useful only when another engineer can explain and reprod - [ ] exact Fanout release, commit when applicable, and verified binary/release artifact checksum; - [ ] exact fanout-bench commit and clean/dirty state; - [ ] exact telemetrygen version; -- [ ] CPU model, architecture, logical CPU count, memory, OS, and generator placement; +- [ ] Hetzner host description, CPU model, architecture, logical CPU count, memory, OS, and generator placement; - [ ] transport, authentication path, batch size, topology, rate, duration, and repetitions; - [ ] raw `report.json`, `report.md`, and `samples.jsonl` from the same run; -- [ ] at least three publish-preset repetitions; +- [ ] three publish-preset repetitions on a fresh default `ccx33` VM; - [ ] signal-specific peak results and a sustained mixed result; - [ ] a separate read-under-write result when claiming interactive performance; -- [ ] requested, completed, failed, and shed query rates beside read latency; +- [ ] requested, completed, failed, and shed operation rates beside read latency; - [ ] all evidence-quality gates passing; - [ ] limitations stated next to the headline, not hidden in a footnote. @@ -32,10 +32,30 @@ Use median values across repetitions for the headline and retain every raw repet Prefer: -> On an Apple M3 Max with generators sharing the host, Fanout accepted X mixed OpenTelemetry items/s for five minutes. The composition was A spans, B log records, and C metric points; dashboard read p95 was Y ms. This is an observed same-host result, not an isolated backend ceiling. +> On a Hetzner Cloud ccx33 (8 dedicated vCPUs, 32 GB RAM) with generators sharing the host, Fanout accepted X mixed OpenTelemetry items/s for five minutes. The composition was A spans, B log records, and C metric points; dashboard read p95 was Y ms. This is an observed same-host result, not an isolated backend ceiling. Avoid: > Fanout does X events/s. The second sentence discards the machine, duration, signal mix, generator placement, read behavior, and definition of “event.” + +## Publication runs + +Every publication run uses one invocation of `scripts/hetzner.sh` with the default Hetzner Cloud `ccx33` (8 dedicated vCPUs, 32 GB RAM), `fsn1`, and `ubuntu-24.04`. Use the publish preset with three repetitions in each group. Separate the ingest-only and read-under-write groups with a standalone `--` so both share one fresh VM while each gets a fresh remote directory and Fanout data directory: + +```sh +export FANOUT_BENCH_HCLOUD_SSH_KEY='benchmark-key' +scripts/hetzner.sh \ + --fanout-version v2026.10.1-alpha \ + --preset publish --repeat 3 \ + --output results/publish-ingest-2026-10-14 \ + -- \ + --fanout-version v2026.10.1-alpha \ + --preset publish --repeat 3 --query-rate 20 \ + --output results/publish-read-write-2026-10-14 +``` + +The script uses the active `hcloud` context or `HCLOUD_TOKEN`, records the VM type, location, and image alongside the detected hardware, rebuilds the harness before provisioning, and waits for SSH once. Groups must be nonempty and use matching SSH port and identity settings; they share a temporary known-hosts file and host description. Every group runs even if an earlier one fails, downloads its own evidence, and receives a one-line summary at the end; any failed group makes the invocation exit nonzero. The script attempts VM deletion after the last group and on early exits, including Ctrl-C. Failed deletion after three attempts prints the unique server name for manual cleanup and exits nonzero. Hetzner bills every started hour, so both groups share the same VM billing time. + +The harness verifies `fanout_v2026.10.1-alpha_linux_amd64.tar.gz` against `SHA256SUMS` on the ccx33. Managed scrapes use private `/metrics` bearer authentication. Read load covers System overview dashboard refresh (batched panels followed by annotations), service-map exemplar drill-down, and trace detail over 15-minute through 24-hour windows. Publish the per-operation methodology, counts and latency, and dashboard spec hash alongside the requested and completed operation rates. diff --git a/internal/bench/managed.go b/internal/bench/managed.go index 71c1f08..08aaf3a 100644 --- a/internal/bench/managed.go +++ b/internal/bench/managed.go @@ -25,6 +25,7 @@ import ( ) type ManagedConfig struct { + HostDescription string FanoutPath string TelemetrygenPath string OutputDir string @@ -65,32 +66,19 @@ func RunManaged(ctx context.Context, managed ManagedConfig, out io.Writer) error if err != nil { return err } - grpcPort, err := freePort() - if err != nil { - return err - } - httpOTLPPort, err := freePort() + authSecret, err := randomHex(32) if err != nil { return err } - authSecret, err := randomHex(32) + metricsToken, err := randomHex(32) if err != nil { return err } httpAddr := fmt.Sprintf("127.0.0.1:%d", httpPort) - grpcAddr := fmt.Sprintf("127.0.0.1:%d", grpcPort) - metricsURL := "http://" + httpAddr + "/-/metrics" + grpcAddr := httpAddr + metricsURL := "http://" + httpAddr + "/metrics" baseURL := "http://" + httpAddr - env := append(os.Environ(), - "FANOUT_DATA_DIR="+dataDir, - "FANOUT_AUTH_CODE_SECRET="+authSecret, - "FANOUT_METRICS_PUBLIC=true", - "FANOUT_MCP_ENABLED=false", - "FANOUT_ALERTS_ENABLED=false", - "FANOUT_HTTP_ADDR="+httpAddr, - "FANOUT_OTLP_GRPC_ADDR="+grpcAddr, - fmt.Sprintf("FANOUT_OTLP_HTTP_ADDR=127.0.0.1:%d", httpOTLPPort), - ) + env := fanoutEnvironment(os.Environ(), dataDir, httpAddr, authSecret, metricsToken) actualVersion := strings.TrimSpace(commandOutput(managed.FanoutPath, "version")) if actualVersion == "" { @@ -140,7 +128,7 @@ func RunManaged(ctx context.Context, managed ManagedConfig, out io.Writer) error line = strings.ReplaceAll(line, match[1], "[REDACTED]") } logMu.Lock() - fmt.Fprintln(logFile, line) + fmt.Fprintln(logFile, redact(line, authSecret, metricsToken)) logMu.Unlock() } } @@ -183,7 +171,7 @@ func RunManaged(ctx context.Context, managed ManagedConfig, out io.Writer) error return err } - cfg := Config{Host: managed.ProvisionedHost, SSHPort: managed.ProvisionedSSHPort, SSHIdentity: managed.ProvisionedSSHIdentity, Preset: managed.Preset, Endpoint: grpcAddr, MetricsURL: metricsURL, Insecure: true, Telemetrygen: managed.TelemetrygenPath, Token: ingestToken, OutputDir: managed.OutputDir, FanoutVersion: actualVersion, FanoutArtifact: managed.FanoutArtifact, DataDir: dataDir, SampleInterval: managed.SampleInterval, GeneratorTimeout: 30 * time.Second, Cooldown: managed.Cooldown, Repeat: managed.Repeat, QueryURL: baseURL, QueryCookie: queryCookie, QueryRate: managed.QueryRate, SameHost: true, BatchSize: managed.BatchSize, TelemetrygenVersion: managed.TelemetrygenVersion, ProcessOverride: managed.ProcessOverride, WorkerOverride: managed.WorkerOverride, RateOverride: managed.RateOverride, DurationOverride: managed.DurationOverride} + cfg := Config{Host: managed.ProvisionedHost, HostDescription: managed.HostDescription, SSHPort: managed.ProvisionedSSHPort, SSHIdentity: managed.ProvisionedSSHIdentity, Preset: managed.Preset, Endpoint: grpcAddr, MetricsURL: metricsURL, MetricsToken: metricsToken, Insecure: true, Telemetrygen: managed.TelemetrygenPath, Token: ingestToken, OutputDir: managed.OutputDir, FanoutVersion: actualVersion, FanoutArtifact: managed.FanoutArtifact, DataDir: dataDir, SampleInterval: managed.SampleInterval, GeneratorTimeout: 30 * time.Second, Cooldown: managed.Cooldown, Repeat: managed.Repeat, QueryURL: baseURL, QueryCookie: queryCookie, QueryRate: managed.QueryRate, SameHost: true, BatchSize: managed.BatchSize, TelemetrygenVersion: managed.TelemetrygenVersion, ProcessOverride: managed.ProcessOverride, WorkerOverride: managed.WorkerOverride, RateOverride: managed.RateOverride, DurationOverride: managed.DurationOverride} runner := Runner{Config: cfg, Version: managed.HarnessVersion, Out: out, DeferSummary: true} report, runErr := runner.Run(ctx) stopFanout() @@ -203,7 +191,7 @@ func RunManaged(ctx context.Context, managed ManagedConfig, out io.Writer) error verifyOutput, verifyErr := verify.CombinedOutput() verifyDetail := "authoritative Parquet storage verified" if verifyErr != nil { - verifyDetail = redact(strings.TrimSpace(string(verifyOutput)), ingestToken, setupToken) + verifyDetail = redact(strings.TrimSpace(string(verifyOutput)), ingestToken, setupToken, queryCookie, authSecret, metricsToken) if verifyDetail == "" { verifyDetail = verifyErr.Error() } @@ -216,7 +204,7 @@ func RunManaged(ctx context.Context, managed ManagedConfig, out io.Writer) error if err := os.MkdirAll(managed.OutputDir, 0o755); err != nil { return err } - repairEvidence := strings.TrimSpace(redact(string(verifyOutput), ingestToken, setupToken)) + repairEvidence := strings.TrimSpace(redact(string(verifyOutput), ingestToken, setupToken, queryCookie, authSecret, metricsToken)) if repairEvidence == "" { repairEvidence = verifyDetail } @@ -241,7 +229,7 @@ func RunManaged(ctx context.Context, managed ManagedConfig, out io.Writer) error func completeSetup(ctx context.Context, baseURL, setupToken string) (string, string, error) { jar, _ := cookiejar.New(nil) client := &http.Client{Timeout: 10 * time.Second, Jar: jar} - payload, _ := json.Marshal(map[string]string{"email": "benchmark@localhost", "name": "Fanout Benchmark", "setup_token": setupToken}) + payload, _ := json.Marshal(map[string]string{"email": "benchmark@localhost", "display_name": "Fanout Benchmark", "setup_token": setupToken}) deadline := time.Now().Add(20 * time.Second) for { req, err := http.NewRequestWithContext(ctx, http.MethodPost, baseURL+"/api/auth/setup", bytes.NewReader(payload)) @@ -265,7 +253,7 @@ func completeSetup(ctx context.Context, baseURL, setupToken string) (string, str return "", "", err } if decoded.IngestToken == "" { - return "", "", errors.New("Fanout setup returned no ingest token") + return "", "", errors.New("Fanout setup response missing ingest_token") } parsed, _ := url.Parse(baseURL) cookies := jar.Cookies(parsed) @@ -273,10 +261,13 @@ func completeSetup(ctx context.Context, baseURL, setupToken string) (string, str for _, cookie := range cookies { pairs = append(pairs, cookie.Name+"="+cookie.Value) } + if len(pairs) == 0 { + return "", "", errors.New("Fanout setup returned no session cookie") + } return decoded.IngestToken, strings.Join(pairs, "; "), nil } if resp.StatusCode >= 400 && resp.StatusCode < 500 { - return "", "", fmt.Errorf("Fanout setup status %d: %s", resp.StatusCode, body) + return "", "", fmt.Errorf("Fanout setup status %d", resp.StatusCode) } } if time.Now().After(deadline) { @@ -332,3 +323,24 @@ func copyEvidenceFile(source, destination string) error { } return closeErr } + +// Fanout rejects unknown FANOUT_ names. Isolate the managed product from the +// caller's Fanout and benchmark settings, including any AI provider selection. +// Reuse this exact environment for offline repair verification. +func fanoutEnvironment(parent []string, dataDir, addr, authSecret, metricsToken string) []string { + env := make([]string, 0, len(parent)+7) + for _, entry := range parent { + if !strings.HasPrefix(entry, "FANOUT_") { + env = append(env, entry) + } + } + return append(env, + "FANOUT_ADDR="+addr, + "FANOUT_DATA_DIR="+dataDir, + "FANOUT_AUTH_CODE_SECRET="+authSecret, + "FANOUT_METRICS_TOKEN="+metricsToken, + "FANOUT_METRICS_PUBLIC=false", + "FANOUT_MCP_ENABLED=false", + "FANOUT_ALERTS_ENABLED=false", + ) +} diff --git a/internal/bench/managed_test.go b/internal/bench/managed_test.go new file mode 100644 index 0000000..283ea6a --- /dev/null +++ b/internal/bench/managed_test.go @@ -0,0 +1,120 @@ +package bench + +import ( + "context" + "encoding/json" + "fmt" + "net/http" + "net/http/httptest" + "os" + "strings" + "testing" +) + +func TestFanoutEnvironmentUsesCurrentSchema(t *testing.T) { + // Schema env tags captured from internal/config/config.go at 870c90db. + schema, err := os.ReadFile("testdata/fanout-v2026.10.1-alpha-env.txt") + if err != nil { + t.Fatal(err) + } + knownNames := map[string]bool{} + for _, name := range strings.Fields(string(schema)) { + knownNames[name] = true + } + want := map[string]string{ + "FANOUT_ADDR": "127.0.0.1:12345", + "FANOUT_DATA_DIR": "/tmp/data", + "FANOUT_AUTH_CODE_SECRET": "auth-secret", + "FANOUT_METRICS_TOKEN": "metrics-secret", + "FANOUT_METRICS_PUBLIC": "false", + "FANOUT_MCP_ENABLED": "false", + "FANOUT_ALERTS_ENABLED": "false", + } + parent := []string{"PATH=/bin", "FANOUT_HTTP_ADDR=:7520", "FANOUT_AI_PROVIDER=openai", "FANOUT_BENCH_QUERY_COOKIE=secret", "FANOUT_ADDR=:1", "FANOUT_DATA_DIR=old", "FANOUT_AUTH_CODE_SECRET_FILE=/secret", "FANOUT_METRICS_TOKEN_FILE=/secret"} + env := fanoutEnvironment(parent, "/tmp/data", "127.0.0.1:12345", "auth-secret", "metrics-secret") + seen := map[string]bool{} + for _, entry := range env { + name, value, _ := strings.Cut(entry, "=") + if seen[name] { + t.Fatalf("duplicate setting %s", name) + } + seen[name] = true + if !strings.HasPrefix(name, "FANOUT_") { + continue + } + if !knownNames[name] { + t.Errorf("unknown Fanout schema setting %s", name) + } + expected, known := want[name] + if !known { + t.Errorf("unknown or uncontrolled Fanout setting %s", name) + } else if value != expected { + t.Errorf("%s=%q, want %q", name, value, expected) + } + } + for name := range want { + if !seen[name] { + t.Errorf("missing %s", name) + } + } + if !seen["PATH"] { + t.Error("lost non-Fanout environment") + } +} + +func TestSetupRequiresIngestToken(t *testing.T) { + for _, response := range []string{`{"status":"authenticated"}`, `{"ingest_token":""}`} { + t.Run(response, func(t *testing.T) { + requests := 0 + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + requests++ + if r.URL.Path != "/api/auth/setup" || r.Method != http.MethodPost || r.Header.Get("Fanout-Request") != "1" { + t.Errorf("unexpected request: %s %s", r.Method, r.URL.Path) + } + var body map[string]string + if err := json.NewDecoder(r.Body).Decode(&body); err != nil { + t.Error(err) + } + if body["display_name"] != "Fanout Benchmark" || body["email"] != "benchmark@localhost" || body["setup_token"] != "setup-secret" { + t.Errorf("wrong setup shape: %v", body) + } + http.SetCookie(w, &http.Cookie{Name: "fanout_session", Value: "session-secret", Path: "/", HttpOnly: true}) + fmt.Fprint(w, response) + })) + defer server.Close() + token, cookie, err := completeSetup(context.Background(), server.URL, "setup-secret") + if err == nil || !strings.Contains(err.Error(), "missing ingest_token") { + t.Fatalf("error=%v, want missing ingest_token", err) + } + if token != "" || cookie != "" || requests != 1 { + t.Fatalf("token=%q cookie=%q requests=%d", token, cookie, requests) + } + }) + } +} + +func TestMetricsPathAndBearerAuthentication(t *testing.T) { + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path != "/metrics" { + t.Errorf("path=%s", r.URL.Path) + } + if r.Header.Get("Authorization") != "Bearer metrics-secret" { + http.Error(w, "private", http.StatusUnauthorized) + return + } + fmt.Fprintln(w, `fanout_ingest_rows_total{signal="spans"} 42`) + })) + defer server.Close() + runner := Runner{Config: Config{MetricsURL: server.URL + "/metrics", MetricsToken: "metrics-secret"}, client: server.Client()} + snap, err := runner.scrape(context.Background()) + if err != nil { + t.Fatal(err) + } + if acceptedCounts(snap).Spans != 42 { + t.Fatal("accepted counter not scraped") + } + runner.Config.MetricsToken = "" + if _, err := runner.scrape(context.Background()); err == nil { + t.Fatal("unauthenticated scrape succeeded") + } +} diff --git a/internal/bench/query.go b/internal/bench/query.go index b3f02a3..efb375a 100644 --- a/internal/bench/query.go +++ b/internal/bench/query.go @@ -1,8 +1,10 @@ package bench import ( + "bytes" "context" "encoding/json" + "fmt" "io" "math" "net/http" @@ -13,33 +15,35 @@ import ( "time" ) -var queryTargets = []string{ - "/api/observability/overview?window=15m&limit=100", - "/api/observability/topology?window=1h&limit=100", - "/api/observability/performance?window=6h&limit=100", - "/api/observability/trace?window=12h&limit=100", - "/api/observability/logs?window=24h&limit=100", -} - -// maxQueryBody bounds a single dashboard response. A response larger than this -// is still a success: the server answered, and only the JSON well-formedness -// check is skipped because the tail was never read. +// maxQueryBody bounds a single response. Oversized responses fail because +// their panel errors and discovered trace contents cannot be validated. const maxQueryBody = 32 << 20 type queryRun struct { - wg sync.WaitGroup - hist *histogram - failures int64 - attempts int64 - shed int64 - canceled int64 - stop chan struct{} - stopOnce sync.Once - cancel context.CancelFunc + targets []queryTarget + traceMu sync.Mutex + traces map[string]traceReference + selections map[string]drillSelection + kindsMu sync.Mutex + kinds map[string]*operationStats + wg sync.WaitGroup + hist *histogram + failures int64 + attempts int64 + shed int64 + canceled int64 + stop chan struct{} + stopOnce sync.Once + cancel context.CancelFunc +} + +type operationStats struct { + dispatched, failures, canceled int64 + hist *histogram } func newQueryRun() *queryRun { - return &queryRun{hist: newHistogram(), stop: make(chan struct{}), cancel: func() {}} + return &queryRun{targets: []queryTarget{{name: "dashboard-refresh", body: []byte(`{}`)}}, traces: map[string]traceReference{}, selections: map[string]drillSelection{}, kinds: map[string]*operationStats{}, hist: newHistogram(), stop: make(chan struct{}), cancel: func() {}} } func (q *queryRun) start(ctx context.Context, client *http.Client, base, cookie string, rate float64) { @@ -106,39 +110,122 @@ func (q *queryRun) finish() { } func (q *queryRun) request(ctx context.Context, client *http.Client, base, cookie string, targetIndex int) { - target := strings.TrimRight(base, "/") + queryTargets[targetIndex%len(queryTargets)] + query := q.targets[targetIndex%len(q.targets)] + kind := query.name started := time.Now() - req, err := http.NewRequestWithContext(ctx, http.MethodGet, target, nil) - if err != nil { - atomic.AddInt64(&q.failures, 1) - return - } - req.Header.Set("Cookie", cookie) - resp, err := client.Do(req) - if err != nil { - if ctx.Err() != nil { + success := false + defer func() { + elapsed := time.Since(started) + q.kindsMu.Lock() + defer q.kindsMu.Unlock() + stats := q.kinds[kind] + if stats == nil { + stats = &operationStats{hist: newHistogram()} + q.kinds[kind] = stats + } + stats.dispatched++ + if success { + q.hist.record(elapsed) + stats.hist.record(elapsed) + } else if ctx.Err() != nil { atomic.AddInt64(&q.canceled, 1) + stats.canceled++ } else { atomic.AddInt64(&q.failures, 1) + stats.failures++ + } + }() + refresh := func() bool { + body, ok := queryHTTP(ctx, client, base, cookie, http.MethodPost, "/api/panels/query", query.body) + if !ok { + return false + } + annotations, valid := q.panelResponse(body, query) + // refresh.ts requests annotations after the panel batch, even if a + // panel reports an error. Count the whole operation as failed either way. + if annotations != nil { + _, ok = queryHTTP(ctx, client, base, cookie, http.MethodPost, "/api/annotations", annotations) } + return valid && ok + } + if kind == "dashboard-refresh" { + success = refresh() return } - body, readErr := io.ReadAll(io.LimitReader(resp.Body, maxQueryBody+1)) - resp.Body.Close() - if readErr != nil { - if ctx.Err() != nil { - atomic.AddInt64(&q.canceled, 1) - } else { - atomic.AddInt64(&q.failures, 1) - } + q.traceMu.Lock() + trace := q.traces[query.window] + q.traceMu.Unlock() + if kind == "trace-detail" && trace.id != "" { + body, ok := queryHTTP(ctx, client, base, cookie, http.MethodGet, traceDetailPath(trace), nil) + success = ok && traceDetailResponse(body) return } - truncated := len(body) > maxQueryBody - if resp.StatusCode != http.StatusOK || (!truncated && !json.Valid(body)) { - atomic.AddInt64(&q.failures, 1) + // Without a trace reference, this slot performs the UI exemplar selection + // operation. Report its actual kind instead of a successful detail read. + kind = "drill-" + query.kind + q.traceMu.Lock() + selection, found := q.selections[query.window+"/"+query.panelID] + q.traceMu.Unlock() + if !found { + // A click needs a previously loaded panel. Use this slot for a refresh; + // the next slot can select its result, keeping one user operation per tick. + kind = "dashboard-refresh" + success = refresh() return } - q.hist.record(time.Since(started)) + body, ok := queryHTTP(ctx, client, base, cookie, http.MethodPost, "/api/panels/exemplars", exemplarBody(query, selection)) + success = ok && q.exemplarResponse(body, query, selection) +} + +func queryHTTP(ctx context.Context, client *http.Client, base, cookie, method, path string, body []byte) ([]byte, bool) { + req, err := http.NewRequestWithContext(ctx, method, strings.TrimRight(base, "/")+path, bytes.NewReader(body)) + if err != nil { + return nil, false + } + req.Header.Set("Cookie", cookie) + req.Header.Set("Fanout-Request", "1") + if method == http.MethodPost { + req.Header.Set("Content-Type", "application/json") + } + resp, err := client.Do(req) + if err != nil { + return nil, false + } + defer resp.Body.Close() + response, err := io.ReadAll(io.LimitReader(resp.Body, maxQueryBody+1)) + return response, err == nil && resp.StatusCode == http.StatusOK && len(response) <= maxQueryBody && json.Valid(response) +} + +func (q *queryRun) completedByKind() map[string]int64 { + q.kindsMu.Lock() + defer q.kindsMu.Unlock() + completed := make(map[string]int64, len(q.kinds)) + for kind, stats := range q.kinds { + completed[kind] = stats.hist.snapshot().Count + } + return completed +} + +// Keep per-operation latency evidence inside the methodology string. +func (q *queryRun) operationSummary(phase string) string { + q.kindsMu.Lock() + defer q.kindsMu.Unlock() + if len(q.kinds) == 0 { + return "" + } + var names []string + for name := range q.kinds { + names = append(names, name) + } + sort.Strings(names) + var b strings.Builder + fmt.Fprintf(&b, "\n\nRead operations for %s (fallback slots are attributed to the operation performed):\n\n| Kind | Dispatched | Completed | Failures | Canceled | p95 | p99 |\n|---|---:|---:|---:|---:|---:|---:|\n", phase) + for _, name := range names { + stats := q.kinds[name] + latency := stats.hist.snapshot() + fmt.Fprintf(&b, "| %s | %d | %d | %d | %d | %.1fms | %.1fms |\n", name, stats.dispatched, latency.Count, stats.failures, stats.canceled, latency.P95Ms, latency.P99Ms) + } + return b.String() } type histogram struct { diff --git a/internal/bench/query_test.go b/internal/bench/query_test.go index 80b3580..e6f6670 100644 --- a/internal/bench/query_test.go +++ b/internal/bench/query_test.go @@ -13,7 +13,7 @@ func TestQueryRunMaintainsOpenLoopScheduleThroughLatency(t *testing.T) { server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { time.Sleep(100 * time.Millisecond) w.Header().Set("Content-Type", "application/json") - fmt.Fprint(w, `{}`) + fmt.Fprint(w, `{"results":[{"status":"empty"}]}`) })) defer server.Close() @@ -41,7 +41,7 @@ func TestQueryRunFinishDrainsInFlightRequestsWithoutFailures(t *testing.T) { server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { time.Sleep(150 * time.Millisecond) w.Header().Set("Content-Type", "application/json") - fmt.Fprint(w, `{"rows":[]}`) + fmt.Fprint(w, `{"results":[{"status":"empty"}]}`) })) defer server.Close() @@ -67,7 +67,7 @@ func TestQueryRunFinishDrainsInFlightRequestsWithoutFailures(t *testing.T) { func TestQueryRunDoesNotShedWhileWorkersAreAvailable(t *testing.T) { server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { w.Header().Set("Content-Type", "application/json") - fmt.Fprint(w, `{}`) + fmt.Fprint(w, `{"results":[{"status":"empty"}]}`) })) defer server.Close() @@ -86,9 +86,8 @@ func TestQueryRunDoesNotShedWhileWorkersAreAvailable(t *testing.T) { } } -// A response larger than the read cap is a success: the server answered, only -// the tail was left unread. -func TestQueryRunTreatsOversizeResponseAsSuccess(t *testing.T) { +// A response above the cap cannot be checked for panel errors and must fail. +func TestQueryRunTreatsOversizeResponseAsFailure(t *testing.T) { server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { w.Header().Set("Content-Type", "application/json") w.Write([]byte(`{"rows":"`)) @@ -107,10 +106,7 @@ func TestQueryRunTreatsOversizeResponseAsSuccess(t *testing.T) { queries := newQueryRun() queries.request(context.Background(), &http.Client{Timeout: 30 * time.Second}, server.URL, "session=test", 0) - if queries.failures != 0 { - t.Fatalf("failures=%d; a truncated read was counted as a server error", queries.failures) - } - if got := queries.hist.snapshot().Count; got != 1 { - t.Fatalf("completed=%d; want 1", got) + if queries.failures != 1 || queries.hist.snapshot().Count != 0 { + t.Fatalf("failures=%d completed=%d; truncated read escaped validation", queries.failures, queries.hist.snapshot().Count) } } diff --git a/internal/bench/query_workload.go b/internal/bench/query_workload.go new file mode 100644 index 0000000..77bbc40 --- /dev/null +++ b/internal/bench/query_workload.go @@ -0,0 +1,317 @@ +package bench + +import ( + "context" + "crypto/sha256" + "encoding/json" + "errors" + "fmt" + "io" + "net/http" + "net/url" + "sort" + "strings" + "time" +) + +var queryWindows = []string{"15m", "1h", "6h", "12h", "24h"} + +type queryTarget struct { + name, window string + body []byte + dashboard json.RawMessage + panels []string + timePanels map[string]bool + drills map[string]string + panelID, kind string + annotations bool +} + +// Read the owner's built-in dashboard once before measurement. Passing its +// complete spec preserves the release's SQL, variables and panel definitions. +func loadQueryTargets(ctx context.Context, client *http.Client, base, cookie string) ([]queryTarget, string, error) { + get := func(path string, value any) error { + req, err := http.NewRequestWithContext(ctx, http.MethodGet, strings.TrimRight(base, "/")+path, nil) + if err != nil { + return err + } + req.Header.Set("Cookie", cookie) + req.Header.Set("Fanout-Request", "1") + resp, err := client.Do(req) + if err != nil { + return err + } + defer resp.Body.Close() + if resp.StatusCode != http.StatusOK { + return fmt.Errorf("dashboard discovery status %d", resp.StatusCode) + } + return json.NewDecoder(io.LimitReader(resp.Body, 512<<10)).Decode(value) + } + var list struct { + Dashboards []struct { + ID string `json:"id"` + Name string `json:"name"` + IsDefault bool `json:"is_default"` + } `json:"dashboards"` + } + if err := get("/api/dashboards", &list); err != nil { + return nil, "", err + } + for _, item := range list.Dashboards { + if !item.IsDefault || item.Name != "System overview" { + continue + } + var record struct { + Spec json.RawMessage `json:"spec"` + } + if err := get("/api/dashboards/"+url.PathEscape(item.ID), &record); err != nil { + return nil, "", err + } + return systemQueryTargets(record.Spec) + } + return nil, "", errors.New("built-in System overview dashboard not found") +} + +// page.tsx initially makes every panel visible. Model that full-dashboard +// load at a fixed 1664px viewport (1600px container), including all widths. +func systemQueryTargets(spec json.RawMessage) ([]queryTarget, string, error) { + var dashboard struct { + Name string `json:"name"` + Time struct { + Refresh string `json:"refresh"` + Compare string `json:"compare"` + } `json:"time"` + Annotations struct { + Deploys *bool `json:"deploys"` + Anomalies *bool `json:"anomalies"` + } `json:"annotations"` + Panels []struct { + ID string `json:"id"` + Viz string `json:"viz"` + Width int `json:"width"` + Grid *struct { + W int `json:"w"` + } `json:"grid"` + } `json:"panels"` + } + if err := json.Unmarshal(spec, &dashboard); err != nil { + return nil, "", err + } + if dashboard.Name != "System overview" || len(dashboard.Panels) == 0 { + return nil, "", errors.New("invalid System overview spec") + } + ids := []string{} + widths := map[string]int{} + timePanels := map[string]bool{} + drills := map[string]string{} + var drillIDs []string + for _, p := range dashboard.Panels { + ids = append(ids, p.ID) + width := p.Width + if width == 0 { + width = 6 + } + if p.Grid != nil { + width = p.Grid.W + } + // use-panel-results.ts rounds to the nearest 100, with a 100px minimum. + widths[p.ID] = max(100, int(float64(1600*width)/1200+.5)*100) + timePanels[p.ID] = p.Viz == "timeseries" || p.Viz == "heatmap" || p.Viz == "state_timeline" + if p.Viz == "service_map" { + drills[p.ID] = "traces" + drillIDs = append(drillIDs, p.ID) + } + } + sort.Strings(ids) + refresh := dashboard.Time.Refresh + if refresh == "" { + refresh = "30s" + } + annotations := !(dashboard.Annotations.Deploys != nil && !*dashboard.Annotations.Deploys && dashboard.Annotations.Anomalies != nil && !*dashboard.Annotations.Anomalies) + var targets []queryTarget + kinds := []string{"dashboard-refresh"} + for _, id := range drillIDs { + kinds = append(kinds, "drill-"+drills[id]+" ("+id+")") + } + if len(drillIDs) > 0 { + kinds = append(kinds, "trace-detail") + } + for _, window := range queryWindows { + body, _ := json.Marshal(map[string]any{ + "dashboard": spec, "panels": ids, "time": map[string]string{"range": window, "refresh": refresh}, + "vars": map[string]string{"service": "$__all"}, "widths": widths, "compare": dashboard.Time.Compare == "previous_period", + }) + base := queryTarget{window: window, body: body, dashboard: spec, panels: ids, timePanels: timePanels, drills: drills, annotations: annotations} + base.name = "dashboard-refresh" + targets = append(targets, base) + for _, id := range drillIDs { + drill := base + drill.name, drill.panelID, drill.kind = "drill-"+drills[id], id, drills[id] + targets = append(targets, drill) + } + if len(drillIDs) > 0 { + detail := base + detail.name, detail.panelID, detail.kind = "trace-detail", drillIDs[0], drills[drillIDs[0]] + targets = append(targets, detail) + } + } + digest := sha256.Sum256(spec) + method := fmt.Sprintf(" Reads rotate equally through user operations %s at 15m, 1h, 6h, 12h and 24h. Dashboard refresh models the initial all-visible panel batch (sorted IDs, a 1664px viewport, rounded layout widths, service All, comparison per spec), followed by annotations over returned time-panel bounds. Drill-down selects the first service-map node over its returned full window. Trace detail reuses the discovered trace's namespace and absolute window; without a reference its slot performs drill-down; without a cached selection the slot performs dashboard refresh instead. Each scheduled read measures all requests in its operation and fails on any request or panel error, missing panel, or empty discovered trace detail. No standalone trace search or log search is issued by this UI. System overview spec SHA-256: %x.", strings.Join(kinds, ", "), digest) + return targets, method, nil +} + +type traceReference struct{ id, namespace, from, to string } +type drillSelection struct { + from, to string + dimensions map[string]string +} + +func traceDetailPath(trace traceReference) string { + params := url.Values{"namespace": {trace.namespace}, "from": {trace.from}, "to": {trace.to}, "limit": {"200"}} + return "/api/traces/" + url.PathEscape(trace.id) + "?" + params.Encode() +} + +func exemplarBody(query queryTarget, selection drillSelection) []byte { + body, _ := json.Marshal(map[string]any{ + "dashboard": query.dashboard, "panel_id": query.panelID, "kind": query.kind, + "time": map[string]string{"from": selection.from, "to": selection.to, "refresh": "off"}, + "from": selection.from, "to": selection.to, "dimensions": selection.dimensions, + "vars": map[string]string{"service": "$__all"}, + }) + return body +} + +// JS Date.toISOString() always includes milliseconds. +func uiTime(ms int64) string { return time.UnixMilli(ms).UTC().Format("2006-01-02T15:04:05.000Z") } + +type panelResult struct { + ID string `json:"id"` + Status string `json:"status"` + Error string `json:"error"` + From *int64 `json:"from_ms"` + To *int64 `json:"to_ms"` + Frame *struct { + Columns []struct { + Name string `json:"name"` + } `json:"columns"` + Values [][]json.RawMessage `json:"values"` + } `json:"frame"` +} + +func (q *queryRun) panelResponse(body []byte, query queryTarget) ([]byte, bool) { + var response struct { + Results []panelResult `json:"results"` + } + if json.Unmarshal(body, &response) != nil || len(response.Results) == 0 { + return nil, false + } + seen := map[string]bool{} + valid := true + var from, to int64 + for _, result := range response.Results { + if seen[result.ID] || (result.Status != "ok" && result.Status != "empty") || result.Error != "" { + valid = false + } + seen[result.ID] = true + if result.From == nil || result.To == nil || *result.From >= *result.To { + continue + } + if query.timePanels[result.ID] { + if from == 0 || *result.From < from { + from = *result.From + } + if *result.To > to { + to = *result.To + } + } + if _, drill := query.drills[result.ID]; !drill || result.Frame == nil || result.Status != "ok" || result.Error != "" { + continue + } + selection := drillSelection{from: uiTime(*result.From), to: uiTime(*result.To), dimensions: map[string]string{}} + // service_map's UI selection contains {service} from the selected node. + serviceColumn, kindColumn := -1, -1 + for i, column := range result.Frame.Columns { + if column.Name == "service" { + serviceColumn = i + } + if column.Name == "kind" { + kindColumn = i + } + } + if serviceColumn >= 0 && serviceColumn < len(result.Frame.Values) { + for row, cell := range result.Frame.Values[serviceColumn] { + var service, kind string + _ = json.Unmarshal(cell, &service) + if kindColumn >= 0 && kindColumn < len(result.Frame.Values) && row < len(result.Frame.Values[kindColumn]) { + _ = json.Unmarshal(result.Frame.Values[kindColumn][row], &kind) + } + if service != "" && (kindColumn < 0 || kind == "node") { + selection.dimensions["service"] = service + break + } + } + } + if len(selection.dimensions) > 0 { + q.traceMu.Lock() + q.selections[query.window+"/"+result.ID] = selection + q.traceMu.Unlock() + } + } + var dashboard struct{ Panels []struct{ ID, Viz string } } + _ = json.Unmarshal(query.dashboard, &dashboard) + for _, id := range query.panels { + // Text panels are omitted from API results. + text := false + for _, p := range dashboard.Panels { + if p.ID == id && p.Viz == "text" { + text = true + } + } + if !seen[id] && !text { + valid = false + } + } + if query.annotations && from < to { + body, _ := json.Marshal(map[string]string{"from": uiTime(from), "to": uiTime(to)}) + return body, valid + } + return nil, valid +} + +func (q *queryRun) exemplarResponse(body []byte, query queryTarget, selection drillSelection) bool { + var response struct { + Traces []struct { + ID string `json:"trace_id"` + Namespace string `json:"namespace"` + } `json:"traces"` + Logs json.RawMessage `json:"logs"` + } + if json.Unmarshal(body, &response) != nil { + return false + } + if query.kind == "logs" { + return len(response.Logs) > 0 && string(response.Logs) != "null" + } + if response.Traces == nil { + return false + } + for _, trace := range response.Traces { + if trace.ID == "" { + continue + } + q.traceMu.Lock() + q.traces[query.window] = traceReference{trace.ID, trace.Namespace, selection.from, selection.to} + q.traceMu.Unlock() + break + } + return true +} + +func traceDetailResponse(body []byte) bool { + var response struct { + Data struct { + Spans []json.RawMessage `json:"spans"` + } `json:"data"` + } + return json.Unmarshal(body, &response) == nil && len(response.Data.Spans) > 0 +} diff --git a/internal/bench/query_workload_test.go b/internal/bench/query_workload_test.go new file mode 100644 index 0000000..40ff371 --- /dev/null +++ b/internal/bench/query_workload_test.go @@ -0,0 +1,454 @@ +package bench + +import ( + "context" + "encoding/json" + "fmt" + "net/http" + "net/http/httptest" + "os" + "reflect" + "strings" + "testing" + "time" +) + +func systemSpec(t *testing.T) []byte { + t.Helper() + // DefaultSpec from Fanout 870c90db, including defaults_sql.go and spec.go. + spec, err := os.ReadFile("testdata/system-overview-v2026.10.1-alpha.json") + if err != nil { + t.Fatal(err) + } + return spec +} + +func decodeBody(t *testing.T, body []byte) map[string]any { + t.Helper() + var value map[string]any + if err := json.Unmarshal(body, &value); err != nil { + t.Fatal(err) + } + return value +} + +func TestSystemOperationsMatchUIRequestConstruction(t *testing.T) { + spec := systemSpec(t) + targets, method, err := systemQueryTargets(spec) + if err != nil { + t.Fatal(err) + } + ids := []string{"check_latency", "check_states", "containers", "endpoints", "error_logs", "error_services", "errors", "hosts", "latency", "latency_p95", "requests", "service_health", "service_map", "traffic", "versions"} + if len(targets) != 15 { + t.Fatalf("targets=%d", len(targets)) + } + for w, window := range queryWindows { + refresh, drill, detail := targets[w*3], targets[w*3+1], targets[w*3+2] + if refresh.name != "dashboard-refresh" || drill.name != "drill-traces" || drill.panelID != "service_map" || detail.name != "trace-detail" { + t.Fatalf("incorrect operations: %+v", targets[w*3:w*3+3]) + } + widths := map[string]any{} + panelIDs := []any{} + for _, id := range ids { + widths[id] = float64(800) + panelIDs = append(panelIDs, id) + } + for _, id := range []string{"check_latency", "check_states", "containers", "hosts", "service_health", "service_map", "versions"} { + widths[id] = float64(1600) + } + widths["error_services"], widths["endpoints"] = float64(500), float64(1100) + want := map[string]any{"dashboard": decodeBody(t, spec), "panels": panelIDs, "time": map[string]any{"range": window, "refresh": "30s"}, "vars": map[string]any{"service": "$__all"}, "widths": widths, "compare": false} + if got := decodeBody(t, refresh.body); !reflect.DeepEqual(got, want) { + t.Fatalf("UI panel batch mismatch: got %v want %v", got, want) + } + selection := drillSelection{from: "2026-10-01T12:00:00.000Z", to: "2026-10-01T13:00:00.000Z", dimensions: map[string]string{"service": "checkout"}} + wantDrill := map[string]any{"dashboard": decodeBody(t, spec), "panel_id": "service_map", "kind": "traces", "time": map[string]any{"from": selection.from, "to": selection.to, "refresh": "off"}, "from": selection.from, "to": selection.to, "dimensions": map[string]any{"service": "checkout"}, "vars": map[string]any{"service": "$__all"}} + if got := decodeBody(t, exemplarBody(drill, selection)); !reflect.DeepEqual(got, wantDrill) { + t.Fatalf("UI exemplar mismatch: %v", got) + } + // drill.tsx sends logs only for a logs-source panel with a logs selection. + drill.kind, drill.panelID = "logs", "error_logs" + wantDrill["kind"], wantDrill["panel_id"] = "logs", "error_logs" + if got := decodeBody(t, exemplarBody(drill, selection)); !reflect.DeepEqual(got, wantDrill) { + t.Fatalf("UI logs exemplar mismatch: %v", got) + } + } + for _, name := range []string{"dashboard-refresh", "drill-traces", "trace-detail", "SHA-256"} { + if !strings.Contains(method, name) { + t.Errorf("methodology omits %s", name) + } + } +} + +// Mirrors a panel batch response, including the effective window from discovery +// and a selectable service-map node. No invented search dashboard is needed. +func panelBatch(t *testing.T, query queryTarget, from, to int64) []byte { + t.Helper() + results := []any{} + for _, id := range query.panels { + result := map[string]any{"id": id, "status": "empty", "from_ms": from, "to_ms": to} + if id == "service_map" { + result["status"] = "ok" + result["frame"] = map[string]any{"columns": []any{map[string]string{"name": "kind"}, map[string]string{"name": "service"}}, "values": [][]string{{"node"}, {"checkout"}}} + } + results = append(results, result) + } + body, err := json.Marshal(map[string]any{"results": results}) + if err != nil { + t.Fatal(err) + } + return body +} + +func TestWorkloadDiscoversAuthenticatedBuiltInDashboard(t *testing.T) { + spec := systemSpec(t) + requests := 0 + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + requests++ + if r.Header.Get("Cookie") != "fanout_session=test" || r.Method != http.MethodGet { + t.Error("discovery missing authentication") + } + switch r.URL.Path { + case "/api/dashboards": + fmt.Fprint(w, `{"dashboards":[{"id":"system","name":"System overview","is_default":true}]}`) + case "/api/dashboards/system": + _ = json.NewEncoder(w).Encode(map[string]json.RawMessage{"spec": spec}) + default: + t.Errorf("unexpected route %s", r.URL.Path) + http.NotFound(w, r) + } + })) + defer server.Close() + targets, _, err := loadQueryTargets(context.Background(), server.Client(), server.URL, "fanout_session=test") + if err != nil { + t.Fatal(err) + } + if requests != 2 || len(targets) != 15 { + t.Fatalf("requests=%d targets=%d", requests, len(targets)) + } +} + +func TestRefreshIncludesAnnotationsAndMeasuresWholeOperation(t *testing.T) { + targets, _, _ := systemQueryTargets(systemSpec(t)) + paths := []string{} + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + paths = append(paths, r.URL.Path) + if r.Method != http.MethodPost || r.Header.Get("Cookie") != "fanout_session=test" || r.Header.Get("Fanout-Request") != "1" || r.Header.Get("Content-Type") != "application/json" { + t.Error("UI headers/method missing") + } + time.Sleep(15 * time.Millisecond) + switch r.URL.Path { + case "/api/panels/query": + w.Write(panelBatch(t, targets[0], 1000, 2000)) + case "/api/annotations": + var body map[string]string + _ = json.NewDecoder(r.Body).Decode(&body) + if !reflect.DeepEqual(body, map[string]string{"from": "1970-01-01T00:00:01.000Z", "to": "1970-01-01T00:00:02.000Z"}) { + t.Errorf("annotation body=%v", body) + } + fmt.Fprint(w, `{"deploys":[],"anomalies":[]}`) + default: + t.Errorf("unexpected path %s", r.URL.Path) + } + })) + defer server.Close() + q := newQueryRun() + q.targets = targets + q.request(context.Background(), server.Client(), server.URL, "fanout_session=test", 0) + if !reflect.DeepEqual(paths, []string{"/api/panels/query", "/api/annotations"}) || q.failures != 0 || q.hist.snapshot().Count != 1 || q.hist.snapshot().MeanMs < 30 { + t.Fatalf("paths=%v failures=%d latency=%+v", paths, q.failures, q.hist.snapshot()) + } + if summary := q.operationSummary("mixed"); !strings.Contains(summary, "| dashboard-refresh | 1 | 1 | 0 |") { + t.Errorf("missing kind evidence: %s", summary) + } +} + +func TestTraceReferencesRemainScopedToDiscoveryWindow(t *testing.T) { + targets, _, _ := systemQueryTargets(systemSpec(t)) + q := newQueryRun() + q.targets = targets + bounds := map[string]drillSelection{} + for i, window := range queryWindows { + from, to := int64(1000+i*10000), int64(2000+i*10000) + if _, ok := q.panelResponse(panelBatch(t, targets[i*3], from, to), targets[i*3]); !ok { + t.Fatal("invalid panel batch") + } + selection := q.selections[window+"/service_map"] + bounds[window] = selection + // Exemplar discovery preserves the absolute panel-response bounds. + body := []byte(fmt.Sprintf(`{"traces":[{"trace_id":"trace-%s","namespace":"bench namespace"}]}`, window)) + if !q.exemplarResponse(body, targets[i*3+1], selection) { + t.Fatal("invalid exemplars") + } + } + count := 0 + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + window := queryWindows[count] + count++ + params := r.URL.Query() + if r.Method != http.MethodGet || r.URL.Path != "/api/traces/trace-"+window || params.Get("namespace") != "bench namespace" || params.Get("from") != bounds[window].from || params.Get("to") != bounds[window].to || params.Get("limit") != "200" || len(params) != 4 || r.Header.Get("Fanout-Request") != "1" { + t.Errorf("incorrect detail request %s", r.URL) + } + fmt.Fprint(w, `{"data":{"spans":[{"span_id":"span"}]}}`) + })) + defer server.Close() + for i := range queryWindows { + q.request(context.Background(), server.Client(), server.URL, "fanout_session=test", i*3+2) + } + if q.failures != 0 || q.hist.snapshot().Count != 5 { + t.Fatalf("failures=%d completed=%d", q.failures, q.hist.snapshot().Count) + } +} + +func TestTraceDetailFallsBackOnlyWithinItsOwnWindow(t *testing.T) { + targets, _, _ := systemQueryTargets(systemSpec(t)) + q := newQueryRun() + q.targets = targets + q.traces["15m"] = traceReference{"other-window", "ns", "old-from", "old-to"} + q.selections["1h/service_map"] = drillSelection{"2026-10-01T12:00:00.000Z", "2026-10-01T13:00:00.000Z", map[string]string{"service": "checkout"}} + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path != "/api/panels/exemplars" { + t.Errorf("wrong fallback path %s", r.URL.Path) + } + var body map[string]any + _ = json.NewDecoder(r.Body).Decode(&body) + if !reflect.DeepEqual(body, decodeBody(t, exemplarBody(targets[4], q.selections["1h/service_map"]))) { + t.Error("incorrect fallback exemplar body") + } + fmt.Fprint(w, `{"traces":[{"trace_id":"this-window","namespace":"ns"}]}`) + })) + defer server.Close() + q.request(context.Background(), server.Client(), server.URL, "fanout_session=test", 5) + if q.failures != 0 || q.traces["1h"].id != "this-window" || q.traces["15m"].id != "other-window" { + t.Fatalf("traces=%v failures=%d", q.traces, q.failures) + } + if !strings.Contains(q.operationSummary("mixed"), "| drill-traces | 1 | 1 |") { + t.Fatal("fallback reported as detail") + } +} + +func TestSustainedQueryGateFailsWhenExemplarsStayEmpty(t *testing.T) { + targets, _, _ := systemQueryTargets(systemSpec(t)) + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + switch r.URL.Path { + case "/api/panels/query": + w.Write(panelBatch(t, targets[0], 1000, 2000)) + case "/api/annotations": + fmt.Fprint(w, `{}`) + case "/api/panels/exemplars": + fmt.Fprint(w, `{"traces":[]}`) + default: + t.Errorf("unexpected request: %s", r.URL.Path) + } + })) + defer server.Close() + q := newQueryRun() + q.targets = targets + for i := 0; i < 2*len(targets); i++ { + q.request(context.Background(), server.Client(), server.URL, "fanout_session=test", i) + } + trial := Trial{ + Spec: PhaseSpec{Kind: "sustained", Signal: Mixed, QueryRate: 30}, + ElapsedSeconds: 1, Accepted: Counts{Spans: 1}, + QueryLatency: q.hist.snapshot(), QueryFailures: q.failures, + QueryKindCompleted: q.completedByKind(), + } + if trial.QueryLatency.Count != 30 || trial.QueryFailures != 0 || trial.QueryKindCompleted["dashboard-refresh"] != 10 || trial.QueryKindCompleted["drill-traces"] != 20 || trial.QueryKindCompleted["trace-detail"] != 0 { + t.Fatalf("unexpected operation counts: %+v", trial) + } + if queryTargetMet(trial) { + t.Fatal("query target met without a successful trace-detail operation") + } + summary := summarize(Report{Settings: Settings{QueryRate: 30}, Trials: []Trial{trial}}, 0, 0) + for _, gate := range summary.Gates { + if gate.Name == "sustained-query-load" { + if gate.Passed || summary.Passed || !strings.Contains(gate.Detail, "zero successful operations: trace-detail") { + t.Fatalf("missing trace detail did not fail the gate: %+v", gate) + } + return + } + } + t.Fatal("sustained query gate missing") +} + +func TestEmptyDiscoveredTraceDetailFails(t *testing.T) { + for _, response := range []string{`{"data":{"spans":[]}}`, `{"data":{}}`, `{"data":{"spans":null}}`} { + t.Run(response, func(t *testing.T) { + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { fmt.Fprint(w, response) })) + defer server.Close() + q := newQueryRun() + q.targets = []queryTarget{{name: "trace-detail", window: "1h"}} + q.traces["1h"] = traceReference{"trace", "ns", "2026-10-01T12:00:00.000Z", "2026-10-01T13:00:00.000Z"} + q.request(context.Background(), server.Client(), server.URL, "fanout_session=test", 0) + if q.failures != 1 || q.hist.snapshot().Count != 0 { + t.Fatal("empty detail counted as success") + } + }) + } +} + +func TestAnyOperationRequestOrPanelErrorFails(t *testing.T) { + for _, defect := range []string{"panel-error", "missing-panel", "annotation-http", "annotation-json", "exemplar-http"} { + t.Run(defect, func(t *testing.T) { + targets, _, _ := systemQueryTargets(systemSpec(t)) + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + switch r.URL.Path { + case "/api/panels/query": + body := panelBatch(t, targets[0], 1000, 2000) + if defect == "panel-error" { + body = []byte(strings.Replace(string(body), `"status":"empty"`, `"status":"error"`, 1)) + } + if defect == "missing-panel" { + body = []byte(`{"results":[{"id":"traffic","status":"empty"}]}`) + } + w.Write(body) + case "/api/annotations": + if defect == "annotation-http" { + http.Error(w, "failed", 500) + } else if defect == "annotation-json" { + fmt.Fprint(w, `broken`) + } else { + fmt.Fprint(w, `{}`) + } + case "/api/panels/exemplars": + http.Error(w, "failed", 500) + default: + t.Errorf("unexpected route %s", r.URL.Path) + } + })) + defer server.Close() + q := newQueryRun() + q.targets = targets + index := 0 + if defect == "exemplar-http" { + index = 1 + q.selections["15m/service_map"] = drillSelection{"2026-10-01T12:00:00.000Z", "2026-10-01T13:00:00.000Z", map[string]string{"service": "checkout"}} + } + q.request(context.Background(), server.Client(), server.URL, "fanout_session=test", index) + if q.failures != 1 || q.hist.snapshot().Count != 0 { + t.Fatalf("failures=%d completed=%d", q.failures, q.hist.snapshot().Count) + } + }) + } +} + +func TestEmptyDashboardDoesNotInventDrillSelection(t *testing.T) { + targets, _, _ := systemQueryTargets(systemSpec(t)) + requests := 0 + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + requests++ + if r.URL.Path == "/api/annotations" { + fmt.Fprint(w, `{}`) + return + } + if r.URL.Path != "/api/panels/query" { + t.Errorf("invented selection: %s", r.URL.Path) + } + body := decodeBody(t, panelBatch(t, targets[0], 1000, 2000)) + for _, raw := range body["results"].([]any) { + result := raw.(map[string]any) + delete(result, "frame") + result["status"] = "empty" + } + _ = json.NewEncoder(w).Encode(body) + })) + defer server.Close() + q := newQueryRun() + q.targets = targets + q.request(context.Background(), server.Client(), server.URL, "fanout_session=test", 2) + if requests != 2 || q.failures != 0 || q.hist.snapshot().Count != 1 || !strings.Contains(q.operationSummary("mixed"), "| dashboard-refresh | 1 | 1 |") { + t.Fatalf("requests=%d failures=%d summary=%s", requests, q.failures, q.operationSummary("mixed")) + } +} + +func TestAnnotationWindowUnionAndDisabledAnnotations(t *testing.T) { + query := queryTarget{timePanels: map[string]bool{"traffic": true, "latency": true}, annotations: true} + body := []byte(`{"results":[{"id":"traffic","status":"empty","from_ms":2000,"to_ms":5000},{"id":"latency","status":"empty","from_ms":1000,"to_ms":3000},{"id":"requests","status":"empty","from_ms":0,"to_ms":9000}]}`) + q := newQueryRun() + annotations, ok := q.panelResponse(body, query) + if !ok || !reflect.DeepEqual(decodeBody(t, annotations), map[string]any{"from": "1970-01-01T00:00:01.000Z", "to": "1970-01-01T00:00:05.000Z"}) { + t.Fatalf("annotations=%s ok=%t", annotations, ok) + } + query.annotations = false + if annotations, ok = q.panelResponse(body, query); !ok || annotations != nil { + t.Fatal("disabled annotations requested") + } + query.annotations, query.timePanels = true, nil + if annotations, ok = q.panelResponse(body, query); !ok || annotations != nil { + t.Fatal("annotations requested without time panels") + } +} + +func TestUIBatchIncludesTextIDsAndRoundedGridWidths(t *testing.T) { + spec := []byte(`{"name":"System overview","panels":[{"id":"z","viz":"text","width":12},{"id":"a","viz":"stat","width":8,"grid":{"w":1}}],"time":{"refresh":"off","compare":"previous_period"}}`) + targets, _, err := systemQueryTargets(spec) + if err != nil { + t.Fatal(err) + } + body := decodeBody(t, targets[0].body) + if !reflect.DeepEqual(body["panels"], []any{"a", "z"}) || !reflect.DeepEqual(body["widths"], map[string]any{"a": float64(100), "z": float64(1600)}) || body["compare"] != true { + t.Fatalf("batch=%v", body) + } + q := newQueryRun() + if _, ok := q.panelResponse([]byte(`{"results":[{"id":"a","status":"empty"}]}`), targets[0]); !ok { + t.Fatal("text panel incorrectly required a result") + } +} + +func TestOperationEvidencePreservesSchemaAndWorkloadIdentity(t *testing.T) { + report := fixtureReport() + before := workloadHash(report.Settings, nil) + q := newQueryRun() + q.kinds["drill-traces"] = &operationStats{dispatched: 1, hist: newHistogram()} + q.kinds["drill-traces"].hist.record(time.Millisecond) + report.Settings.Methodology += q.operationSummary("mixed") + if workloadHash(report.Settings, nil) != before { + t.Fatal("read outcomes changed workload identity") + } + data, err := json.Marshal(report) + if err != nil { + t.Fatal(err) + } + var restored Report + decoder := json.NewDecoder(strings.NewReader(string(data))) + decoder.DisallowUnknownFields() + if err := decoder.Decode(&restored); err != nil { + t.Fatal(err) + } + if !strings.Contains(Markdown(restored), "| drill-traces | 1 | 1 |") { + t.Fatal("operation evidence lost during report roundtrip") + } +} + +func TestDiscoveryFallbackKeepsOneUserOperationPerSlot(t *testing.T) { + targets, _, _ := systemQueryTargets(systemSpec(t)) + paths := []string{} + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + paths = append(paths, r.URL.Path) + switch r.URL.Path { + case "/api/panels/query": + w.Write(panelBatch(t, targets[0], 1000, 2000)) + case "/api/annotations": + fmt.Fprint(w, `{}`) + case "/api/panels/exemplars": + fmt.Fprint(w, `{"traces":[{"trace_id":"trace","namespace":"ns"}]}`) + case "/api/traces/trace": + fmt.Fprint(w, `{"data":{"spans":[{"span_id":"span"}]}}`) + default: + t.Errorf("unexpected path %s", r.URL.Path) + } + })) + defer server.Close() + q := newQueryRun() + q.targets = targets + for i, want := range [][]string{{"/api/panels/query", "/api/annotations"}, {"/api/panels/exemplars"}, {"/api/traces/trace"}} { + paths = nil + q.request(context.Background(), server.Client(), server.URL, "fanout_session=test", 2) + if !reflect.DeepEqual(paths, want) { + t.Fatalf("slot %d: paths=%v want %v", i, paths, want) + } + } + if q.failures != 0 || q.hist.snapshot().Count != 3 { + t.Fatalf("failures=%d latency=%+v", q.failures, q.hist.snapshot()) + } +} diff --git a/internal/bench/remote.go b/internal/bench/remote.go index b0122d3..ee6df06 100644 --- a/internal/bench/remote.go +++ b/internal/bench/remote.go @@ -20,6 +20,7 @@ import ( ) var sshTargetPattern = regexp.MustCompile(`^(?:[A-Za-z0-9._-]+@)?(?:[A-Za-z0-9._-]+|\[[0-9A-Fa-f:]+\])$`) +var fanoutVersionPattern = regexp.MustCompile(`^v[0-9]{4}\.(?:[1-9]|1[0-2])\.(?:0|[1-9][0-9]*)(?:-(?:alpha|beta|rc))?$`) func RunRemote(ctx context.Context, cfg Config, version string, out io.Writer) error { if !sshTargetPattern.MatchString(cfg.Host) || strings.HasPrefix(cfg.Host, "-") { @@ -31,6 +32,9 @@ func RunRemote(ctx context.Context, cfg Config, version string, out io.Writer) e if cfg.FanoutVersion == "" || cfg.FanoutVersion == "unknown" { return errors.New("--fanout-version is required for a remote run") } + if !fanoutVersionPattern.MatchString(cfg.FanoutVersion) { + return fmt.Errorf("invalid Fanout release %q: want vYYYY.M.N[-alpha|-beta|-rc]", cfg.FanoutVersion) + } if _, err := Plan(cfg); err != nil { return err } @@ -221,8 +225,11 @@ func moduleRoot() (string, error) { } func fetchFanout(ctx context.Context, dir, version, goos, goarch string) (string, string, error) { + asset, err := fanoutAsset(version, goos, goarch) + if err != nil { + return "", "", err + } base := "https://github.com/labstack/fanout/releases/download/" + version + "/" - asset := "fanout_" + version + "_" + goos + "_" + goarch + ".tar.gz" sumsURL := base + "SHA256SUMS" assetURL := base + asset sums, err := downloadSmall(ctx, sumsURL, 1<<20) @@ -281,6 +288,16 @@ func fetchFanout(ctx context.Context, dir, version, goos, goarch string) (string return binaryPath, assetURL + "#sha256=" + got, nil } +func fanoutAsset(version, goos, goarch string) (string, error) { + if !fanoutVersionPattern.MatchString(version) { + return "", fmt.Errorf("invalid Fanout release %q", version) + } + if goos != "linux" || (goarch != "amd64" && goarch != "arm64") { + return "", fmt.Errorf("unsupported Fanout platform %s/%s", goos, goarch) + } + return "fanout_" + version + "_" + goos + "_" + goarch + ".tar.gz", nil +} + func downloadSmall(ctx context.Context, url string, limit int64) ([]byte, error) { req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil) if err != nil { @@ -340,7 +357,7 @@ func extractBinary(archivePath, destination, name string) error { func managedRemoteCommand(stage string, cfg Config, version, artifact string) string { args := []string{stage + "/fanout-bench", "managed", "--fanout", stage + "/fanout", "--telemetrygen", stage + "/telemetrygen", "--output", stage + "/results", "--fanout-version", cfg.FanoutVersion, "--fanout-artifact", artifact, "--preset", cfg.Preset, "--repeat", fmt.Sprint(cfg.Repeat), "--batch-size", fmt.Sprint(cfg.BatchSize), "--query-rate", fmt.Sprint(cfg.QueryRate), "--sample-interval", cfg.SampleInterval.String(), "--cooldown", cfg.Cooldown.String(), "--harness-version", version} args = append(args, "--telemetrygen-version", cfg.TelemetrygenVersion) - args = append(args, "--provisioned-host", cfg.Host) + args = append(args, "--provisioned-host", redactedHost(cfg.Host), "--host-description", cfg.HostDescription) args = append(args, "--provisioned-ssh-port", fmt.Sprint(cfg.SSHPort), "--provisioned-ssh-identity", cfg.SSHIdentity) args = append(args, "--processes", fmt.Sprint(cfg.ProcessOverride), "--workers", fmt.Sprint(cfg.WorkerOverride), "--rate", fmt.Sprint(cfg.RateOverride), "--duration", cfg.DurationOverride.String()) quoted := make([]string, len(args)) @@ -355,6 +372,9 @@ func sshArgs(cfg Config) []string { if cfg.SSHIdentity != "" { args = append(args, "-i", cfg.SSHIdentity) } + if cfg.SSHKnownHosts != "" { + args = append(args, "-o", "UserKnownHostsFile="+cfg.SSHKnownHosts) + } return args } func scpArgs(cfg Config) []string { @@ -362,6 +382,9 @@ func scpArgs(cfg Config) []string { if cfg.SSHIdentity != "" { args = append(args, "-i", cfg.SSHIdentity) } + if cfg.SSHKnownHosts != "" { + args = append(args, "-o", "UserKnownHostsFile="+cfg.SSHKnownHosts) + } return args } func sshRun(ctx context.Context, cfg Config, out io.Writer, command string) error { diff --git a/internal/bench/remote_test.go b/internal/bench/remote_test.go index 46fd553..6440b6c 100644 --- a/internal/bench/remote_test.go +++ b/internal/bench/remote_test.go @@ -2,17 +2,74 @@ package bench import ( "archive/tar" + "bytes" "compress/gzip" "context" + "crypto/sha256" "encoding/json" + "fmt" + "io" "net/http" "net/http/httptest" "os" "path/filepath" + "slices" "strings" "testing" ) +type releaseTransport func(*http.Request) (*http.Response, error) + +func (f releaseTransport) RoundTrip(r *http.Request) (*http.Response, error) { return f(r) } + +func TestFetchMaturityReleaseVerifiesNamedAsset(t *testing.T) { + var archive bytes.Buffer + gz := gzip.NewWriter(&archive) + tarball := tar.NewWriter(gz) + if err := tarball.WriteHeader(&tar.Header{Name: "./fanout", Typeflag: tar.TypeReg, Mode: 0o755, Size: 6}); err != nil { + t.Fatal(err) + } + if _, err := tarball.Write([]byte("binary")); err != nil { + t.Fatal(err) + } + if err := tarball.Close(); err != nil { + t.Fatal(err) + } + if err := gz.Close(); err != nil { + t.Fatal(err) + } + digest := fmt.Sprintf("%x", sha256.Sum256(archive.Bytes())) + const base = "https://github.com/labstack/fanout/releases/download/v2026.10.1-alpha/" + const asset = "fanout_v2026.10.1-alpha_linux_amd64.tar.gz" + requests := 0 + previous := http.DefaultClient + t.Cleanup(func() { http.DefaultClient = previous }) + http.DefaultClient = &http.Client{Transport: releaseTransport(func(r *http.Request) (*http.Response, error) { + requests++ + var body []byte + switch r.URL.String() { + case base + "SHA256SUMS": + body = []byte(digest + " " + asset + "\n") + case base + asset: + body = archive.Bytes() + default: + return nil, fmt.Errorf("unexpected release request %s", r.URL) + } + return &http.Response{StatusCode: http.StatusOK, Body: io.NopCloser(bytes.NewReader(body)), Header: make(http.Header)}, nil + })} + path, identity, err := fetchFanout(context.Background(), t.TempDir(), "v2026.10.1-alpha", "linux", "amd64") + if err != nil { + t.Fatal(err) + } + data, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + if string(data) != "binary" || requests != 2 || identity != base+asset+"#sha256="+digest { + t.Fatalf("data=%q requests=%d identity=%s", data, requests, identity) + } +} + func TestSSHTargetValidation(t *testing.T) { valid := []string{"root@example.com", "bench-host", "ubuntu@10.0.0.4", "root@[2001:db8::1]"} invalid := []string{"-oProxyCommand=bad", "root@host;bad", "root@host name", "root@", ""} @@ -28,6 +85,23 @@ func TestSSHTargetValidation(t *testing.T) { } } +func TestFanoutReleaseVersionsAndAssets(t *testing.T) { + for _, version := range []string{"v2026.10.1-alpha", "v2026.10.2-beta", "v2026.10.3-rc", "v2026.10.4", "v2026.1.0"} { + got, err := fanoutAsset(version, "linux", "amd64") + if err != nil { + t.Fatal(err) + } + if want := "fanout_" + version + "_linux_amd64.tar.gz"; got != want { + t.Errorf("asset=%q want %q", got, want) + } + } + for _, version := range []string{"2026.10.1", "v2026.0.1", "v2026.13.1", "v2026.01.1", "v2026.10.01", "v2026.10.1-alpha.1", "v2026.10.1-preview", "v2026.10.1/evil", "v2026.10.1-rc;bad", ""} { + if _, err := fanoutAsset(version, "linux", "amd64"); err == nil { + t.Errorf("accepted invalid release %q", version) + } + } +} + func TestParseRemotePlatformIgnoresLoginNoise(t *testing.T) { osName, arch, err := parseRemotePlatform("Welcome to the host\n__FANOUT_BENCH_PLATFORM__ Linux aarch64\n") if err != nil { @@ -47,13 +121,16 @@ func TestGoInstallOutputUsesCrossCompileDirectory(t *testing.T) { } func TestManagedRemoteCommandQuotesEveryValue(t *testing.T) { - cfg := Config{Host: "root@host", SSHPort: 2222, SSHIdentity: "/keys/bench key", FanoutVersion: "v2026.8.11", Preset: "quick", Repeat: 1, BatchSize: 1000, TelemetrygenVersion: "v0.157.0"} + cfg := Config{Host: "root@192.0.2.10", HostDescription: "Hetzner Cloud ccx33, fsn1, ubuntu-24.04", SSHPort: 2222, SSHIdentity: "/keys/bench key", FanoutVersion: "v2026.10.1-alpha", Preset: "quick", Repeat: 1, BatchSize: 1000, TelemetrygenVersion: "v0.157.0"} got := managedRemoteCommand("/tmp/fanout-bench-safe", cfg, "dev", "asset#sha256=abc") - for _, want := range []string{"'managed'", "'--fanout-version'", "'v2026.8.11'", "'--provisioned-host'", "'root@host'", "'--provisioned-ssh-port'", "'2222'", "'--provisioned-ssh-identity'", "'/keys/bench key'"} { + for _, want := range []string{"'managed'", "'--fanout-version'", "'v2026.10.1-alpha'", "'--provisioned-host'", "'root@'", "'--host-description'", "'Hetzner Cloud ccx33, fsn1, ubuntu-24.04'", "'--provisioned-ssh-port'", "'2222'", "'--provisioned-ssh-identity'", "'/keys/bench key'"} { if !strings.Contains(got, want) { t.Errorf("command missing %s: %s", want, got) } } + if strings.Contains(got, "192.0.2.10") { + t.Fatal("remote agent does not need the SSH address") + } if strings.Contains(got, "Bearer") { t.Fatal("remote command must not carry credentials") } @@ -158,3 +235,26 @@ func TestCopyEvidenceFileOverwritesPreviousRun(t *testing.T) { t.Fatalf("destination=%q", got) } } + +func TestKnownHostsReachesSSHAndSCPArgs(t *testing.T) { + for name, build := range map[string]func(Config) []string{"ssh": sshArgs, "scp": scpArgs} { + t.Run(name, func(t *testing.T) { + cfg := Config{SSHPort: 2222, SSHIdentity: "/keys/benchmark key", SSHKnownHosts: "/tmp/run scope/known_hosts"} + args := build(cfg) + for _, pair := range [][]string{{"-o", "UserKnownHostsFile=" + cfg.SSHKnownHosts}, {"-i", cfg.SSHIdentity}} { + index := slices.Index(args, pair[0]) + if pair[0] == "-o" { + index = slices.Index(args, pair[1]) - 1 + } + if index < 0 || index+1 >= len(args) || args[index] != pair[0] || args[index+1] != pair[1] { + t.Fatalf("missing option pair %q in %q", pair, args) + } + } + for _, arg := range build(Config{SSHPort: 22}) { + if strings.HasPrefix(arg, "UserKnownHostsFile=") { + t.Fatal("default known-hosts behavior changed") + } + } + }) + } +} diff --git a/internal/bench/report.go b/internal/bench/report.go index 8773031..ea5a975 100644 --- a/internal/bench/report.go +++ b/internal/bench/report.go @@ -146,7 +146,11 @@ func Markdown(r Report) string { } fmt.Fprintf(&b, "\n**Meaning:** accepted throughput is the delta of `fanout_ingest_rows_total` divided by the interval between the two counter scrapes that bracket the phase. It is not telemetrygen's requested or locally generated count.\n\n") fmt.Fprintf(&b, "## Test identity\n\n| Setting | Value |\n|---|---|\n") - settings := [][2]string{{"Fanout version", display(r.Build.FanoutVersion)}, {"Fanout commit", display(r.Build.FanoutCommit)}, {"Fanout artifact", display(r.Build.FanoutArtifact)}, {"Harness", r.Build.HarnessVersion + " (`" + r.Build.HarnessCommit + "`)"}, {"telemetrygen", display(r.Build.TelemetrygenVersion)}, {"Machine", fmt.Sprintf("%s/%s · %s · %d logical CPUs · %s RAM", r.Host.OS, r.Host.Architecture, display(r.Host.CPUModel), r.Host.LogicalCPUs, formatBytes(float64(r.Host.MemoryBytes)))}, {"Generator placement", map[bool]string{true: "same host as Fanout", false: "separate/remote"}[r.Host.GeneratorLocal]}, {"Preset", r.Settings.Preset}, {"Transport", r.Settings.Transport}, {"Batch size", fmt.Sprintf("%d", r.Settings.BatchSize)}, {"Trace shape", "1 parent + 1 child span per telemetrygen trace tick"}, {"Repeat", fmt.Sprintf("%d", r.Settings.Repeat)}, {"Sample interval", fmt.Sprintf("%.1fs", r.Settings.SampleSeconds)}, {"Workload SHA-256", "`" + r.WorkloadSHA256 + "`"}} + machine := display(r.Host.CPUModel) + if r.Host.Description != "" { + machine += " · " + r.Host.Description + } + settings := [][2]string{{"Fanout version", display(r.Build.FanoutVersion)}, {"Fanout commit", display(r.Build.FanoutCommit)}, {"Fanout artifact", display(r.Build.FanoutArtifact)}, {"Harness", r.Build.HarnessVersion + " (`" + r.Build.HarnessCommit + "`)"}, {"telemetrygen", display(r.Build.TelemetrygenVersion)}, {"Machine", fmt.Sprintf("%s/%s · %s · %d logical CPUs · %s RAM", r.Host.OS, r.Host.Architecture, machine, r.Host.LogicalCPUs, formatBytes(float64(r.Host.MemoryBytes)))}, {"Generator placement", map[bool]string{true: "same host as Fanout", false: "separate/remote"}[r.Host.GeneratorLocal]}, {"Preset", r.Settings.Preset}, {"Transport", r.Settings.Transport}, {"Batch size", fmt.Sprintf("%d", r.Settings.BatchSize)}, {"Trace shape", "1 parent + 1 child span per telemetrygen trace tick"}, {"Repeat", fmt.Sprintf("%d", r.Settings.Repeat)}, {"Sample interval", fmt.Sprintf("%.1fs", r.Settings.SampleSeconds)}, {"Workload SHA-256", "`" + r.WorkloadSHA256 + "`"}} for _, row := range settings { fmt.Fprintf(&b, "| %s | %s |\n", row[0], row[1]) } @@ -159,7 +163,7 @@ func Markdown(r Report) string { fmt.Fprintf(&b, "- %s **%s:** %s\n", map[bool]string{true: "✅", false: "❌"}[gate.Passed], gate.Name, gate.Detail) } if r.Settings.QueryRate > 0 { - fmt.Fprintf(&b, "\n## Read-under-write latency\n\nThe mixed phases used a bounded open-loop scheduler targeting %.0f authenticated dashboard queries/s across overview, topology, performance, trace, and logs endpoints.\n\n", r.Settings.QueryRate) + fmt.Fprintf(&b, "\n## Read-under-write latency\n\nThe mixed phases used a bounded open-loop scheduler targeting %.0f authenticated user operations/s across System overview dashboard refresh (batched panels and annotations), exemplar drill-down, and trace detail. Latency covers every request in an operation; operation kinds and their counts/latencies appear in the methodology.\n\n", r.Settings.QueryRate) fmt.Fprintf(&b, "| Phase | Target/s | Completed/s | Attempted | p50 | p95 | p99 | Errors | Shed | Canceled at end |\n|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|\n") for _, t := range r.Trials { if t.Spec.Signal != Mixed || t.Spec.QueryRate <= 0 { @@ -169,7 +173,7 @@ func Markdown(r Report) string { } fmt.Fprintln(&b) } - fmt.Fprintf(&b, "\n## Methodology and limits\n\n") + fmt.Fprintf(&b, "\n## Methodology and limits\n\n%s\n\n", r.Settings.Methodology) for _, warning := range r.Summary.Warnings { fmt.Fprintf(&b, "- %s\n", warning) } @@ -178,8 +182,23 @@ func Markdown(r Report) string { } func reproductionCommand(c Config) string { + command := reproductionRunCommand(c) + if c.HostDescription != "" { + command += " --host-description " + shellQuote(c.HostDescription) + } + return command +} + +func redactedHost(host string) string { + if user, _, ok := strings.Cut(host, "@"); ok { + return user + "@" + } + return "" +} + +func reproductionRunCommand(c Config) string { if c.Host != "" { - parts := []string{"fanout-bench run", "--host " + shellQuote(c.Host), "--fanout-version " + shellQuote(c.FanoutVersion), "--preset " + c.Preset, "--repeat " + fmt.Sprint(c.Repeat), "--telemetrygen-version " + shellQuote(c.TelemetrygenVersion), "--batch-size " + fmt.Sprint(c.BatchSize), "--sample-interval " + c.SampleInterval.String(), "--cooldown " + c.Cooldown.String()} + parts := []string{"fanout-bench run", "--host " + shellQuote(redactedHost(c.Host)), "--fanout-version " + shellQuote(c.FanoutVersion), "--preset " + c.Preset, "--repeat " + fmt.Sprint(c.Repeat), "--telemetrygen-version " + shellQuote(c.TelemetrygenVersion), "--batch-size " + fmt.Sprint(c.BatchSize), "--sample-interval " + c.SampleInterval.String(), "--cooldown " + c.Cooldown.String()} if c.SSHPort != 0 && c.SSHPort != 22 { parts = append(parts, "--ssh-port "+fmt.Sprint(c.SSHPort)) } diff --git a/internal/bench/report_test.go b/internal/bench/report_test.go index d6a9d61..f0f3707 100644 --- a/internal/bench/report_test.go +++ b/internal/bench/report_test.go @@ -1,6 +1,8 @@ package bench import ( + "context" + "encoding/json" "os" "path/filepath" "strings" @@ -9,7 +11,7 @@ import ( ) func fixtureReport() Report { - return Report{SchemaVersion: SchemaVersion, RunID: "test", StartedAt: time.Date(2026, 8, 28, 12, 0, 0, 0, time.UTC), Build: Build{HarnessVersion: "v1", HarnessCommit: "abc", TelemetrygenVersion: "v0.157.0", FanoutVersion: "v2026.8.11"}, Host: Host{OS: "darwin", Architecture: "arm64", CPUModel: "M3 Max", LogicalCPUs: 14, MemoryBytes: 36 << 30, GeneratorLocal: true}, Settings: Settings{Preset: "quick", Transport: "plaintext gRPC", BatchSize: 1000, SampleSeconds: 1, Methodology: "accepted"}, WorkloadSHA256: strings.Repeat("a", 64), Trials: []Trial{{Spec: PhaseSpec{Name: "traces", Signal: Traces, Kind: "sweep", ProcessesPerSignal: 1, WorkersPerProcess: 2}, ElapsedSeconds: 5, OfferedPerSecond: 2000, AcceptedPerSecond: 1900, AcceptanceRatio: .95, Accepted: Counts{Spans: 9500}, Resources: Resources{AverageCPUCores: 1.2, PeakRSSBytes: 100 << 20}, Classification: "target-sustained"}}, Summary: Summary{PeakTraces: Peak{"traces", 2000, 1900, 1.2, 100 << 20}, Passed: true, Gates: []Gate{{"counters", true, "ok"}}, Warnings: []string{"synthetic"}}, ReproductionCommand: "FANOUT_BENCH_INGEST_TOKEN=… fanout-bench run"} + return Report{SchemaVersion: SchemaVersion, RunID: "test", StartedAt: time.Date(2026, 10, 14, 12, 0, 0, 0, time.UTC), Build: Build{HarnessVersion: "v1", HarnessCommit: "abc", TelemetrygenVersion: "v0.157.0", FanoutVersion: "v2026.10.1-alpha"}, Host: Host{OS: "linux", Architecture: "amd64", CPUModel: "AMD EPYC", LogicalCPUs: 8, MemoryBytes: 32 << 30, GeneratorLocal: true}, Settings: Settings{Preset: "quick", Transport: "plaintext gRPC", BatchSize: 1000, SampleSeconds: 1, Methodology: "accepted"}, WorkloadSHA256: strings.Repeat("a", 64), Trials: []Trial{{Spec: PhaseSpec{Name: "traces", Signal: Traces, Kind: "sweep", ProcessesPerSignal: 1, WorkersPerProcess: 2}, ElapsedSeconds: 5, OfferedPerSecond: 2000, AcceptedPerSecond: 1900, AcceptanceRatio: .95, Accepted: Counts{Spans: 9500}, Resources: Resources{AverageCPUCores: 1.2, PeakRSSBytes: 100 << 20}, Classification: "target-sustained"}}, Summary: Summary{PeakTraces: Peak{"traces", 2000, 1900, 1.2, 100 << 20}, Passed: true, Gates: []Gate{{"counters", true, "ok"}}, Warnings: []string{"synthetic"}}, ReproductionCommand: "FANOUT_BENCH_INGEST_TOKEN=… fanout-bench run"} } func TestArtifactsAreCompleteAndContainNoSecrets(t *testing.T) { @@ -51,15 +53,15 @@ func TestMarkdownExplainsAcceptedRowsAndLimits(t *testing.T) { } func TestReproductionCommandPutsSecretEnvironmentFirst(t *testing.T) { - got := reproductionCommand(Config{Preset: "quick", Endpoint: "localhost:4317", MetricsURL: "http://localhost", FanoutVersion: "x", Repeat: 1, QueryURL: "http://localhost", QueryRate: 5}) + got := reproductionCommand(Config{Preset: "quick", Endpoint: "localhost:7520", MetricsURL: "http://localhost", FanoutVersion: "x", Repeat: 1, QueryURL: "http://localhost", QueryRate: 5}) if !strings.HasPrefix(got, "FANOUT_BENCH_INGEST_TOKEN=… FANOUT_BENCH_QUERY_COOKIE=… fanout-bench run") { t.Fatalf("bad command: %s", got) } } func TestRemoteReproductionCommandIncludesSSHConnection(t *testing.T) { - got := reproductionCommand(Config{Host: "root@bench", SSHPort: 2222, SSHIdentity: "/keys/bench key", Preset: "quick", FanoutVersion: "v1", TelemetrygenVersion: "v2", Repeat: 1, BatchSize: 1000, SampleInterval: time.Second, Cooldown: time.Second}) - for _, want := range []string{"--host 'root@bench'", "--ssh-port 2222", "--ssh-identity '/keys/bench key'"} { + got := reproductionCommand(Config{Host: "root@192.0.2.10", SSHKnownHosts: "/tmp/run/known_hosts", SSHPort: 2222, SSHIdentity: "/keys/bench key", Preset: "quick", FanoutVersion: "v2026.10.1-alpha", TelemetrygenVersion: "v0.157.0", Repeat: 1, BatchSize: 1000, SampleInterval: time.Second, Cooldown: time.Second}) + for _, want := range []string{"--host 'root@'", "--ssh-port 2222", "--ssh-identity '/keys/bench key'"} { if !strings.Contains(got, want) { t.Errorf("command missing %q: %s", want, got) } @@ -92,3 +94,69 @@ func TestRenderedRatesSurviveZeroElapsedTrial(t *testing.T) { } } } + +func TestReproductionCommandRedactsAddress(t *testing.T) { + for _, target := range []struct{ host, redacted string }{ + {"root@192.0.2.10", "root@"}, + {"benchmark@example.test", "benchmark@"}, + {"192.0.2.10", ""}, + {"root@[2001:db8::1]", "root@"}, + } { + t.Run(target.host, func(t *testing.T) { + report := fixtureReport() + report.ReproductionCommand = reproductionCommand(Config{Host: target.host, SSHKnownHosts: "/tmp/run/known_hosts"}) + data, err := json.Marshal(report) + if err != nil { + t.Fatal(err) + } + if strings.Contains(string(data), target.host) || strings.Contains(string(data), "ssh-known-hosts") || strings.Contains(string(data), "/tmp/run/known_hosts") { + t.Fatalf("report retained address or run-scoped known-hosts: %s", data) + } + if !strings.Contains(report.ReproductionCommand, "--host "+shellQuote(target.redacted)) { + t.Fatalf("missing redacted host: %s", report.ReproductionCommand) + } + }) + } +} + +func TestHostDescriptionRecordedAndRendered(t *testing.T) { + // Cancel before load: report initialization must still preserve host identity. + ctx, cancel := context.WithCancel(context.Background()) + cancel() + description := "Hetzner Cloud ccx33, fsn1, ubuntu-24.04" + generator := filepath.Join(t.TempDir(), "telemetrygen") + if err := os.WriteFile(generator, nil, 0o600); err != nil { + t.Fatal(err) + } + runner := Runner{Config: Config{Host: "root@192.0.2.10", HostDescription: description, Telemetrygen: generator, Token: "test-token", Endpoint: "localhost:7520", MetricsURL: "http://localhost:7520/metrics", Preset: "quick", Repeat: 1, BatchSize: 1000, SampleInterval: time.Second, GeneratorTimeout: time.Second}} + report, err := runner.Run(ctx) + if err == nil || report == nil { + t.Fatalf("expected partial report on cancellation, got report=%v err=%v", report, err) + } + dir := t.TempDir() + if err := WriteArtifacts(dir, *report); err != nil { + t.Fatal(err) + } + data, err := os.ReadFile(filepath.Join(dir, "report.json")) + if err != nil { + t.Fatal(err) + } + var recorded struct { + Host struct { + Description string `json:"description"` + } `json:"host"` + } + if err := json.Unmarshal(data, &recorded); err != nil { + t.Fatal(err) + } + if recorded.Host.Description != description { + t.Fatalf("recorded description=%q", recorded.Host.Description) + } + markdown, err := os.ReadFile(filepath.Join(dir, "report.md")) + if err != nil { + t.Fatal(err) + } + if !strings.Contains(string(markdown), display(report.Host.CPUModel)+" · "+description) { + t.Fatal("host description missing beside CPU model") + } +} diff --git a/internal/bench/runner.go b/internal/bench/runner.go index fce8fd5..3200d1e 100644 --- a/internal/bench/runner.go +++ b/internal/bench/runner.go @@ -27,6 +27,8 @@ type Runner struct { Out io.Writer DeferSummary bool client *http.Client + queryTargets []queryTarget + readSummary string } func (r *Runner) Run(ctx context.Context) (*Report, error) { @@ -47,6 +49,13 @@ func (r *Runner) Run(ctx context.Context) (*Report, error) { if err != nil { return nil, err } + var readMethod string + if r.Config.QueryRate > 0 { + r.queryTargets, readMethod, err = loadQueryTargets(ctx, r.client, r.Config.QueryURL, r.Config.QueryCookie) + if err != nil { + return nil, fmt.Errorf("read workload: %s", redact(err.Error(), r.Config.Token, r.Config.QueryCookie, r.Config.MetricsToken)) + } + } started := time.Now().UTC() report := &Report{ SchemaVersion: SchemaVersion, @@ -56,6 +65,8 @@ func (r *Runner) Run(ctx context.Context) (*Report, error) { Host: detectHost(r.Config.SameHost), Settings: settingsFrom(r.Config), } + report.Host.Description = r.Config.HostDescription + report.Settings.Methodology += readMethod report.WorkloadSHA256 = workloadHash(report.Settings, plan) report.ReproductionCommand = reproductionCommand(r.Config) printRunHeader(r.Out, report, plan) @@ -79,6 +90,7 @@ func (r *Runner) Run(ctx context.Context) (*Report, error) { return r.abandon(report, fmt.Errorf("phase %s: %w", spec.Name, err)) } report.Trials = append(report.Trials, trial) + report.Settings.Methodology += r.readSummary printTrialLine(r.Out, trial) } @@ -110,7 +122,7 @@ func (r *Runner) Run(ctx context.Context) (*Report, error) { func (r *Runner) abandon(report *Report, cause error) (*Report, error) { report.FinishedAt = time.Now().UTC() report.Summary = summarize(*report, report.Summary.CooldownRSS, report.Summary.CooldownHeap) - report.Summary.Gates = append(report.Summary.Gates, Gate{"run-completed", false, redact(cause.Error(), r.Config.Token, r.Config.QueryCookie)}) + report.Summary.Gates = append(report.Summary.Gates, Gate{"run-completed", false, redact(cause.Error(), r.Config.Token, r.Config.QueryCookie, r.Config.MetricsToken)}) report.Summary.Passed = false report.Summary.Warnings = append(report.Summary.Warnings, "This run ended before the suite finished. The phases recorded here are partial evidence and must not be published as a result.") return report, cause @@ -134,7 +146,7 @@ func (r *Runner) validate() error { case cfg.QueryRate < 0: return errors.New("query rate cannot be negative") case cfg.QueryRate > 10_000: - return errors.New("query rate cannot exceed 10000 requests/s") + return errors.New("query rate cannot exceed 10000 operations/s") case cfg.QueryRate > 0 && strings.TrimSpace(cfg.QueryURL) == "": return errors.New("query URL is required when query rate is positive") case cfg.QueryRate > 0 && strings.TrimSpace(cfg.QueryCookie) == "": @@ -163,6 +175,7 @@ func (r *Runner) runPhase(ctx context.Context, spec PhaseSpec) (Trial, error) { go r.sample(phaseCtx, started, samplesDone) queries := newQueryRun() if spec.QueryRate > 0 { + queries.targets = r.queryTargets // Queries run under the parent context, not the phase context, so that // finish() can drain in-flight requests instead of cancelling them. queries.start(ctx, r.client, r.Config.QueryURL, r.Config.QueryCookie, spec.QueryRate) @@ -213,15 +226,17 @@ func (r *Runner) runPhase(ctx context.Context, spec PhaseSpec) (Trial, error) { RowsDropped: dropped, GeneratorProcesses: processCount, GeneratorFailures: failures, ProcessRestarted: restarted, Resources: resources, QueryLatency: queries.hist.snapshot(), QueryAttempts: queries.attempts, QueryFailures: queries.failures, - QueryShed: queries.shed, QueryCanceled: queries.canceled, + QueryKindCompleted: queries.completedByKind(), + QueryShed: queries.shed, QueryCanceled: queries.canceled, Samples: samples, } trial.Classification, trial.ClassificationNotes = classify(trial) + r.readSummary = queries.operationSummary(spec.Name) if trial.Spec.QueryRate > 0 && !queryTargetMet(trial) { trial.ClassificationNotes = append(trial.ClassificationNotes, "the requested query rate was not sustained") } if runErr != nil { - trial.ClassificationNotes = append(trial.ClassificationNotes, redact(runErr.Error(), r.Config.Token, r.Config.QueryCookie)) + trial.ClassificationNotes = append(trial.ClassificationNotes, redact(runErr.Error(), r.Config.Token, r.Config.QueryCookie, r.Config.MetricsToken)) } return trial, nil } @@ -266,7 +281,7 @@ func (r *Runner) runGenerator(ctx context.Context, signal Signal, spec PhaseSpec output := &boundedBuffer{limit: 16 << 10} cmd.Stdout, cmd.Stderr = output, output if err := cmd.Run(); err != nil { - return fmt.Errorf("telemetrygen %s: %w: %s", signal, err, redact(output.String(), r.Config.Token, r.Config.QueryCookie)) + return fmt.Errorf("telemetrygen %s: %w: %s", signal, err, redact(output.String(), r.Config.Token, r.Config.QueryCookie, r.Config.MetricsToken)) } return nil } @@ -414,10 +429,23 @@ func queryTargetMet(t Trial) bool { if t.ElapsedSeconds <= 0 { return false } + if t.Spec.Kind == "sustained" && len(missingQueryKinds(t)) > 0 { + return false + } completedRate := perSecond(float64(t.QueryLatency.Count), t.ElapsedSeconds) return completedRate >= t.Spec.QueryRate*.95 && t.QueryFailures == 0 && t.QueryShed == 0 } +func missingQueryKinds(t Trial) []string { + var missing []string + for _, kind := range []string{"dashboard-refresh", "drill-traces", "trace-detail"} { + if t.QueryKindCompleted[kind] == 0 { + missing = append(missing, kind) + } + } + return missing +} + func summarize(report Report, cooldownRSS, cooldownHeap float64) Summary { s := Summary{CooldownRSS: cooldownRSS, CooldownHeap: cooldownHeap, Passed: true} for _, trial := range report.Trials { @@ -475,6 +503,9 @@ func summarize(report Report, cooldownRSS, cooldownHeap float64) Summary { queryOK = queryTargetMet(t) completedRate := perSecond(float64(t.QueryLatency.Count), t.ElapsedSeconds) detail = fmt.Sprintf("%.1f/%.1f completed/s, %d errors, %d shed", completedRate, t.Spec.QueryRate, t.QueryFailures, t.QueryShed) + if missing := missingQueryKinds(t); len(missing) > 0 { + detail += "; zero successful operations: " + strings.Join(missing, ", ") + } break } s.Gates = append(s.Gates, Gate{"sustained-query-load", queryOK, detail}) @@ -509,6 +540,9 @@ func settingsFrom(c Config) Settings { } func workloadHash(settings Settings, plan []PhaseSpec) string { + // Per-operation latency results share the methodology field; only its + // workload definition belongs in the workload identity. + settings.Methodology, _, _ = strings.Cut(settings.Methodology, "\n\nRead operations for ") payload, _ := json.Marshal(struct { Settings Settings Plan []PhaseSpec diff --git a/internal/bench/runner_test.go b/internal/bench/runner_test.go index ab3e465..f2213cd 100644 --- a/internal/bench/runner_test.go +++ b/internal/bench/runner_test.go @@ -8,7 +8,7 @@ import ( ) func TestGeneratorArgsMatchPinnedTelemetrygenCLI(t *testing.T) { - cfg := Config{Endpoint: "localhost:4317", Token: "token with space", GeneratorTimeout: 30, Insecure: true} + cfg := Config{Endpoint: "localhost:7520", Token: "token with space", GeneratorTimeout: 30, Insecure: true} spec := PhaseSpec{Duration: 5, RatePerWorker: 2500, WorkersPerProcess: 8, BatchSize: 1000} args := generatorArgs(cfg, Traces, spec) for _, want := range []string{"--otlp-header", "Authorization=\"Bearer token with space\"", "--child-spans", "1", "--otlp-insecure"} { diff --git a/internal/bench/testdata/fanout-v2026.10.1-alpha-env.txt b/internal/bench/testdata/fanout-v2026.10.1-alpha-env.txt new file mode 100644 index 0000000..ce3b226 --- /dev/null +++ b/internal/bench/testdata/fanout-v2026.10.1-alpha-env.txt @@ -0,0 +1,46 @@ +FANOUT_ADDR +FANOUT_AI_MODEL +FANOUT_AI_PROVIDER +FANOUT_ALERTS_ENABLED +FANOUT_ALERTS_EVALUATION_INTERVAL +FANOUT_ALERTS_HISTORY_DAYS +FANOUT_AUTH_CODE_SECRET +FANOUT_AUTH_MODE +FANOUT_DATA_DIR +FANOUT_DEFAULT_NAMESPACE +FANOUT_DUCKDB_MAX_CONNECTIONS +FANOUT_DUCKDB_MEMORY +FANOUT_DUCKDB_THREADS +FANOUT_INGEST_BATCH_SIZE +FANOUT_INGEST_MAX_IN_FLIGHT_BYTES +FANOUT_MAINTENANCE_INTERVAL +FANOUT_MCP_ENABLED +FANOUT_METRICS_PUBLIC +FANOUT_METRICS_TOKEN +FANOUT_OIDC_ADMIN_GROUPS +FANOUT_OIDC_ALLOWED_DOMAINS +FANOUT_OIDC_ALLOWED_GROUPS +FANOUT_OIDC_AUTO_PROVISION +FANOUT_OIDC_CLIENT_ID +FANOUT_OIDC_CLIENT_SECRET +FANOUT_OIDC_DEFAULT_ROLE +FANOUT_OIDC_EMAIL_CLAIM +FANOUT_OIDC_EMAIL_VERIFICATION +FANOUT_OIDC_ISSUER_URL +FANOUT_OIDC_OPERATOR_GROUPS +FANOUT_PPROF_ENABLED +FANOUT_PUBLIC_URL +FANOUT_RETENTION_DAYS +FANOUT_ROLLUP_INTERVAL +FANOUT_ROLLUP_SKIP_TO_LATEST +FANOUT_SELF_SIGNUP +FANOUT_SESSION_ABSOLUTE_TTL +FANOUT_SESSION_IDLE_TTL +FANOUT_SMTP_FROM +FANOUT_SMTP_HOST +FANOUT_SMTP_PASSWORD +FANOUT_SMTP_PORT +FANOUT_SMTP_USERNAME +FANOUT_TLS_CERT_FILE +FANOUT_TLS_KEY_FILE +FANOUT_TRUSTED_PROXY_CIDRS diff --git a/internal/bench/testdata/system-overview-v2026.10.1-alpha.json b/internal/bench/testdata/system-overview-v2026.10.1-alpha.json new file mode 100644 index 0000000..1ab0cd2 --- /dev/null +++ b/internal/bench/testdata/system-overview-v2026.10.1-alpha.json @@ -0,0 +1,347 @@ +{ + "version": 1, + "name": "System overview", + "description": "Traffic, errors and latency, reachability, host and container capacity, and service versions.", + "time": { + "range": "1h", + "refresh": "30s" + }, + "variables": [ + { + "name": "service", + "kind": "query", + "from": "spans", + "field": "service", + "default": "$__all", + "include_all": true + } + ], + "annotations": { + "deploys": true + }, + "panels": [ + { + "id": "requests", + "title": "Requests per second", + "viz": "stat", + "query": { + "from": "spans", + "where": [ + "kind = 'SPAN_KIND_SERVER'", + "service = $service" + ], + "measures": [ + "rate()" + ] + }, + "better": "higher" + }, + { + "id": "errors", + "title": "Error rate", + "viz": "stat", + "query": { + "from": "spans", + "where": [ + "kind = 'SPAN_KIND_SERVER'", + "service = $service" + ], + "measures": [ + "error_rate()" + ] + }, + "thresholds": [ + { + "value": 1, + "status": "warn" + }, + { + "value": 5, + "status": "bad" + } + ], + "better": "lower" + }, + { + "id": "latency_p95", + "title": "p95 latency", + "viz": "stat", + "query": { + "from": "spans", + "where": [ + "kind = 'SPAN_KIND_SERVER'", + "service = $service" + ], + "measures": [ + "p95(duration_ms)" + ] + }, + "thresholds": [ + { + "value": 750, + "status": "warn" + }, + { + "value": 2000, + "status": "bad" + } + ], + "better": "lower" + }, + { + "id": "error_logs", + "title": "Error logs", + "viz": "stat", + "query": { + "from": "logs", + "where": [ + "upper(severity) IN ('ERROR', 'FATAL')", + "service = $service" + ], + "measures": [ + "count()" + ] + }, + "thresholds": [ + { + "value": 1, + "status": "warn" + } + ], + "better": "lower" + }, + { + "id": "traffic", + "title": "Requests by service", + "viz": "timeseries", + "query": { + "from": "spans", + "where": [ + "kind = 'SPAN_KIND_SERVER'", + "service = $service" + ], + "measures": [ + "rate()" + ], + "by": [ + "service" + ], + "bucket": "auto" + }, + "options": { + "style": "stacked" + } + }, + { + "id": "latency", + "title": "p95 latency by service", + "viz": "timeseries", + "query": { + "from": "spans", + "where": [ + "kind = 'SPAN_KIND_SERVER'", + "service = $service" + ], + "measures": [ + "p95(duration_ms)" + ], + "by": [ + "service" + ], + "bucket": "auto" + }, + "thresholds": [ + { + "value": 2000, + "status": "bad" + } + ], + "better": "lower" + }, + { + "id": "error_services", + "title": "Error rate by service", + "viz": "bar", + "width": 4, + "query": { + "from": "spans", + "where": [ + "kind = 'SPAN_KIND_SERVER'" + ], + "measures": [ + "error_rate()" + ], + "by": [ + "service" + ], + "limit": 10 + }, + "click": { + "set_variable": "service" + } + }, + { + "id": "endpoints", + "title": "Slowest endpoints", + "viz": "table", + "width": 8, + "query": { + "from": "spans", + "where": [ + "kind = 'SPAN_KIND_SERVER'", + "service = $service", + "http_route \u003c\u003e ''" + ], + "measures": [ + "p95(duration_ms)", + "error_rate()", + "rate()" + ], + "by": [ + "service", + "http_route" + ], + "sort": "p95", + "limit": 15 + } + }, + { + "id": "check_states", + "title": "Is it reachable?", + "description": "Endpoint and host.name; silence is bad after two intervals. Labels warn if all checks run on service hosts. Supplied by the Collector `httpcheck` receiver. [Watch hosts, containers and endpoints](https://fanout.run/start/send-telemetry#watch-hosts-containers-and-endpoints).", + "viz": "state_timeline", + "width": 12, + "sql": "WITH raw AS (\n SELECT series_id,ingested_unix_nano,namespace,time,value,type,attributes['http.url']::VARCHAR AS url,\n attributes['http.method']::VARCHAR AS method,resource['host.name']::VARCHAR AS host,\n attributes['http.status_class']::VARCHAR AS class,attributes['http.status_code']::VARCHAR AS code\n FROM metrics WHERE time \u003e= make_timestamp_ns(epoch_ns($__from::TIMESTAMP_NS)-600000000000)::TIMESTAMPTZ_NS\n AND time \u003c $__to::TIMESTAMP_NS::TIMESTAMPTZ_NS AND name='httpcheck.status' QUALIFY row_number() OVER (PARTITION BY series_id,time ORDER BY ingested_unix_nano DESC,value DESC)=1),\n keyed AS (SELECT *,to_json([namespace,url,method,host])::VARCHAR AS entity,\n coalesce(namespace,'')||' / '||coalesce(method,'method unknown')||' '||coalesce(url,'endpoint unknown')||' from '||coalesce(host,'vantage unknown') AS endpoint_from FROM raw),\n service_hosts AS (SELECT DISTINCT namespace,resource['host.name']::VARCHAR AS host FROM spans WHERE $__window(start_time) AND service IS NOT NULL AND service\u003c\u003e''),\n check_hosts AS (SELECT DISTINCT namespace,host FROM keyed WHERE $__window(time)),\n vantage AS (SELECT count(*)\u003e0 AND bool_and(coalesce(host\u003c\u003e'',false) AND EXISTS (SELECT 1 FROM service_hosts s WHERE s.namespace=check_hosts.namespace AND s.host=check_hosts.host)) AS all_local FROM check_hosts),\n inventory AS (SELECT DISTINCT entity,endpoint_from||CASE WHEN all_local THEN ' (all checks run on service hosts; outside reachability unverified)' ELSE '' END AS endpoint_from FROM keyed CROSS JOIN vantage WHERE $__window(time)),\n attempts AS (SELECT entity,time,\n CASE WHEN bool_or(url IS NULL OR url='' OR method IS NULL OR method='' OR host IS NULL OR host='' OR type NOT IN ('gauge','updown') OR NOT isfinite(value) OR (class IS NULL AND code IS NULL)) THEN 2\n WHEN bool_or(value\u003e0 AND (class IN ('2xx','3xx') OR try_cast(code AS INTEGER) BETWEEN 200 AND 399)) THEN 0 ELSE 1 END AS failure\n FROM keyed GROUP BY entity,time),\n gaps AS (SELECT *,epoch_ns(time::TIMESTAMP_NS)::HUGEINT-epoch_ns(lag(time) OVER (PARTITION BY entity ORDER BY time)::TIMESTAMP_NS)::HUGEINT AS gap_ns FROM attempts),\n cadence AS (SELECT *,last_value(CASE WHEN gap_ns\u003e0 THEN gap_ns END IGNORE NULLS) OVER (PARTITION BY entity ORDER BY time) AS interval_ns FROM gaps),\n buckets AS (SELECT b::TIMESTAMPTZ_NS AS time,least((b+INTERVAL '1 minute')::TIMESTAMPTZ_NS,$__to::TIMESTAMP_NS::TIMESTAMPTZ_NS) AS at\n FROM generate_series(time_bucket(INTERVAL '1 minute',$__from::TIMESTAMP_NS),$__to::TIMESTAMP_NS,INTERVAL '1 minute') t(b) WHERE b::TIMESTAMPTZ_NS \u003c $__to::TIMESTAMP_NS::TIMESTAMPTZ_NS),\n grid AS (SELECT * FROM inventory CROSS JOIN buckets),\n states AS (SELECT g.time,g.endpoint_from,g.entity,\n CASE WHEN a.interval_ns IS NULL OR a.failure=2 THEN 2 WHEN epoch_ns(g.at::TIMESTAMP_NS)::HUGEINT-epoch_ns(a.time::TIMESTAMP_NS)::HUGEINT\u003e=2*a.interval_ns THEN 2 ELSE a.failure END::DOUBLE AS failure,\n CASE WHEN a.interval_ns IS NULL THEN 'reporting interval unknown' WHEN epoch_ns(g.at::TIMESTAMP_NS)::HUGEINT-epoch_ns(a.time::TIMESTAMP_NS)::HUGEINT\u003e=2*a.interval_ns THEN 'no recent result' WHEN a.failure=2 THEN 'check metadata unsupported or missing' WHEN a.failure=1 THEN 'down' ELSE 'reachable' END AS state\n FROM grid g ASOF LEFT JOIN cadence a ON g.entity=a.entity AND g.at\u003ea.time),\n ranked AS (SELECT *,arg_max(failure,time) OVER (PARTITION BY entity) AS current_failure FROM states)\n SELECT time,endpoint_from,failure,state FROM ranked ORDER BY current_failure DESC,time DESC,endpoint_from", + "thresholds": [ + { + "value": 1, + "status": "bad" + } + ], + "better": "lower" + }, + { + "id": "check_latency", + "title": "Check latency", + "description": "Latest check duration by endpoint and reporting host.name. Supplied by the Collector `httpcheck` receiver. [Watch hosts, containers and endpoints](https://fanout.run/start/send-telemetry#watch-hosts-containers-and-endpoints).", + "viz": "timeseries", + "width": 12, + "sql": "WITH points AS (\n SELECT *,to_json([namespace,attributes['http.url']::VARCHAR,attributes['http.method']::VARCHAR,resource['host.name']::VARCHAR])::VARCHAR AS entity,\n namespace||' / '||coalesce(attributes['http.method']::VARCHAR,'method unknown')||' '||coalesce(attributes['http.url']::VARCHAR,'endpoint unknown')||' from '||coalesce(resource['host.name']::VARCHAR,'vantage unknown') AS endpoint_from,\n value*CASE unit WHEN 's' THEN 1000 WHEN 'ms' THEN 1 WHEN 'us' THEN .001 WHEN 'µs' THEN .001 WHEN 'ns' THEN .000001 END AS duration_ms\n FROM metrics WHERE time \u003e= make_timestamp_ns(epoch_ns($__from::TIMESTAMP_NS)-300000000000)::TIMESTAMPTZ_NS AND time \u003c $__to::TIMESTAMP_NS::TIMESTAMPTZ_NS\n AND name='httpcheck.duration' AND type IN ('gauge','updown') AND isfinite(value)\n QUALIFY row_number() OVER (PARTITION BY series_id,time ORDER BY ingested_unix_nano DESC,value DESC,start_time DESC NULLS LAST)=1),\n inventory AS (SELECT DISTINCT series_id,endpoint_from FROM points WHERE $__window(time)),\n buckets AS (SELECT b::TIMESTAMPTZ_NS AS time,least((b+INTERVAL '1 minute')::TIMESTAMPTZ_NS,$__to::TIMESTAMP_NS::TIMESTAMPTZ_NS) AS at\n FROM generate_series(time_bucket(INTERVAL '1 minute',$__from::TIMESTAMP_NS),$__to::TIMESTAMP_NS,INTERVAL '1 minute') t(b) WHERE b::TIMESTAMPTZ_NS \u003c $__to::TIMESTAMP_NS::TIMESTAMPTZ_NS),\n grid AS (SELECT * FROM inventory CROSS JOIN buckets)\n SELECT g.time,g.endpoint_from,sum(CASE WHEN p.time\u003e=make_timestamp_ns(epoch_ns(g.at::TIMESTAMP_NS)-300000000000)::TIMESTAMPTZ_NS THEN p.duration_ms END)::DOUBLE AS latency\n FROM grid g ASOF LEFT JOIN points p ON g.series_id=p.series_id AND g.at\u003ep.time GROUP BY g.time,g.endpoint_from ORDER BY g.time DESC,g.endpoint_from", + "unit": "ms" + }, + { + "id": "service_health", + "title": "Is anything failing or slow?", + "description": "Service health and error trend with deploy markers. Supplied by the Collector `otlp` receiver. [Watch hosts, containers and endpoints](https://fanout.run/start/send-telemetry#watch-hosts-containers-and-endpoints).", + "viz": "health", + "width": 12, + "query": { + "from": "spans", + "where": [ + "service = $service" + ] + } + }, + { + "id": "service_map", + "title": "Services and external dependencies", + "description": "Failing or slow services and external dependencies observed in client spans. Supplied by the Collector `otlp` receiver. [Watch hosts, containers and endpoints](https://fanout.run/start/send-telemetry#watch-hosts-containers-and-endpoints).", + "viz": "service_map", + "width": 12, + "height": "l", + "query": { + "from": "spans", + "where": [ + "service = $service" + ] + } + }, + { + "id": "hosts", + "title": "Is anything running out? Hosts", + "description": "CPU busy, memory used and the fullest filesystem; worst first. Not reporting after two intervals is bad. Supplied by the Collector `hostmetrics` receiver. [Watch hosts, containers and endpoints](https://fanout.run/start/send-telemetry#watch-hosts-containers-and-endpoints).", + "viz": "table", + "width": 12, + "sql": "WITH raw AS (SELECT *,resource['host.name']::VARCHAR AS host,resource['container.id']::VARCHAR AS container_id,\n resource['container.name']::VARCHAR AS container_name,attributes['state']::VARCHAR AS state,\n attributes['mountpoint']::VARCHAR AS mountpoint,attributes['device']::VARCHAR AS device\n FROM metrics WHERE time \u003e= make_timestamp_ns(epoch_ns($__from::TIMESTAMP_NS)-600000000000)::TIMESTAMPTZ_NS AND time \u003c $__to::TIMESTAMP_NS::TIMESTAMPTZ_NS AND name IN ('system.cpu.time','system.memory.usage','system.filesystem.usage')),\n keyed AS (SELECT *,host AS entity,CASE name WHEN 'system.cpu.time' THEN 1 WHEN 'system.memory.usage' THEN 2 ELSE 3 END AS priority FROM raw WHERE host IS NOT NULL AND host\u003c\u003e''),\n inventory AS (SELECT namespace,entity,any_value(host) AS host,any_value(container_id) AS container_id,arg_max(container_name,time) AS container FROM keyed WHERE $__window(time) GROUP BY namespace,entity),\n witness_family AS (SELECT namespace,entity,min(priority) AS priority FROM keyed GROUP BY namespace,entity),\n witness AS (SELECT DISTINCT k.namespace,k.entity,k.time FROM keyed k JOIN witness_family f USING(namespace,entity,priority)),\n gaps AS (SELECT *,epoch_ns(time::TIMESTAMP_NS)::HUGEINT-epoch_ns(lag(time) OVER (PARTITION BY namespace,entity ORDER BY time)::TIMESTAMP_NS)::HUGEINT AS gap_ns FROM witness),\n cadence AS (SELECT namespace,entity,max(time) AS last_seen,arg_max(gap_ns,time) FILTER (WHERE gap_ns\u003e0) AS interval_ns FROM gaps GROUP BY namespace,entity),\n unique_points AS (SELECT * FROM keyed WHERE isfinite(value) QUALIFY row_number() OVER (PARTITION BY series_id,time ORDER BY ingested_unix_nano DESC,value DESC,start_time DESC NULLS LAST)=1),\n latest AS (SELECT * FROM unique_points WHERE time \u003e= make_timestamp_ns(epoch_ns($__to::TIMESTAMP_NS)-300000000000)::TIMESTAMPTZ_NS AND type IN ('gauge','updown') QUALIFY row_number() OVER (PARTITION BY series_id ORDER BY time DESC)=1),\n cpu_previous AS (SELECT *,lag(value) OVER w AS previous_value,lag(time) OVER w AS previous_time,lag(start_time) OVER w AS previous_start FROM unique_points WHERE name='system.cpu.time' AND type='counter' WINDOW w AS (PARTITION BY series_id ORDER BY time)),\n cpu_delta AS (SELECT *,CASE WHEN temporality='delta' OR value\u003cprevious_value OR (start_time IS NOT NULL AND previous_start IS NOT NULL AND start_time\u003c\u003eprevious_start) THEN value ELSE value-previous_value END AS increase,\n CASE WHEN temporality='delta' AND start_time IS NOT NULL AND start_time\u003ctime THEN start_time ELSE previous_time END AS elapsed_from FROM cpu_previous WHERE temporality IN ('delta','cumulative') AND (temporality='delta' OR previous_time IS NOT NULL)),\n cpu_overlap AS (SELECT *, (epoch_ns(time::TIMESTAMP_NS)::HUGEINT-epoch_ns(elapsed_from::TIMESTAMP_NS)::HUGEINT)::DOUBLE/1e9 AS elapsed_s,\n greatest(0,(epoch_ns(least(time,$__to::TIMESTAMP_NS::TIMESTAMPTZ_NS)::TIMESTAMP_NS)::HUGEINT-epoch_ns(greatest(elapsed_from,$__from::TIMESTAMP_NS::TIMESTAMPTZ_NS)::TIMESTAMP_NS)::HUGEINT)::DOUBLE/1e9) AS overlap_s FROM cpu_delta WHERE elapsed_from\u003ctime),\n cpu_intervals AS (SELECT *,CASE WHEN elapsed_s\u003c=600 THEN increase*overlap_s/elapsed_s WHEN $__window(time) THEN increase END AS rate_share,\n CASE WHEN elapsed_s\u003c=600 THEN overlap_s WHEN $__window(time) THEN elapsed_s END AS covered_s FROM cpu_overlap),\n cpu_series AS (SELECT namespace,entity,series_id,any_value(state) AS state,\n CASE WHEN max(time)\u003cmake_timestamp_ns(epoch_ns($__to::TIMESTAMP_NS)-300000000000)::TIMESTAMPTZ_NS THEN NULL\n ELSE coalesce(sum(rate_share) FILTER (WHERE covered_s\u003e0)/nullif(sum(covered_s) FILTER (WHERE covered_s\u003e0),0),arg_max(increase/elapsed_s,time)) END AS cpu_rate FROM cpu_intervals GROUP BY namespace,entity,series_id),\n cpu AS (SELECT namespace,entity,100.0*sum(cpu_rate) FILTER (WHERE state\u003c\u003e'idle')/nullif(sum(cpu_rate),0) AS cpu_percent FROM cpu_series GROUP BY namespace,entity),\n memory AS (SELECT namespace,entity,100.0*sum(value) FILTER (WHERE state='used')/nullif(sum(value),0) AS memory_percent FROM latest WHERE name='system.memory.usage' GROUP BY namespace,entity),\n mounts AS (SELECT namespace,entity,mountpoint,device,sum(value) FILTER (WHERE state IN ('used','reserved')) AS used_bytes,sum(value) AS total_bytes FROM latest WHERE name='system.filesystem.usage' GROUP BY namespace,entity,mountpoint,device),\n fullest AS (SELECT *,100.0*used_bytes/nullif(total_bytes,0) AS filesystem_percent FROM mounts QUALIFY row_number() OVER (PARTITION BY namespace,entity ORDER BY filesystem_percent DESC NULLS LAST,mountpoint,device)=1),\n results AS (SELECT i.namespace,i.host,CASE WHEN i.host IS NULL OR i.host='' THEN 'stable resource identity missing'\n WHEN c.interval_ns IS NULL THEN 'reporting interval unknown'\n WHEN epoch_ns($__to::TIMESTAMP_NS)::HUGEINT-epoch_ns(c.last_seen::TIMESTAMP_NS)::HUGEINT \u003e= 2*c.interval_ns THEN 'not reporting' ELSE 'reporting' END AS reporting,cpu.cpu_percent,m.memory_percent,f.filesystem_percent,f.mountpoint,f.used_bytes,f.total_bytes,c.last_seen FROM inventory i LEFT JOIN cadence c USING(namespace,entity) LEFT JOIN cpu USING(namespace,entity) LEFT JOIN memory m USING(namespace,entity) LEFT JOIN fullest f USING(namespace,entity))\n SELECT namespace,host,CASE WHEN reporting\u003c\u003e'reporting' THEN 100 ELSE greatest(cpu_percent,memory_percent,filesystem_percent) END::DOUBLE AS status,reporting,cpu_percent,memory_percent,filesystem_percent,mountpoint,used_bytes,total_bytes,last_seen,\n CASE WHEN memory_percent IS NULL OR filesystem_percent IS NULL OR cpu_percent IS NULL THEN 'measurement missing, unsupported, stale or denominator unavailable' END AS measurement\n FROM results ORDER BY (reporting\u003c\u003e'reporting') DESC,status DESC NULLS LAST,namespace,host", + "thresholds": [ + { + "value": 80, + "status": "warn" + }, + { + "value": 90, + "status": "bad" + } + ], + "better": "lower", + "options": { + "columns": [ + { + "field": "status", + "format": "status" + }, + { + "field": "cpu_percent", + "format": "unit", + "unit": "percent" + }, + { + "field": "memory_percent", + "format": "unit", + "unit": "percent" + }, + { + "field": "filesystem_percent", + "format": "unit", + "unit": "percent" + } + ] + } + }, + { + "id": "containers", + "title": "Containers", + "description": "CPU and memory usage against the container limit; worst first. Not reporting after two intervals is bad. Supplied by the Collector `docker_stats` receiver. [Watch hosts, containers and endpoints](https://fanout.run/start/send-telemetry#watch-hosts-containers-and-endpoints).", + "viz": "table", + "width": 12, + "sql": "WITH raw AS (SELECT *,resource['host.name']::VARCHAR AS host,resource['container.id']::VARCHAR AS container_id,\n resource['container.name']::VARCHAR AS container_name,attributes['state']::VARCHAR AS state,\n attributes['mountpoint']::VARCHAR AS mountpoint,attributes['device']::VARCHAR AS device\n FROM metrics WHERE time \u003e= make_timestamp_ns(epoch_ns($__from::TIMESTAMP_NS)-600000000000)::TIMESTAMPTZ_NS AND time \u003c $__to::TIMESTAMP_NS::TIMESTAMPTZ_NS AND name IN ('container.cpu.utilization','container.memory.usage.total','container.memory.usage.limit')),\n keyed AS (SELECT *,to_json([host,container_id])::VARCHAR AS entity,CASE name WHEN 'container.cpu.utilization' THEN 1 WHEN 'container.memory.usage.total' THEN 2 ELSE 3 END AS priority FROM raw WHERE host IS NOT NULL AND host\u003c\u003e'' AND container_id IS NOT NULL AND container_id\u003c\u003e''),\n inventory AS (SELECT namespace,entity,any_value(host) AS host,any_value(container_id) AS container_id,arg_max(container_name,time) AS container FROM keyed WHERE $__window(time) GROUP BY namespace,entity),\n witness_family AS (SELECT namespace,entity,min(priority) AS priority FROM keyed GROUP BY namespace,entity),\n witness AS (SELECT DISTINCT k.namespace,k.entity,k.time FROM keyed k JOIN witness_family f USING(namespace,entity,priority)),\n gaps AS (SELECT *,epoch_ns(time::TIMESTAMP_NS)::HUGEINT-epoch_ns(lag(time) OVER (PARTITION BY namespace,entity ORDER BY time)::TIMESTAMP_NS)::HUGEINT AS gap_ns FROM witness),\n cadence AS (SELECT namespace,entity,max(time) AS last_seen,arg_max(gap_ns,time) FILTER (WHERE gap_ns\u003e0) AS interval_ns FROM gaps GROUP BY namespace,entity),\n unique_points AS (SELECT * FROM keyed WHERE isfinite(value) QUALIFY row_number() OVER (PARTITION BY series_id,time ORDER BY ingested_unix_nano DESC,value DESC,start_time DESC NULLS LAST)=1),\n latest AS (SELECT * FROM unique_points WHERE time \u003e= make_timestamp_ns(epoch_ns($__to::TIMESTAMP_NS)-300000000000)::TIMESTAMPTZ_NS AND type IN ('gauge','updown') QUALIFY row_number() OVER (PARTITION BY series_id ORDER BY time DESC)=1), values_by_container AS (SELECT namespace,entity,\n sum(value*CASE unit WHEN '%' THEN 1 WHEN '1' THEN 100 END) FILTER (WHERE name='container.cpu.utilization') AS cpu_percent,\n sum(value) FILTER (WHERE name='container.memory.usage.total') AS total_bytes,sum(value) FILTER (WHERE name='container.memory.usage.limit') AS limit_bytes FROM latest GROUP BY namespace,entity),\n results AS (SELECT i.namespace,i.host,coalesce(i.container,i.container_id) AS container,CASE WHEN i.host IS NULL OR i.host='' OR i.container_id IS NULL OR i.container_id='' THEN 'stable resource identity missing'\n WHEN c.interval_ns IS NULL THEN 'reporting interval unknown'\n WHEN epoch_ns($__to::TIMESTAMP_NS)::HUGEINT-epoch_ns(c.last_seen::TIMESTAMP_NS)::HUGEINT \u003e= 2*c.interval_ns THEN 'not reporting' ELSE 'reporting' END AS reporting,v.cpu_percent,100.0*v.total_bytes/nullif(v.limit_bytes,0) AS memory_percent,v.total_bytes,v.limit_bytes,c.last_seen FROM inventory i LEFT JOIN cadence c USING(namespace,entity) LEFT JOIN values_by_container v USING(namespace,entity))\n SELECT namespace,host,container,CASE WHEN reporting\u003c\u003e'reporting' THEN 100 ELSE greatest(cpu_percent,memory_percent) END::DOUBLE AS status,reporting,cpu_percent,memory_percent,total_bytes,limit_bytes,last_seen,\n CASE WHEN limit_bytes IS NULL OR limit_bytes=0 THEN 'memory limit not reported' WHEN cpu_percent IS NULL OR memory_percent IS NULL THEN 'measurement missing, unsupported or stale' END AS measurement\n FROM results ORDER BY (reporting\u003c\u003e'reporting') DESC,status DESC NULLS LAST,namespace,host,container", + "thresholds": [ + { + "value": 80, + "status": "warn" + }, + { + "value": 90, + "status": "bad" + } + ], + "better": "lower", + "options": { + "columns": [ + { + "field": "status", + "format": "status" + }, + { + "field": "cpu_percent", + "format": "unit", + "unit": "percent" + }, + { + "field": "memory_percent", + "format": "unit", + "unit": "percent" + } + ] + } + }, + { + "id": "versions", + "title": "What changed?", + "description": "Service versions, first seen in the selected range, runtime and SDK. Supplied by the Collector `otlp` receiver. [Watch hosts, containers and endpoints](https://fanout.run/start/send-telemetry#watch-hosts-containers-and-endpoints).", + "viz": "table", + "width": 12, + "sql": "WITH observations AS (SELECT namespace,service,resource['service.version']::VARCHAR AS version,resource['process.runtime.name']::VARCHAR AS runtime_name,resource['process.runtime.version']::VARCHAR AS runtime_version,resource['telemetry.sdk.language']::VARCHAR AS sdk_language,resource['telemetry.sdk.version']::VARCHAR AS sdk_version,start_time AS seen FROM spans WHERE $__window(start_time)\n UNION ALL SELECT namespace,service,resource['service.version']::VARCHAR,resource['process.runtime.name']::VARCHAR,resource['process.runtime.version']::VARCHAR,resource['telemetry.sdk.language']::VARCHAR,resource['telemetry.sdk.version']::VARCHAR,time FROM logs WHERE $__window(time)\n UNION ALL SELECT namespace,service,resource['service.version']::VARCHAR,resource['process.runtime.name']::VARCHAR,resource['process.runtime.version']::VARCHAR,resource['telemetry.sdk.language']::VARCHAR,resource['telemetry.sdk.version']::VARCHAR,time FROM metrics WHERE $__window(time)) SELECT namespace,service,version,min(seen) AS first_seen,runtime_name,runtime_version,sdk_language,sdk_version FROM observations WHERE version IS NOT NULL AND version\u003c\u003e'' GROUP BY namespace,service,version,runtime_name,runtime_version,sdk_language,sdk_version ORDER BY first_seen DESC,namespace,service,version" + } + ] +} diff --git a/internal/bench/types.go b/internal/bench/types.go index 02e5679..7aad1ed 100644 --- a/internal/bench/types.go +++ b/internal/bench/types.go @@ -17,6 +17,8 @@ var Signals = []Signal{Traces, Logs, Metrics} type Config struct { Host string + HostDescription string + SSHKnownHosts string SSHIdentity string SSHPort int KeepRemote bool @@ -74,6 +76,7 @@ func (p PhaseSpec) OfferedPerSecond() float64 { } type Host struct { + Description string `json:"description,omitempty"` OS string `json:"os"` Architecture string `json:"architecture"` CPUModel string `json:"cpu_model,omitempty"` @@ -177,6 +180,7 @@ type Trial struct { ProcessRestarted bool `json:"fanout_restarted"` Resources Resources `json:"resources"` QueryLatency Latency `json:"query_latency_ms,omitempty"` + QueryKindCompleted map[string]int64 `json:"query_completed_by_kind,omitempty"` QueryAttempts int64 `json:"query_attempts,omitempty"` QueryFailures int64 `json:"query_failures,omitempty"` QueryShed int64 `json:"query_shed,omitempty"` diff --git a/justfile b/justfile index 59ab8a1..c4ac578 100644 --- a/justfile +++ b/justfile @@ -8,6 +8,10 @@ build: mkdir -p bin go build -trimpath -ldflags "-s -w -X main.version=$(git describe --always --dirty 2>/dev/null || echo dev)" -o bin/fanout-bench ./cmd/fanout-bench +[positional-arguments] +hetzner *args: + scripts/hetzner.sh "$@" + telemetrygen: mkdir -p bin GOBIN="$(pwd)/bin" go install github.com/open-telemetry/opentelemetry-collector-contrib/cmd/telemetrygen@{{telemetrygen_version}} diff --git a/scripts/hetzner.sh b/scripts/hetzner.sh new file mode 100755 index 0000000..9cbbc56 --- /dev/null +++ b/scripts/hetzner.sh @@ -0,0 +1,180 @@ +#!/usr/bin/env bash +set -euo pipefail + +if [[ $# -eq 0 ]]; then + echo "Usage: scripts/hetzner.sh [-- ]…" >&2 + echo "Each nonempty group runs in order on one VM; SSH port and identity must agree across groups." >&2 + exit 2 +fi + +# Validate every group before building or provisioning. Keep the original "$@" +# intact so slicing it below preserves argument boundaries and quoting. +group_lengths=() +group_args=() +group_length=0 +group_count=0 +ssh_port=22 +ssh_identity= +validate_group() { + local group_ssh_port=22 group_ssh_identity= + if [[ $group_length -eq 0 ]]; then + echo "Run groups must not be empty" >&2 + exit 2 + fi + while [[ $# -gt 0 ]]; do + case "$1" in + --ssh-identity|--ssh-port|-ssh-identity|-ssh-port) + if [[ $# -lt 2 ]]; then + echo "Missing value for $1" >&2 + exit 2 + fi + if [[ $1 == *ssh-identity ]]; then + group_ssh_identity=$2 + else + group_ssh_port=$2 + fi + shift + ;; + --ssh-identity=*|-ssh-identity=*) group_ssh_identity=${1#*=} ;; + --ssh-port=*|-ssh-port=*) group_ssh_port=${1#*=} ;; + esac + shift + done + if [[ $group_count -eq 0 ]]; then + ssh_port=$group_ssh_port + ssh_identity=$group_ssh_identity + elif [[ $group_ssh_port != "$ssh_port" || $group_ssh_identity != "$ssh_identity" ]]; then + echo "SSH port and identity must agree across all run groups" >&2 + exit 2 + fi + group_lengths+=("$group_length") + group_count=$((group_count + 1)) +} +for arg in "$@"; do + if [[ $arg == -- ]]; then + # The explicit check also avoids expanding an empty array under Bash 3. + if [[ $group_length -eq 0 ]]; then + echo "Run groups must not be empty" >&2 + exit 2 + fi + validate_group "${group_args[@]}" + group_args=() + group_length=0 + else + group_args+=("$arg") + group_length=$((group_length + 1)) + fi +done +if [[ $group_length -eq 0 ]]; then + echo "Run groups must not be empty" >&2 + exit 2 +fi +validate_group "${group_args[@]}" + +: "${FANOUT_BENCH_HCLOUD_SSH_KEY:?Set FANOUT_BENCH_HCLOUD_SSH_KEY to a registered Hetzner SSH key name (or comma-separated names)}" +server_type=${FANOUT_BENCH_HCLOUD_TYPE:-ccx33} +location=${FANOUT_BENCH_HCLOUD_LOCATION:-fsn1} +image=${FANOUT_BENCH_HCLOUD_IMAGE:-ubuntu-24.04} + +for command in hcloud ssh go just od; do + if ! command -v "$command" >/dev/null 2>&1; then + echo "Required command not found: $command" >&2 + exit 1 + fi +done + +repo_dir=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd) +cd "$repo_dir" +just build + +key_args=() +IFS=',' read -r -a keys <<< "$FANOUT_BENCH_HCLOUD_SSH_KEY" +for key in "${keys[@]}"; do + if [[ -z "$key" ]]; then + echo "Hetzner SSH key names must not be empty" >&2 + exit 2 + fi + key_args+=(--ssh-key "$key") +done + +run_dir=$(mktemp -d "${TMPDIR:-/tmp}/fanout-bench-hetzner.XXXXXX") +known_hosts="$run_dir/known_hosts" +touch "$known_hosts" +nonce=$(od -An -N4 -tx1 /dev/urandom | tr -d ' \n') +server_name="fanout-bench-$(date -u +%Y%m%dT%H%M%SZ)-$nonce" +# Invoked indirectly by the EXIT trap. +# shellcheck disable=SC2329 +cleanup() { + local status=$? attempt deleted=false + trap - EXIT + trap '' INT TERM + for attempt in 1 2 3; do + if hcloud server delete "$server_name" >/dev/null; then + deleted=true + break + fi + if [[ $attempt -lt 3 ]]; then + sleep 2 + fi + done + if [[ $deleted == false ]]; then + echo "Failed to delete Hetzner server $server_name after three attempts; delete it manually." >&2 + if [[ $status -eq 0 ]]; then + status=1 + fi + fi + rm -rf -- "$run_dir" + exit "$status" +} +# Arm cleanup before creation so an interrupted or failed create is covered too. +trap cleanup EXIT +trap 'exit 130' INT +trap 'exit 143' TERM +hcloud server create --name "$server_name" --type "$server_type" \ + --location "$location" --image "$image" --label purpose=fanout-bench \ + "${key_args[@]}" >/dev/null +ip=$(hcloud server ip "$server_name") +if [[ -z "$ip" ]]; then + echo "Hetzner server $server_name has no IPv4 address" >&2 + exit 1 +fi + +ssh_args=(-o BatchMode=yes -o StrictHostKeyChecking=accept-new + -o "UserKnownHostsFile=$known_hosts" -o ConnectTimeout=5 + -o ConnectionAttempts=1 -o ServerAliveInterval=5 -o ServerAliveCountMax=1 + -p "$ssh_port") +if [[ -n "$ssh_identity" ]]; then + ssh_args+=(-i "$ssh_identity") +fi +deadline=$((SECONDS + 300)) +until ssh "${ssh_args[@]}" "root@$ip" true 2>/dev/null; do + if (( SECONDS >= deadline )); then + echo "Timed out waiting for SSH on Hetzner server $server_name" >&2 + exit 1 + fi + sleep 2 +done + +statuses=() +status=0 +offset=1 +for group_length in "${group_lengths[@]}"; do + if bin/fanout-bench run "${@:offset:group_length}" \ + --host "root@$ip" --ssh-known-hosts "$known_hosts" \ + --host-description "Hetzner Cloud $server_type, $location, $image"; then + statuses+=(0) + else + statuses+=("$?") + status=1 + fi + offset=$((offset + group_length + 1)) +done + +for index in "${!statuses[@]}"; do + if [[ ${statuses[index]} -eq 0 ]]; then + printf 'Group %d/%d: PASS\n' "$((index + 1))" "${#statuses[@]}" + else + printf 'Group %d/%d: FAIL (exit %d)\n' "$((index + 1))" "${#statuses[@]}" "${statuses[index]}" + fi +done +exit "$status" From 5ccdce7555443dfdf43231d059f72e74ac0d3716 Mon Sep 17 00:00:00 2001 From: Vishal Rana Date: Wed, 14 Oct 2026 17:38:30 -0700 Subject: [PATCH 2/4] fix: delete the Hetzner VM promptly when the script is stopped Bash defers a trap until the foreground child exits, so TERM on the script left the benchmark running and the VM billing until it finished. Run each group as a background child and wait for it; INT or TERM now forwards the signal, waits for the child, skips the remaining groups, and the existing cleanup deletes the VM. --- scripts/hetzner.sh | 22 ++++++++++++++++++---- 1 file changed, 18 insertions(+), 4 deletions(-) diff --git a/scripts/hetzner.sh b/scripts/hetzner.sh index 9cbbc56..8e1f4dc 100755 --- a/scripts/hetzner.sh +++ b/scripts/hetzner.sh @@ -127,9 +127,20 @@ cleanup() { exit "$status" } # Arm cleanup before creation so an interrupted or failed create is covered too. +benchmark_pid= +# Invoked indirectly by the signal traps. +# shellcheck disable=SC2329 +interrupt() { + trap '' INT TERM + if [[ -n $benchmark_pid ]]; then + kill -s "$1" "$benchmark_pid" 2>/dev/null || true + wait "$benchmark_pid" || true + fi + exit "$2" +} trap cleanup EXIT -trap 'exit 130' INT -trap 'exit 143' TERM +trap 'interrupt INT 130' INT +trap 'interrupt TERM 143' TERM hcloud server create --name "$server_name" --type "$server_type" \ --location "$location" --image "$image" --label purpose=fanout-bench \ "${key_args[@]}" >/dev/null @@ -159,14 +170,17 @@ statuses=() status=0 offset=1 for group_length in "${group_lengths[@]}"; do - if bin/fanout-bench run "${@:offset:group_length}" \ + bin/fanout-bench run "${@:offset:group_length}" \ --host "root@$ip" --ssh-known-hosts "$known_hosts" \ - --host-description "Hetzner Cloud $server_type, $location, $image"; then + --host-description "Hetzner Cloud $server_type, $location, $image" <&0 & + benchmark_pid=$! + if wait "$benchmark_pid"; then statuses+=(0) else statuses+=("$?") status=1 fi + benchmark_pid= offset=$((offset + group_length + 1)) done From ed7386f319f9b4419ced35425e6cecdbf46b936f Mon Sep 17 00:00:00 2001 From: Vishal Rana Date: Wed, 14 Oct 2026 21:37:10 -0700 Subject: [PATCH 3/4] feat: run generators on a separate host so the Fanout VM only ingests Provisioned runs placed telemetrygen on the Fanout VM, so the generators took a large share of its CPUs and the result was a same-host ceiling. scripts/hetzner.sh now creates a private network with a Fanout-only VM (ccx33) and a generator VM (ccx43). The managed runner on the Fanout host binds Fanout to its private address and launches every telemetrygen process on the generator host over SSH, with a run-scoped key, keepalives, a client deadline and a remote timeout. A per-run sshd drop-in lifts MaxStartups for the largest phase's simultaneous launches. Reports record both hosts and the generator placement, and redact every address in the reproduction command. Cleanup removes the key, the drop-in, both VMs and the network, including after an interrupted create. Plain --host runs stay same-host. --- README.md | 41 ++++++--- cmd/fanout-bench/main.go | 15 ++++ cmd/fanout-bench/main_test.go | 90 ++++++++++++++++++- docs/publishing.md | 16 ++-- internal/bench/managed.go | 102 ++++++++++++++++------ internal/bench/managed_test.go | 42 +++++++++ internal/bench/remote.go | 152 ++++++++++++++++++++++++++++++--- internal/bench/remote_test.go | 77 +++++++++++++++++ internal/bench/report.go | 8 +- internal/bench/report_test.go | 64 ++++++++++++++ internal/bench/runner.go | 39 ++++++++- internal/bench/runner_test.go | 75 ++++++++++++++++ internal/bench/types.go | 91 +++++++++++--------- scripts/hetzner.sh | 144 ++++++++++++++++++++++++------- 14 files changed, 824 insertions(+), 132 deletions(-) diff --git a/README.md b/README.md index 633d7c3..8eeb72b 100644 --- a/README.md +++ b/README.md @@ -22,20 +22,20 @@ With pinned telemetrygen v0.157, each generated trace exports one parent and one ## Quick start -Prerequisites: [hcloud](https://github.com/hetznercloud/cli) with an active context (or `HCLOUD_TOKEN`), a registered Hetzner SSH key available to your SSH client, Go 1.27+, [just](https://github.com/casey/just), and SSH/SCP. +Prerequisites: [hcloud](https://github.com/hetznercloud/cli) with an active context (or `HCLOUD_TOKEN`), a registered Hetzner SSH key available to your SSH client, Go 1.27+, `ssh-keygen`, [just](https://github.com/casey/just), and SSH/SCP. ```sh export FANOUT_BENCH_HCLOUD_SSH_KEY='benchmark-key' just hetzner --fanout-version v2026.10.1-alpha --preset quick ``` -The script runs `just build` before provisioning, creates a fresh Hetzner Cloud VM with a random name suffix, waits for SSH, runs the harness for each group, downloads the evidence, and attempts deletion on every exit, including failures and Ctrl-C. If deletion fails after three attempts, it prints the server name for manual deletion and exits nonzero. Hetzner bills every started hour, so grouping runs in one invocation shares that VM time. +The script runs `just build` before provisioning, creates a private network and two fresh Hetzner Cloud VMs with unique names, waits for SSH on both, runs the harness for each group, and downloads the evidence. Cleanup attempts to delete both servers and then the network on every exit, including failures and Ctrl-C. After three failed attempts it names each resource requiring manual deletion and exits nonzero. Hetzner bills every started hour for each VM, so grouping runs shares the billing time of both machines. -The defaults are `ccx33` (8 dedicated vCPUs, 32 GB RAM), `fsn1`, and `ubuntu-24.04`. Override them with `FANOUT_BENCH_HCLOUD_TYPE`, `FANOUT_BENCH_HCLOUD_LOCATION`, and `FANOUT_BENCH_HCLOUD_IMAGE`. `FANOUT_BENCH_HCLOUD_SSH_KEY` also accepts comma-separated key names. Each invocation shares one temporary known-hosts file across its groups, leaving your SSH known-hosts file untouched. Pass `--ssh-identity ~/.ssh/benchmark` if the registered key is not available through your SSH agent. +The defaults are a `ccx33` for Fanout (8 dedicated vCPUs, 32 GB RAM) and a separate `ccx43` for generators (16 dedicated vCPUs, 64 GB RAM), both in `fsn1` on `ubuntu-24.04`. OTLP travels over a private network (`10.77.0.0/16`, cloud subnet `10.77.0.0/24`) in that location’s network zone. Override the VM types with `FANOUT_BENCH_HCLOUD_TYPE` and `FANOUT_BENCH_HCLOUD_GENERATOR_TYPE`, and location/image with `FANOUT_BENCH_HCLOUD_LOCATION` and `FANOUT_BENCH_HCLOUD_IMAGE`. The hourly VM cost is **ccx33 hourly rate + ccx43 hourly rate**, plus applicable public IPv4 charges and tax; budget for both machines for every started hour. Check the selected location’s current rates before provisioning. `FANOUT_BENCH_HCLOUD_SSH_KEY` also accepts comma-separated key names. Each invocation shares one temporary known-hosts file across its groups, leaving your SSH known-hosts file untouched. Pass `--ssh-identity ~/.ssh/benchmark` if the registered key is not available through your SSH agent. -Separate complete sets of `fanout-bench run` flags with a standalone `--`. Groups must be nonempty and use matching SSH port and identity settings (defaults: port 22 and the SSH client's default identity). All groups run in order even if one fails, with a one-line summary per group at the end and a nonzero exit if any failed. They share one VM and host description; each harness run creates a fresh remote directory and Fanout data directory. The VM is deleted after the last group. +Separate complete sets of `fanout-bench run` flags with a standalone `--`. Groups must be nonempty and use matching SSH port and identity settings (defaults: port 22 and the SSH client's default identity). All groups run in order even if one fails, with a one-line summary per group at the end and a nonzero exit if any failed. They share both VMs, the private network, and both host descriptions; each harness run creates fresh stages and a fresh Fanout data directory. Both VMs and the network are deleted after the last group. -Any clean SSH-accessible Linux host is also supported: +A plain `--host` run on any clean SSH-accessible Linux host runs Fanout and telemetrygen together on that host: ```sh just build @@ -58,9 +58,26 @@ Replace `` with the address of your host. The remote harness: 8. downloads `report.json`, `report.md`, and `samples.jsonl` locally; 9. stops Fanout and removes the remote run directory. -For an existing host, pass `--ssh-identity ~/.ssh/benchmark`, `--ssh-port 2222`, or `--ssh-known-hosts ` when needed. `--host-description ` records the provider or machine configuration beside the detected CPU model. Reports redact the SSH address and omit the run-scoped known-hosts path from reproduction commands. `--keep-remote` preserves the isolated directory on an existing host for investigation; the Hetzner script always deletes its VM. +For an existing host, pass `--ssh-identity ~/.ssh/benchmark`, `--ssh-port 2222`, or `--ssh-known-hosts ` when needed. `--host-description ` records the provider or machine configuration beside the detected CPU model. Reports redact the SSH address and omit the run-scoped known-hosts path from reproduction commands. `--keep-remote` preserves the Fanout directory on an existing host for investigation; split runs still remove the generator stage and ephemeral SSH key. The Hetzner script always deletes both VMs and the network. -Nothing is installed system-wide. Existing Fanout services and data directories are not touched. The remote host needs no Go toolchain, Docker daemon, or preconfigured Fanout account. +To use two existing clean Linux hosts, add all three split flags together: + +```sh +./bin/fanout-bench run \ + --host 'root@' \ + --generator-host 'root@' \ + --generator-address 10.77.0.3 \ + --fanout-listen 10.77.0.2 \ + --host-description 'Fanout host' \ + --generator-host-description 'Generator host' \ + --fanout-version v2026.10.1-alpha --preset quick +``` + +The public SSH targets are used by your local machine for provisioning. The two IP flags must be reachable between the hosts, with SSH from Fanout to the generator and OTLP from the generator to Fanout. Provisioning detects both Linux architectures, builds the harness for Fanout and telemetrygen for the generator, and uploads each binary only where it runs. A local temporary ed25519 key grants the Fanout host access to the generator for that run; its public key has a unique authorized-keys marker, and the private key is uploaded with mode `0600` into the Fanout stage. Both stages use mode `0700`. + +`managed` remains the sole orchestrator on the Fanout host: it binds Fanout to the supplied private IP on a free port, performs setup, samples metrics, issues read load, verifies storage, and writes the report. Every telemetrygen process runs through SSH on the generator host, bounded by a remote timeout. Cleanup removes the marked authorized-key line, the private key, and both stages. Reports record both host descriptions and `generator_on_target_host: false`; reproduction commands redact both public targets and omit the private IP flags. + +Nothing is installed system-wide. Existing Fanout services and data directories are not touched. Neither remote host needs a Go toolchain or Docker daemon, and Fanout needs no preconfigured account. The Fanout host needs an SSH client; the generator host needs an SSH server and GNU `timeout` (included in the default Ubuntu image). > `telemetrygen` currently accepts OTLP headers through a command-line flag and prints those headers in its startup configuration. Fanout Bench suppresses and redacts child output, but the short-lived ingest token can still be visible to local process-listing tools while a generator is running. Use a dedicated token and revoke it after the run. @@ -74,7 +91,7 @@ Nothing is installed system-wide. Existing Fanout services and data directories The default topology is 8 workers per process at 2,500 items/s per worker with 1,000-item OTLP batches. A mixed step launches the same topology independently for traces, logs, and metrics. Without read load, the sustained proof reruns the mixed topology with the highest **accepted** throughput. With `--query-rate`, it chooses the highest-throughput topology that also completed at least 95% of the requested reads with no errors or scheduler shedding. If no sweep point qualifies, it repeats the best read-completion point and fails the sustained-read quality gate rather than publishing a write-only peak as mixed capacity. -For publication runs, use the default ccx33 and three repetitions. Run ingest-only and read-under-write as two groups in one invocation on the same fresh VM; see [the publication commands](docs/publishing.md#publication-runs). For example: +For publication runs, dedicate the default ccx33 to Fanout and use the separate ccx43 for generators, with three repetitions. Run ingest-only and read-under-write as two groups sharing the same fresh pair of VMs; see [the publication commands](docs/publishing.md#publication-runs). For example: ```sh scripts/hetzner.sh \ @@ -87,7 +104,7 @@ scripts/hetzner.sh \ --output results/publish-read-write-2026-10-14 ``` -Provisioned runs deliberately place Fanout and generators on the same remote host, measuring the complete single-node product under load. The report labels this an observed same-host ceiling and warns that generator contention can make it lower than Fanout's isolated backend ceiling. +Publication runs measure Fanout’s ingest ceiling with all eight ccx33 vCPUs available to Fanout; telemetrygen runs on the separate ccx43 over the private network. A plain `--host` run without the three split flags remains a same-host run, and its report warns about generator contention. ## Read and write together @@ -123,8 +140,8 @@ The terminal output is optimized for decisions: ╭────────────────────────────────────────────────────────────────────╮ │ FANOUT BENCH • accepted-row performance, with evidence │ ╰────────────────────────────────────────────────────────────────────╯ - Target localhost:7520 (plaintext gRPC) - Generator telemetrygen v0.157.0 same-host=true + Target 10.77.0.2:39281 (plaintext gRPC) + Generator telemetrygen v0.157.0 same-host=false Suite standard repeat=1 18 phases ~5.2 min load time Evidence Fanout counters @ /metrics samples every 1.0s @@ -220,6 +237,6 @@ export FANOUT_BENCH_METRICS_TOKEN='matching-Fanout-metrics-token' --same-host ``` -Managed runs set `FANOUT_ADDR` to one loopback random port for OTLP/gRPC, OTLP/HTTP, the API, and `/metrics`. Metrics stay private: a per-run `FANOUT_METRICS_TOKEN` is sent as `Authorization: Bearer …` and never written to artifacts. MCP, alerts, and chat stay disabled. +Managed runs set `FANOUT_ADDR` to one random port shared by OTLP/gRPC, OTLP/HTTP, the API, and `/metrics`. Split runs bind the private listen IP; same-host runs bind loopback. Metrics stay private: a per-run `FANOUT_METRICS_TOKEN` is sent as `Authorization: Bearer …` and never written to artifacts. MCP, alerts, and chat stay disabled. Private local metrics endpoints use `FANOUT_BENCH_METRICS_TOKEN`. Local authenticated read load uses `FANOUT_BENCH_QUERY_COOKIE`. Secrets have no value flags and are never persisted. telemetrygen does expose its OTLP header in its process arguments while running, so use only a dedicated short-lived token and revoke it afterward. Managed remote runs create and destroy this token automatically. diff --git a/cmd/fanout-bench/main.go b/cmd/fanout-bench/main.go index 75f95a7..e33e977 100644 --- a/cmd/fanout-bench/main.go +++ b/cmd/fanout-bench/main.go @@ -67,6 +67,12 @@ func commonFlags(name string, args []string) (bench.Config, error) { fs.StringVar(&cfg.SSHIdentity, "ssh-identity", "", "SSH private key path") fs.StringVar(&cfg.SSHKnownHosts, "ssh-known-hosts", "", "run-scoped SSH known-hosts file") fs.StringVar(&cfg.HostDescription, "host-description", "", "host description recorded alongside hardware evidence") + if name == "run" { + fs.StringVar(&cfg.GeneratorHost, "generator-host", "", "SSH target to provision as a separate generator host") + fs.StringVar(&cfg.GeneratorAddress, "generator-address", "", "generator host IP reachable from the Fanout host") + fs.StringVar(&cfg.FanoutListen, "fanout-listen", "", "Fanout host IP reachable from the generator host") + fs.StringVar(&cfg.GeneratorHostDescription, "generator-host-description", "", "generator host description recorded in the report") + } fs.IntVar(&cfg.SSHPort, "ssh-port", 22, "SSH port") fs.BoolVar(&cfg.KeepRemote, "keep-remote", false, "preserve the isolated remote run directory after downloading artifacts") fs.StringVar(&cfg.Preset, "preset", "standard", "suite: quick, standard, or publish") @@ -98,6 +104,9 @@ func commonFlags(name string, args []string) (bench.Config, error) { if err := fs.Parse(args); err != nil { return cfg, err } + if err := bench.ValidateGeneratorHosts(cfg); err != nil { + return cfg, err + } cfg.Token = os.Getenv(tokenEnv) cfg.MetricsToken = os.Getenv(metricsTokenEnv) cfg.QueryCookie = os.Getenv(queryCookieEnv) @@ -145,6 +154,11 @@ func managed(ctx context.Context, args []string) error { var cfg bench.ManagedConfig fs.StringVar(&cfg.FanoutPath, "fanout", "./fanout", "provisioned Fanout binary") fs.StringVar(&cfg.TelemetrygenPath, "telemetrygen", "./telemetrygen", "provisioned telemetrygen binary") + fs.StringVar(&cfg.Listen, "listen", "127.0.0.1", "Fanout listen IP for OTLP, metrics, and queries") + fs.StringVar(&cfg.GeneratorSSH, "generator-ssh", "", "SSH target for telemetrygen launches") + fs.IntVar(&cfg.GeneratorSSHPort, "generator-ssh-port", 22, "SSH port for telemetrygen launches") + fs.StringVar(&cfg.GeneratorSSHIdentity, "generator-ssh-identity", "", "ephemeral SSH private key for the generator host") + fs.StringVar(&cfg.GeneratorTelemetrygen, "generator-telemetrygen", "", "telemetrygen path on the generator host") fs.StringVar(&cfg.OutputDir, "output", "./results", "remote evidence directory") fs.StringVar(&cfg.FanoutVersion, "fanout-version", "unknown", "Fanout version") fs.StringVar(&cfg.FanoutArtifact, "fanout-artifact", "", "verified release artifact identity") @@ -158,6 +172,7 @@ func managed(ctx context.Context, args []string) error { fs.StringVar(&cfg.TelemetrygenVersion, "telemetrygen-version", "v0.157.0", "telemetrygen version") fs.StringVar(&cfg.ProvisionedHost, "provisioned-host", "", "SSH target used to construct the redacted reproduction command") fs.StringVar(&cfg.HostDescription, "host-description", "", "host description recorded alongside hardware evidence") + fs.StringVar(&cfg.GeneratorHostDescription, "generator-host-description", "", "generator host description recorded in the report") fs.IntVar(&cfg.ProvisionedSSHPort, "provisioned-ssh-port", 22, "SSH port recorded in remote evidence") fs.StringVar(&cfg.ProvisionedSSHIdentity, "provisioned-ssh-identity", "", "SSH identity path recorded in remote evidence") fs.IntVar(&cfg.ProcessOverride, "processes", 0, "override process count per signal") diff --git a/cmd/fanout-bench/main_test.go b/cmd/fanout-bench/main_test.go index 3ff122d..ede4b02 100644 --- a/cmd/fanout-bench/main_test.go +++ b/cmd/fanout-bench/main_test.go @@ -1,6 +1,14 @@ package main -import "testing" +import ( + "context" + "os" + "os/exec" + "strings" + "testing" + + "github.com/labstack/fanout-bench/internal/bench" +) func TestLocalDefaultsUseSharedFanoutListener(t *testing.T) { cfg, err := commonFlags("test", nil) @@ -12,6 +20,47 @@ func TestLocalDefaultsUseSharedFanoutListener(t *testing.T) { } } +func TestSplitReproductionCommandPassesValidationAfterSubstitution(t *testing.T) { + executable, err := os.Executable() + if err != nil { + t.Fatal(err) + } + cfg, err := commonFlags("run", []string{ + "--host", "root@192.0.2.10", "--generator-host", "root@192.0.2.11", + "--generator-address", "10.77.0.3", "--fanout-listen", "10.77.0.2", + "--ssh-port", "2222", "--telemetrygen", executable, "--preset", "quick", + }) + if err != nil { + t.Fatal(err) + } + cfg.Token = "test-token" + ctx, cancel := context.WithCancel(context.Background()) + cancel() + runner := bench.Runner{Config: cfg} + report, err := runner.Run(ctx) + if err == nil || report == nil { + t.Fatalf("expected partial report on cancellation, got report=%v err=%v", report, err) + } + command := strings.NewReplacer( + "", "192.0.2.20", "", "192.0.2.21", + "", "10.77.0.13", "", "10.77.0.12", + ).Replace(report.ReproductionCommand) + // Parse shell quoting without executing the reproduction command. + output, err := exec.Command("bash", "-c", "set -- "+command+"; printf '%s\\0' \"$@\"").Output() + if err != nil { + t.Fatal(err) + } + args := strings.Split(strings.TrimSuffix(string(output), "\x00"), "\x00") + reproduced, err := commonFlags("run", args[2:]) + if err != nil { + t.Fatalf("substituted reproduction failed validation: %v\n%s", err, command) + } + if reproduced.Host != "root@192.0.2.20" || reproduced.GeneratorHost != "root@192.0.2.21" || + reproduced.GeneratorAddress != "10.77.0.13" || reproduced.FanoutListen != "10.77.0.12" || reproduced.SSHPort != 2222 { + t.Fatalf("reproduction lost topology: %+v", reproduced) + } +} + func TestHostFlags(t *testing.T) { cfg, err := commonFlags("test", []string{"--ssh-known-hosts", "/tmp/run scope/known_hosts", "--host-description", "Hetzner Cloud ccx33, fsn1, ubuntu-24.04"}) if err != nil { @@ -21,3 +70,42 @@ func TestHostFlags(t *testing.T) { t.Fatalf("known-hosts=%q description=%q", cfg.SSHKnownHosts, cfg.HostDescription) } } + +func TestSplitFlagsRequiredTogetherAndRemoteOnly(t *testing.T) { + flags := [][]string{{"--generator-host", "root@generator.example"}, {"--generator-address", "10.77.0.3"}, {"--fanout-listen", "10.77.0.2"}} + for mask := 0; mask < 8; mask++ { + for _, remote := range []bool{false, true} { + var args []string + if remote { + args = append(args, "--host", "root@fanout.example") + } + for i, pair := range flags { + if mask&(1< On a Hetzner Cloud ccx33 (8 dedicated vCPUs, 32 GB RAM) with generators sharing the host, Fanout accepted X mixed OpenTelemetry items/s for five minutes. The composition was A spans, B log records, and C metric points; dashboard read p95 was Y ms. This is an observed same-host result, not an isolated backend ceiling. +> On a Hetzner Cloud ccx33 (8 dedicated vCPUs, 32 GB RAM) with telemetrygen on a separate ccx43 (16 dedicated vCPUs, 64 GB RAM) in the same location over a private network, Fanout accepted X mixed OpenTelemetry items/s for five minutes. The composition was A spans, B log records, and C metric points; dashboard read p95 was Y ms. The ccx33 ran Fanout and the benchmark orchestrator; all generator processes ran on the ccx43. Avoid: @@ -42,7 +42,7 @@ The second sentence discards the machine, duration, signal mix, generator placem ## Publication runs -Every publication run uses one invocation of `scripts/hetzner.sh` with the default Hetzner Cloud `ccx33` (8 dedicated vCPUs, 32 GB RAM), `fsn1`, and `ubuntu-24.04`. Use the publish preset with three repetitions in each group. Separate the ingest-only and read-under-write groups with a standalone `--` so both share one fresh VM while each gets a fresh remote directory and Fanout data directory: +Every publication run uses one invocation of `scripts/hetzner.sh` with Fanout alone on the default Hetzner Cloud `ccx33` (8 dedicated vCPUs, 32 GB RAM) and generators on a separate `ccx43` (16 dedicated vCPUs, 64 GB RAM), both in `fsn1` on `ubuntu-24.04`. The two VMs share a private network (`10.77.0.0/16`, cloud subnet `10.77.0.0/24`) in the location’s network zone. Public IPs serve provisioning; generators send OTLP to Fanout over private IPs. `FANOUT_BENCH_HCLOUD_TYPE` and `FANOUT_BENCH_HCLOUD_GENERATOR_TYPE` override the Fanout and generator types respectively; location, image, and SSH-key settings apply to both. Use the publish preset with three repetitions in each group. Separate the ingest-only and read-under-write groups with a standalone `--` so both groups share the fresh pair of VMs and network while each gets fresh stages and a Fanout data directory: ```sh export FANOUT_BENCH_HCLOUD_SSH_KEY='benchmark-key' @@ -56,6 +56,10 @@ scripts/hetzner.sh \ --output results/publish-read-write-2026-10-14 ``` -The script uses the active `hcloud` context or `HCLOUD_TOKEN`, records the VM type, location, and image alongside the detected hardware, rebuilds the harness before provisioning, and waits for SSH once. Groups must be nonempty and use matching SSH port and identity settings; they share a temporary known-hosts file and host description. Every group runs even if an earlier one fails, downloads its own evidence, and receives a one-line summary at the end; any failed group makes the invocation exit nonzero. The script attempts VM deletion after the last group and on early exits, including Ctrl-C. Failed deletion after three attempts prints the unique server name for manual cleanup and exits nonzero. Hetzner bills every started hour, so both groups share the same VM billing time. +The script uses the active `hcloud` context or `HCLOUD_TOKEN`, records both VM types, location, and image alongside the detected Fanout hardware, rebuilds the harness before provisioning, and waits for SSH on both VMs. Groups must be nonempty and use matching SSH port and identity settings; they share a temporary known-hosts file and both host descriptions. Every group runs even if an earlier one fails, downloads its own evidence, and receives a one-line summary at the end; any failed group makes the invocation exit nonzero. The script attempts to delete both servers and then the network after the last group and on early exits, including Ctrl-C. Failed deletion after three attempts prints each resource name for manual cleanup and exits nonzero. Hetzner bills every started hour on both VMs. The hourly VM cost is **ccx33 hourly rate + ccx43 hourly rate** for the chosen location, plus applicable public IPv4 charges and tax; a run crossing an hour boundary incurs another started hour on each VM. Confirm current rates before provisioning; grouping the runs shares the billing time of both machines. -The harness verifies `fanout_v2026.10.1-alpha_linux_amd64.tar.gz` against `SHA256SUMS` on the ccx33. Managed scrapes use private `/metrics` bearer authentication. Read load covers System overview dashboard refresh (batched panels followed by annotations), service-map exemplar drill-down, and trace detail over 15-minute through 24-hour windows. Publish the per-operation methodology, counts and latency, and dashboard spec hash alongside the requested and completed operation rates. +The local provisioner detects both Linux platforms, builds the harness for Fanout and telemetrygen for the generator, and uploads only the binaries needed by each host. It generates an ephemeral ed25519 key, appends the public key to the generator’s root authorized keys with a unique marker, and puts the private key (`0600`) in the Fanout stage (`0700`). `managed` on Fanout owns setup, lifecycle, sampling, reads, storage verification, and reporting. Only telemetrygen launches move to the generator, through private-network SSH with a bounded remote timeout. Cleanup removes the marked key and both stages. Reports record the generator description in the host section and `generator_on_target_host: false`; reproduction commands redact both public targets and omit the private addresses. + +A plain `fanout-bench run --host root@` without a generator host remains a same-host run on any clean Linux host. It is useful for diagnostics, but generator CPU contention limits what it says about Fanout’s own ingest ceiling. + +Before uploading to the ccx33, the local provisioner verifies `fanout_v2026.10.1-alpha_linux_amd64.tar.gz` against the release `SHA256SUMS`. Managed scrapes use private `/metrics` bearer authentication. Read load covers System overview dashboard refresh (batched panels followed by annotations), service-map exemplar drill-down, and trace detail over 15-minute through 24-hour windows. Publish the per-operation methodology, counts and latency, and dashboard spec hash alongside the requested and completed operation rates. diff --git a/internal/bench/managed.go b/internal/bench/managed.go index 08aaf3a..15d3c43 100644 --- a/internal/bench/managed.go +++ b/internal/bench/managed.go @@ -25,32 +25,47 @@ import ( ) type ManagedConfig struct { - HostDescription string - FanoutPath string - TelemetrygenPath string - OutputDir string - FanoutVersion string - FanoutArtifact string - Preset string - Repeat int - BatchSize int - QueryRate float64 - SampleInterval time.Duration - Cooldown time.Duration - HarnessVersion string - TelemetrygenVersion string - ProvisionedHost string - ProvisionedSSHPort int - ProvisionedSSHIdentity string - ProcessOverride int - WorkerOverride int - RateOverride float64 - DurationOverride time.Duration + HostDescription string + GeneratorHostDescription string + Listen string + GeneratorSSH string + GeneratorSSHPort int + GeneratorSSHIdentity string + GeneratorTelemetrygen string + FanoutPath string + TelemetrygenPath string + OutputDir string + FanoutVersion string + FanoutArtifact string + Preset string + Repeat int + BatchSize int + QueryRate float64 + SampleInterval time.Duration + Cooldown time.Duration + HarnessVersion string + TelemetrygenVersion string + ProvisionedHost string + ProvisionedSSHPort int + ProvisionedSSHIdentity string + ProcessOverride int + WorkerOverride int + RateOverride float64 + DurationOverride time.Duration } var setupTokenPattern = regexp.MustCompile(`setup_token=([^\s]+)`) func RunManaged(ctx context.Context, managed ManagedConfig, out io.Writer) error { + if managed.GeneratorSSH != "" || managed.GeneratorSSHIdentity != "" || managed.GeneratorTelemetrygen != "" { + if managed.GeneratorSSH == "" || managed.GeneratorSSHIdentity == "" || managed.GeneratorTelemetrygen == "" { + return errors.New("--generator-ssh, --generator-ssh-identity, and --generator-telemetrygen are required together") + } + if !sshTargetPattern.MatchString(managed.GeneratorSSH) || strings.HasPrefix(managed.GeneratorSSH, "-") { + return errors.New("invalid generator SSH target") + } + managed.TelemetrygenPath = managed.GeneratorTelemetrygen + } if managed.FanoutPath == "" || managed.TelemetrygenPath == "" || managed.OutputDir == "" { return errors.New("managed paths are required") } @@ -62,7 +77,7 @@ func RunManaged(ctx context.Context, managed ManagedConfig, out io.Writer) error if err := os.MkdirAll(dataDir, 0o700); err != nil { return err } - httpPort, err := freePort() + httpAddr, err := managedListenAddress(managed.Listen) if err != nil { return err } @@ -74,7 +89,6 @@ func RunManaged(ctx context.Context, managed ManagedConfig, out io.Writer) error if err != nil { return err } - httpAddr := fmt.Sprintf("127.0.0.1:%d", httpPort) grpcAddr := httpAddr metricsURL := "http://" + httpAddr + "/metrics" baseURL := "http://" + httpAddr @@ -171,7 +185,9 @@ func RunManaged(ctx context.Context, managed ManagedConfig, out io.Writer) error return err } - cfg := Config{Host: managed.ProvisionedHost, HostDescription: managed.HostDescription, SSHPort: managed.ProvisionedSSHPort, SSHIdentity: managed.ProvisionedSSHIdentity, Preset: managed.Preset, Endpoint: grpcAddr, MetricsURL: metricsURL, MetricsToken: metricsToken, Insecure: true, Telemetrygen: managed.TelemetrygenPath, Token: ingestToken, OutputDir: managed.OutputDir, FanoutVersion: actualVersion, FanoutArtifact: managed.FanoutArtifact, DataDir: dataDir, SampleInterval: managed.SampleInterval, GeneratorTimeout: 30 * time.Second, Cooldown: managed.Cooldown, Repeat: managed.Repeat, QueryURL: baseURL, QueryCookie: queryCookie, QueryRate: managed.QueryRate, SameHost: true, BatchSize: managed.BatchSize, TelemetrygenVersion: managed.TelemetrygenVersion, ProcessOverride: managed.ProcessOverride, WorkerOverride: managed.WorkerOverride, RateOverride: managed.RateOverride, DurationOverride: managed.DurationOverride} + cfg := managedRunnerConfig(managed, httpAddr, dataDir) + cfg.FanoutVersion = actualVersion + cfg.Token, cfg.MetricsToken, cfg.QueryCookie = ingestToken, metricsToken, queryCookie runner := Runner{Config: cfg, Version: managed.HarnessVersion, Out: out, DeferSummary: true} report, runErr := runner.Run(ctx) stopFanout() @@ -226,6 +242,30 @@ func RunManaged(ctx context.Context, managed ManagedConfig, out io.Writer) error return nil } +func managedRunnerConfig(managed ManagedConfig, httpAddr, dataDir string) Config { + cfg := Config{ + Host: managed.ProvisionedHost, HostDescription: managed.HostDescription, + SSHPort: managed.ProvisionedSSHPort, SSHIdentity: managed.ProvisionedSSHIdentity, + Preset: managed.Preset, Repeat: managed.Repeat, BatchSize: managed.BatchSize, + Endpoint: httpAddr, MetricsURL: "http://" + httpAddr + "/metrics", QueryURL: "http://" + httpAddr, + Insecure: true, Telemetrygen: managed.TelemetrygenPath, OutputDir: managed.OutputDir, + FanoutVersion: managed.FanoutVersion, FanoutArtifact: managed.FanoutArtifact, DataDir: dataDir, + SampleInterval: managed.SampleInterval, GeneratorTimeout: 30 * time.Second, Cooldown: managed.Cooldown, + QueryRate: managed.QueryRate, SameHost: managed.GeneratorSSH == "", TelemetrygenVersion: managed.TelemetrygenVersion, + ProcessOverride: managed.ProcessOverride, WorkerOverride: managed.WorkerOverride, + RateOverride: managed.RateOverride, DurationOverride: managed.DurationOverride, + } + cfg.GeneratorSSH = managed.GeneratorSSH + cfg.GeneratorSSHPort = managed.GeneratorSSHPort + cfg.GeneratorSSHIdentity = managed.GeneratorSSHIdentity + cfg.GeneratorSSHKnownHosts = filepath.Join(filepath.Dir(dataDir), "generator_known_hosts") + cfg.GeneratorHostDescription = managed.GeneratorHostDescription + if managed.GeneratorSSH != "" { + cfg.GeneratorHost = "root@" + } + return cfg +} + func completeSetup(ctx context.Context, baseURL, setupToken string) (string, string, error) { jar, _ := cookiejar.New(nil) client := &http.Client{Timeout: 10 * time.Second, Jar: jar} @@ -284,13 +324,19 @@ func completeSetup(ctx context.Context, baseURL, setupToken string) (string, str } } -func freePort() (int, error) { - listener, err := net.Listen("tcp", "127.0.0.1:0") +func managedListenAddress(ip string) (string, error) { + if ip == "" { + ip = "127.0.0.1" + } + if parsed := net.ParseIP(ip); parsed == nil || parsed.IsUnspecified() { + return "", fmt.Errorf("--listen requires a concrete IP address, got %q", ip) + } + listener, err := net.Listen("tcp", net.JoinHostPort(ip, "0")) if err != nil { - return 0, err + return "", err } defer listener.Close() - return listener.Addr().(*net.TCPAddr).Port, nil + return listener.Addr().String(), nil } func randomHex(bytesCount int) (string, error) { raw := make([]byte, bytesCount) diff --git a/internal/bench/managed_test.go b/internal/bench/managed_test.go index 283ea6a..9e8c001 100644 --- a/internal/bench/managed_test.go +++ b/internal/bench/managed_test.go @@ -4,13 +4,55 @@ import ( "context" "encoding/json" "fmt" + "net" "net/http" "net/http/httptest" "os" + "path/filepath" "strings" "testing" ) +func TestManagedListenDerivesAllEndpointsAndPlacement(t *testing.T) { + for _, ip := range []string{"", "127.0.0.1"} { + addr, err := managedListenAddress(ip) + if err != nil { + t.Fatal(err) + } + host, port, err := net.SplitHostPort(addr) + if err != nil || host != "127.0.0.1" || port == "0" || port == "" { + t.Fatalf("address=%q, err=%v", addr, err) + } + } + for _, ip := range []string{"hostname", "10.77.0.2:7520", "0.0.0.0", "::"} { + if _, err := managedListenAddress(ip); err == nil { + t.Errorf("accepted invalid listen IP %q", ip) + } + } + // These addresses need not belong to the test machine: endpoint derivation + // is independent of port reservation and works for private IPv4 and IPv6. + for _, addr := range []string{"10.77.0.2:12345", "[fd77::2]:12345"} { + for _, split := range []bool{false, true} { + managed := ManagedConfig{TelemetrygenPath: "/generator/telemetrygen", HostDescription: "Fanout ccx33", GeneratorHostDescription: "Generator ccx43"} + if split { + managed.GeneratorSSH = "root@10.77.0.3" + managed.GeneratorSSHPort = 2222 + managed.GeneratorSSHIdentity = "/stage/generator_key" + } + cfg := managedRunnerConfig(managed, addr, "/stage/data") + if cfg.Endpoint != addr || cfg.MetricsURL != "http://"+addr+"/metrics" || cfg.QueryURL != "http://"+addr { + t.Fatalf("endpoints=%+v", cfg) + } + if cfg.SameHost == split || cfg.GeneratorSSH != managed.GeneratorSSH || cfg.GeneratorSSHPort != managed.GeneratorSSHPort || cfg.GeneratorSSHIdentity != managed.GeneratorSSHIdentity || cfg.GeneratorSSHKnownHosts != filepath.Join("/stage", "generator_known_hosts") { + t.Fatalf("generator placement=%+v", cfg) + } + if cfg.GeneratorHostDescription != managed.GeneratorHostDescription || cfg.HostDescription != managed.HostDescription { + t.Fatal("managed runner lost host descriptions") + } + } + } +} + func TestFanoutEnvironmentUsesCurrentSchema(t *testing.T) { // Schema env tags captured from internal/config/config.go at 870c90db. schema, err := os.ReadFile("testdata/fanout-v2026.10.1-alpha-env.txt") diff --git a/internal/bench/remote.go b/internal/bench/remote.go index ee6df06..16462c4 100644 --- a/internal/bench/remote.go +++ b/internal/bench/remote.go @@ -9,6 +9,7 @@ import ( "errors" "fmt" "io" + "net" "net/http" "os" "os/exec" @@ -22,7 +23,33 @@ import ( var sshTargetPattern = regexp.MustCompile(`^(?:[A-Za-z0-9._-]+@)?(?:[A-Za-z0-9._-]+|\[[0-9A-Fa-f:]+\])$`) var fanoutVersionPattern = regexp.MustCompile(`^v[0-9]{4}\.(?:[1-9]|1[0-2])\.(?:0|[1-9][0-9]*)(?:-(?:alpha|beta|rc))?$`) +// ValidateGeneratorHosts also runs before local execution, so partial split +// configurations cannot silently fall back to same-host load. +func ValidateGeneratorHosts(cfg Config) error { + if cfg.GeneratorHost == "" && cfg.GeneratorAddress == "" && cfg.FanoutListen == "" && cfg.GeneratorHostDescription == "" { + return nil + } + if cfg.Host == "" { + return errors.New("generator host flags require a remote run with --host") + } + if cfg.GeneratorHost == "" || cfg.GeneratorAddress == "" || cfg.FanoutListen == "" { + return errors.New("--generator-host, --generator-address, and --fanout-listen are required together") + } + if !sshTargetPattern.MatchString(cfg.GeneratorHost) || strings.HasPrefix(cfg.GeneratorHost, "-") { + return fmt.Errorf("unsafe generator SSH target %q", cfg.GeneratorHost) + } + for _, address := range []string{cfg.GeneratorAddress, cfg.FanoutListen} { + if ip := net.ParseIP(address); ip == nil || ip.IsUnspecified() { + return fmt.Errorf("invalid host IP %q", address) + } + } + return nil +} + func RunRemote(ctx context.Context, cfg Config, version string, out io.Writer) error { + if err := ValidateGeneratorHosts(cfg); err != nil { + return err + } if !sshTargetPattern.MatchString(cfg.Host) || strings.HasPrefix(cfg.Host, "-") { return fmt.Errorf("unsafe SSH target %q", cfg.Host) } @@ -47,6 +74,11 @@ func RunRemote(ctx context.Context, cfg Config, version string, out io.Writer) e if err := requireCommand("go"); err != nil { return err } + if cfg.GeneratorHost != "" { + if err := requireCommand("ssh-keygen"); err != nil { + return err + } + } fmt.Fprintf(out, "Fanout Bench remote provision\n host %s\n release %s\n preset %s × %d\n", cfg.Host, cfg.FanoutVersion, cfg.Preset, cfg.Repeat) remoteOS, remoteArch, err := detectRemote(ctx, cfg) @@ -57,6 +89,19 @@ func RunRemote(ctx context.Context, cfg Config, version string, out io.Writer) e if remoteOS != "linux" { return fmt.Errorf("remote OS %q is not supported; remote benchmark hosts must run Linux", remoteOS) } + generator := cfg + generator.Host = cfg.GeneratorHost + generatorOS, generatorArch := remoteOS, remoteArch + if cfg.GeneratorHost != "" { + generatorOS, generatorArch, err = detectRemote(ctx, generator) + if err != nil { + return err + } + if generatorOS != "linux" { + return fmt.Errorf("generator OS %q is not supported; generator hosts must run Linux", generatorOS) + } + fmt.Fprintf(out, " generator %s (%s/%s)\n", generator.Host, generatorOS, generatorArch) + } buildDir, err := os.MkdirTemp("", "fanout-bench-provision-*") if err != nil { @@ -69,7 +114,7 @@ func RunRemote(ctx context.Context, cfg Config, version string, out io.Writer) e if err != nil { return fmt.Errorf("build remote harness: %w", err) } - telemetrygenPath, err := buildTelemetrygen(ctx, buildDir, remoteOS, remoteArch, cfg.TelemetrygenVersion) + telemetrygenPath, err := buildTelemetrygen(ctx, buildDir, generatorOS, generatorArch, cfg.TelemetrygenVersion) if err != nil { return fmt.Errorf("build telemetrygen: %w", err) } @@ -79,24 +124,71 @@ func RunRemote(ctx context.Context, cfg Config, version string, out io.Writer) e return fmt.Errorf("fetch Fanout release: %w", err) } - suffix, _ := randomHex(4) - stage := "/tmp/fanout-bench-" + time.Now().UTC().Format("20060102T150405Z") + "-" + suffix - if err := sshRun(ctx, cfg, out, "install -d -m 0700 "+shellQuote(stage)); err != nil { - return fmt.Errorf("create remote stage: %w", err) + suffix, err := randomHex(16) + if err != nil { + return err } + stage := "/tmp/fanout-bench-" + time.Now().UTC().Format("20060102T150405Z") + "-" + suffix + generatorStage := stage + "-generator" + marker := "fanout-bench-" + suffix cleanup := func() { + if cfg.GeneratorHost != "" { + cleanupCtx, cancel := context.WithTimeout(context.Background(), 30*time.Second) + if err := sshRun(cleanupCtx, generator, io.Discard, "status=0; "+removeGeneratorKeyCommand(marker)+" || status=$?; rm -rf -- "+shellQuote(generatorStage)+" || status=$?; exit \"$status\""); err != nil { + fmt.Fprintf(out, "cleanup generator %s (key %s, stage %s): %v\n", generator.Host, marker, generatorStage, err) + } + cancel() + } + command := "rm -rf -- " + shellQuote(stage) if cfg.KeepRemote { fmt.Fprintf(out, "\n remote run preserved: %s:%s\n", cfg.Host, stage) - return + if cfg.GeneratorHost == "" { + return + } + // Even a preserved evidence stage must not retain a private key. + command = "rm -f -- " + shellQuote(stage+"/generator_key") } + // Each host gets its own deadline: an unreachable generator must not + // prevent removal of the Fanout stage and its private key. cleanupCtx, cancel := context.WithTimeout(context.Background(), 30*time.Second) defer cancel() - _ = sshRun(cleanupCtx, cfg, io.Discard, "rm -rf -- "+shellQuote(stage)) + if err := sshRun(cleanupCtx, cfg, io.Discard, command); err != nil { + fmt.Fprintf(out, "cleanup Fanout stage %s:%s: %v\n", cfg.Host, stage, err) + } } defer cleanup() + if err := sshRun(ctx, cfg, out, "install -d -m 0700 "+shellQuote(stage)); err != nil { + return fmt.Errorf("create remote stage: %w", err) + } - fmt.Fprintln(out, " ◌ uploading Fanout, telemetrygen, and the benchmark agent") - if err := scpUpload(ctx, cfg, stage, harnessPath, telemetrygenPath, fanoutPath); err != nil { + fmt.Fprintln(out, " ◌ uploading architecture-matched binaries") + paths := []string{harnessPath, fanoutPath} + if cfg.GeneratorHost == "" { + paths = append(paths, telemetrygenPath) + } else { + if err := sshRun(ctx, generator, out, "install -d -m 0700 "+shellQuote(generatorStage)); err != nil { + return fmt.Errorf("create generator stage: %w", err) + } + if err := scpUpload(ctx, generator, generatorStage, telemetrygenPath); err != nil { + return err + } + if err := sshRun(ctx, generator, out, "chmod 0700 "+shellQuote(generatorStage+"/telemetrygen")); err != nil { + return err + } + keyPath := filepath.Join(buildDir, "generator_key") + if data, err := exec.CommandContext(ctx, "ssh-keygen", "-q", "-t", "ed25519", "-N", "", "-C", marker, "-f", keyPath).CombinedOutput(); err != nil { + return fmt.Errorf("create generator identity: %w: %s", err, data) + } + publicKey, err := os.ReadFile(keyPath + ".pub") + if err != nil { + return err + } + if err := sshRun(ctx, generator, out, authorizeGeneratorKeyCommand(string(publicKey), marker)); err != nil { + return fmt.Errorf("authorize generator identity: %w", err) + } + paths = append(paths, keyPath) + } + if err := scpUpload(ctx, cfg, stage, paths...); err != nil { return err } remoteCommand := managedRemoteCommand(stage, cfg, version, artifact) @@ -355,16 +447,54 @@ func extractBinary(archivePath, destination, name string) error { } func managedRemoteCommand(stage string, cfg Config, version, artifact string) string { - args := []string{stage + "/fanout-bench", "managed", "--fanout", stage + "/fanout", "--telemetrygen", stage + "/telemetrygen", "--output", stage + "/results", "--fanout-version", cfg.FanoutVersion, "--fanout-artifact", artifact, "--preset", cfg.Preset, "--repeat", fmt.Sprint(cfg.Repeat), "--batch-size", fmt.Sprint(cfg.BatchSize), "--query-rate", fmt.Sprint(cfg.QueryRate), "--sample-interval", cfg.SampleInterval.String(), "--cooldown", cfg.Cooldown.String(), "--harness-version", version} + args := []string{stage + "/fanout-bench", "managed", "--fanout", stage + "/fanout", "--output", stage + "/results", "--fanout-version", cfg.FanoutVersion, "--fanout-artifact", artifact, "--preset", cfg.Preset, "--repeat", fmt.Sprint(cfg.Repeat), "--batch-size", fmt.Sprint(cfg.BatchSize), "--query-rate", fmt.Sprint(cfg.QueryRate), "--sample-interval", cfg.SampleInterval.String(), "--cooldown", cfg.Cooldown.String(), "--harness-version", version} args = append(args, "--telemetrygen-version", cfg.TelemetrygenVersion) args = append(args, "--provisioned-host", redactedHost(cfg.Host), "--host-description", cfg.HostDescription) args = append(args, "--provisioned-ssh-port", fmt.Sprint(cfg.SSHPort), "--provisioned-ssh-identity", cfg.SSHIdentity) args = append(args, "--processes", fmt.Sprint(cfg.ProcessOverride), "--workers", fmt.Sprint(cfg.WorkerOverride), "--rate", fmt.Sprint(cfg.RateOverride), "--duration", cfg.DurationOverride.String()) + binaries := []string{stage + "/fanout-bench", stage + "/fanout"} + prefix := "" + if cfg.GeneratorHost == "" { + binaries = append(binaries, stage+"/telemetrygen") + args = append(args, "--telemetrygen", stage+"/telemetrygen") + } else { + address := cfg.GeneratorAddress + if strings.Contains(address, ":") { + address = "[" + address + "]" + } + args = append(args, "--listen", cfg.FanoutListen, "--generator-ssh", "root@"+address, + "--generator-ssh-port", fmt.Sprint(cfg.SSHPort), + "--generator-ssh-identity", stage+"/generator_key", "--generator-telemetrygen", stage+"-generator/telemetrygen", + "--generator-host-description", cfg.GeneratorHostDescription) + prefix = "chmod 0600 " + shellQuote(stage+"/generator_key") + " && " + } quoted := make([]string, len(args)) for i, arg := range args { quoted[i] = shellQuote(arg) } - return "chmod 0700 " + shellQuote(stage+"/fanout-bench") + " " + shellQuote(stage+"/telemetrygen") + " " + shellQuote(stage+"/fanout") + " && " + strings.Join(quoted, " ") + for i := range binaries { + binaries[i] = shellQuote(binaries[i]) + } + return prefix + "chmod 0700 " + strings.Join(binaries, " ") + " && " + strings.Join(quoted, " ") +} + +// generatorSSHDConfig lifts sshd's default MaxStartups (10:30:100) for the run: +// the largest phase opens one connection per generator process at once, and +// the default randomly refuses connections beyond ten unauthenticated ones. +func generatorSSHDConfig(marker string) string { + return "/etc/ssh/sshd_config.d/" + marker + ".conf" +} + +const reloadSSHD = "(systemctl reload ssh 2>/dev/null || systemctl reload sshd)" + +func authorizeGeneratorKeyCommand(publicKey, marker string) string { + return "install -d -m 0700 /root/.ssh && touch /root/.ssh/authorized_keys && chmod 0600 /root/.ssh/authorized_keys && printf '\\n%s\\n' " + shellQuote(strings.TrimSpace(publicKey)) + " >> /root/.ssh/authorized_keys" + + " && printf 'MaxStartups 256\\nMaxSessions 256\\n' > " + shellQuote(generatorSSHDConfig(marker)) + " && " + reloadSSHD +} + +func removeGeneratorKeyCommand(marker string) string { + return "if [ -f /root/.ssh/authorized_keys ]; then sed -i " + shellQuote("/ "+marker+"$/d") + " /root/.ssh/authorized_keys; fi" + + "; if [ -f " + shellQuote(generatorSSHDConfig(marker)) + " ]; then rm -f " + shellQuote(generatorSSHDConfig(marker)) + " && " + reloadSSHD + "; fi" } func sshArgs(cfg Config) []string { diff --git a/internal/bench/remote_test.go b/internal/bench/remote_test.go index 6440b6c..b3c1d86 100644 --- a/internal/bench/remote_test.go +++ b/internal/bench/remote_test.go @@ -136,6 +136,83 @@ func TestManagedRemoteCommandQuotesEveryValue(t *testing.T) { } } +func TestSplitManagedRemoteCommand(t *testing.T) { + cfg := Config{Host: "root@192.0.2.10", SSHPort: 2222, GeneratorHost: "root@192.0.2.11", GeneratorAddress: "10.77.0.3", FanoutListen: "10.77.0.2", GeneratorHostDescription: "Generator's ccx43"} + got := managedRemoteCommand("/tmp/run", cfg, "dev", "verified") + for _, want := range []string{ + "chmod 0600 '/tmp/run/generator_key'", + "'--listen' '10.77.0.2'", "'--generator-ssh' 'root@10.77.0.3'", + "'--generator-ssh-port' '2222'", + "'--generator-ssh-identity' '/tmp/run/generator_key'", + "'--generator-telemetrygen' '/tmp/run-generator/telemetrygen'", + "'--generator-host-description' 'Generator'\\''s ccx43'", + } { + if !strings.Contains(got, want) { + t.Errorf("missing %q in %s", want, got) + } + } + for _, unwanted := range []string{"192.0.2.10", "192.0.2.11", "'/tmp/run/telemetrygen'", "'--telemetrygen'"} { + if strings.Contains(got, unwanted) { + t.Errorf("unexpected %q in %s", unwanted, got) + } + } + cfg.GeneratorAddress = "fd77::3" + if got := managedRemoteCommand("/tmp/run", cfg, "dev", "verified"); !strings.Contains(got, "'root@[fd77::3]'") { + t.Fatalf("IPv6 target missing brackets: %s", got) + } +} + +func TestGeneratorSSHDConfigIsScopedToRun(t *testing.T) { + for _, marker := range []string{"fanout-bench-first", "fanout-bench-second"} { + path := shellQuote("/etc/ssh/sshd_config.d/" + marker + ".conf") + key := "ssh-ed25519 AAAAtest " + marker + install := authorizeGeneratorKeyCommand(key+"\n", marker) + remove := removeGeneratorKeyCommand(marker) + if !strings.Contains(install, shellQuote(key)+" >> /root/.ssh/authorized_keys") || + !strings.Contains(install, " > "+path+" && "+reloadSSHD) { + t.Fatalf("key and run-specific sshd config not installed together: %s", install) + } + if !strings.Contains(remove, shellQuote("/ "+marker+"$/d")) || + !strings.Contains(remove, "rm -f "+path+" && "+reloadSSHD) { + t.Fatalf("cleanup missing run-specific key/config removal and reload: %s", remove) + } + for _, command := range []string{install, remove} { + // Every reference in the drop-in directory must name this run's file. + withoutPath := strings.ReplaceAll(command, path, "") + if strings.Contains(withoutPath, "/etc/ssh/sshd_config.d/") { + t.Fatalf("command touches another sshd drop-in: %s", command) + } + } + } +} + +func TestGeneratorHostValidation(t *testing.T) { + valid := Config{Host: "root@fanout", GeneratorHost: "root@generator", GeneratorAddress: "10.77.0.3", FanoutListen: "10.77.0.2"} + if err := ValidateGeneratorHosts(valid); err != nil { + t.Fatal(err) + } + for _, value := range []string{"hostname", "0.0.0.0", "::", "10.77.0.2:7520"} { + for _, field := range []string{"generator", "fanout"} { + cfg := valid + if field == "generator" { + cfg.GeneratorAddress = value + } else { + cfg.FanoutListen = value + } + if err := ValidateGeneratorHosts(cfg); err == nil { + t.Errorf("accepted %s IP %q", field, value) + } + } + } + for _, value := range []string{"-oProxyCommand=bad", "root@host;bad", "root@host name"} { + cfg := valid + cfg.GeneratorHost = value + if err := ValidateGeneratorHosts(cfg); err == nil { + t.Errorf("accepted unsafe generator SSH target %q", value) + } + } +} + func TestExtractBinarySelectsOnlyNamedRegularFile(t *testing.T) { dir := t.TempDir() archivePath := filepath.Join(dir, "release.tar.gz") diff --git a/internal/bench/report.go b/internal/bench/report.go index ea5a975..6dc6d11 100644 --- a/internal/bench/report.go +++ b/internal/bench/report.go @@ -150,7 +150,7 @@ func Markdown(r Report) string { if r.Host.Description != "" { machine += " · " + r.Host.Description } - settings := [][2]string{{"Fanout version", display(r.Build.FanoutVersion)}, {"Fanout commit", display(r.Build.FanoutCommit)}, {"Fanout artifact", display(r.Build.FanoutArtifact)}, {"Harness", r.Build.HarnessVersion + " (`" + r.Build.HarnessCommit + "`)"}, {"telemetrygen", display(r.Build.TelemetrygenVersion)}, {"Machine", fmt.Sprintf("%s/%s · %s · %d logical CPUs · %s RAM", r.Host.OS, r.Host.Architecture, machine, r.Host.LogicalCPUs, formatBytes(float64(r.Host.MemoryBytes)))}, {"Generator placement", map[bool]string{true: "same host as Fanout", false: "separate/remote"}[r.Host.GeneratorLocal]}, {"Preset", r.Settings.Preset}, {"Transport", r.Settings.Transport}, {"Batch size", fmt.Sprintf("%d", r.Settings.BatchSize)}, {"Trace shape", "1 parent + 1 child span per telemetrygen trace tick"}, {"Repeat", fmt.Sprintf("%d", r.Settings.Repeat)}, {"Sample interval", fmt.Sprintf("%.1fs", r.Settings.SampleSeconds)}, {"Workload SHA-256", "`" + r.WorkloadSHA256 + "`"}} + settings := [][2]string{{"Fanout version", display(r.Build.FanoutVersion)}, {"Fanout commit", display(r.Build.FanoutCommit)}, {"Fanout artifact", display(r.Build.FanoutArtifact)}, {"Harness", r.Build.HarnessVersion + " (`" + r.Build.HarnessCommit + "`)"}, {"telemetrygen", display(r.Build.TelemetrygenVersion)}, {"Machine", fmt.Sprintf("%s/%s · %s · %d logical CPUs · %s RAM", r.Host.OS, r.Host.Architecture, machine, r.Host.LogicalCPUs, formatBytes(float64(r.Host.MemoryBytes)))}, {"Generator host", display(r.Host.GeneratorDescription)}, {"Generator placement", map[bool]string{true: "same host as Fanout", false: "separate/remote"}[r.Host.GeneratorLocal]}, {"Preset", r.Settings.Preset}, {"Transport", r.Settings.Transport}, {"Batch size", fmt.Sprintf("%d", r.Settings.BatchSize)}, {"Trace shape", "1 parent + 1 child span per telemetrygen trace tick"}, {"Repeat", fmt.Sprintf("%d", r.Settings.Repeat)}, {"Sample interval", fmt.Sprintf("%.1fs", r.Settings.SampleSeconds)}, {"Workload SHA-256", "`" + r.WorkloadSHA256 + "`"}} for _, row := range settings { fmt.Fprintf(&b, "| %s | %s |\n", row[0], row[1]) } @@ -186,6 +186,9 @@ func reproductionCommand(c Config) string { if c.HostDescription != "" { command += " --host-description " + shellQuote(c.HostDescription) } + if c.GeneratorHostDescription != "" { + command += " --generator-host-description " + shellQuote(c.GeneratorHostDescription) + } return command } @@ -199,6 +202,9 @@ func redactedHost(host string) string { func reproductionRunCommand(c Config) string { if c.Host != "" { parts := []string{"fanout-bench run", "--host " + shellQuote(redactedHost(c.Host)), "--fanout-version " + shellQuote(c.FanoutVersion), "--preset " + c.Preset, "--repeat " + fmt.Sprint(c.Repeat), "--telemetrygen-version " + shellQuote(c.TelemetrygenVersion), "--batch-size " + fmt.Sprint(c.BatchSize), "--sample-interval " + c.SampleInterval.String(), "--cooldown " + c.Cooldown.String()} + if c.GeneratorHost != "" { + parts = append(parts, "--generator-host 'root@'", "--generator-address ''", "--fanout-listen ''") + } if c.SSHPort != 0 && c.SSHPort != 22 { parts = append(parts, "--ssh-port "+fmt.Sprint(c.SSHPort)) } diff --git a/internal/bench/report_test.go b/internal/bench/report_test.go index f0f3707..22da3fa 100644 --- a/internal/bench/report_test.go +++ b/internal/bench/report_test.go @@ -160,3 +160,67 @@ func TestHostDescriptionRecordedAndRendered(t *testing.T) { t.Fatal("host description missing beside CPU model") } } + +func TestSplitHostPlacementAndDescriptionRecordedAndRendered(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + cancel() + managed := ManagedConfig{ProvisionedHost: "root@", HostDescription: "Hetzner Cloud ccx33", GeneratorHostDescription: "Hetzner Cloud ccx43", GeneratorSSH: "root@10.77.0.3", GeneratorSSHIdentity: "/stage/key", TelemetrygenPath: "/remote/only/telemetrygen", Preset: "quick", Repeat: 1, BatchSize: 1000, SampleInterval: time.Second, TelemetrygenVersion: "v0.157.0"} + cfg := managedRunnerConfig(managed, "10.77.0.2:12345", "/stage/data") + cfg.Token = "test-token" + runner := Runner{Config: cfg} + report, err := runner.Run(ctx) + if err == nil || report == nil { + t.Fatalf("expected a partial report, got report=%v err=%v", report, err) + } + dir := t.TempDir() + if err := WriteArtifacts(dir, *report); err != nil { + t.Fatal(err) + } + data, err := os.ReadFile(filepath.Join(dir, "report.json")) + if err != nil { + t.Fatal(err) + } + var recorded struct { + Host struct { + Description string `json:"description"` + GeneratorDescription string `json:"generator_description"` + GeneratorLocal bool `json:"generator_on_target_host"` + } `json:"host"` + } + if err := json.Unmarshal(data, &recorded); err != nil { + t.Fatal(err) + } + if recorded.Host.GeneratorLocal || recorded.Host.Description != managed.HostDescription || recorded.Host.GeneratorDescription != managed.GeneratorHostDescription { + t.Fatalf("host=%+v", recorded.Host) + } + markdown, err := os.ReadFile(filepath.Join(dir, "report.md")) + if err != nil { + t.Fatal(err) + } + for _, want := range []string{"ccx33", "| Generator host | Hetzner Cloud ccx43 |", "| Generator placement | separate/remote |"} { + if !strings.Contains(string(markdown), want) { + t.Errorf("Markdown missing %q", want) + } + } + loaded, err := LoadReport(filepath.Join(dir, "report.json")) + if err != nil || loaded.Host.GeneratorDescription != managed.GeneratorHostDescription { + t.Fatalf("roundtrip lost generator host: %+v, %v", loaded.Host, err) + } +} + +func TestReproductionCommandRedactsGeneratorHostAndDropsAddresses(t *testing.T) { + for _, host := range []string{"root@192.0.2.11", "bench@generator.example", "root@[2001:db8::3]", "192.0.2.11"} { + cfg := Config{Host: "root@192.0.2.10", GeneratorHost: host, GeneratorAddress: "10.77.0.3", FanoutListen: "10.77.0.2", GeneratorSSH: "root@10.77.0.3", GeneratorSSHIdentity: "/stage/key", GeneratorSSHKnownHosts: "/stage/known_hosts", GeneratorHostDescription: "Hetzner ccx43"} + got := reproductionCommand(cfg) + for _, want := range []string{"--host 'root@'", "--generator-host 'root@'", "--generator-address ''", "--fanout-listen ''", "--generator-host-description 'Hetzner ccx43'"} { + if !strings.Contains(got, want) { + t.Errorf("missing %q in %s", want, got) + } + } + for _, unwanted := range []string{host, "192.0.2.10", cfg.GeneratorAddress, cfg.FanoutListen, "--generator-ssh", "/stage/"} { + if strings.Contains(got, unwanted) { + t.Errorf("unexpected %q in %s", unwanted, got) + } + } + } +} diff --git a/internal/bench/runner.go b/internal/bench/runner.go index 3200d1e..b02b532 100644 --- a/internal/bench/runner.go +++ b/internal/bench/runner.go @@ -66,6 +66,7 @@ func (r *Runner) Run(ctx context.Context) (*Report, error) { Settings: settingsFrom(r.Config), } report.Host.Description = r.Config.HostDescription + report.Host.GeneratorDescription = r.Config.GeneratorHostDescription report.Settings.Methodology += readMethod report.WorkloadSHA256 = workloadHash(report.Settings, plan) report.ReproductionCommand = reproductionCommand(r.Config) @@ -152,6 +153,9 @@ func (r *Runner) validate() error { case cfg.QueryRate > 0 && strings.TrimSpace(cfg.QueryCookie) == "": return errors.New("query session cookie is required when query rate is positive") } + if cfg.GeneratorSSH != "" { + return requireCommand("ssh") + } info, err := os.Stat(cfg.Telemetrygen) if err != nil { return fmt.Errorf("telemetrygen: %w", err) @@ -275,7 +279,8 @@ func (r *Runner) runGenerators(ctx context.Context, spec PhaseSpec) (int, int, e func (r *Runner) runGenerator(ctx context.Context, signal Signal, spec PhaseSpec) error { args := generatorArgs(r.Config, signal, spec) - cmd := exec.CommandContext(ctx, r.Config.Telemetrygen, args...) + cmd, cancel := generatorCommand(ctx, r.Config, spec.Duration, args...) + defer cancel() // telemetrygen logs its full configuration, including headers. Never stream // child output or persist argv. Retain only a bounded tail for redacted errors. output := &boundedBuffer{limit: 16 << 10} @@ -286,6 +291,30 @@ func (r *Runner) runGenerator(ctx context.Context, signal Signal, spec PhaseSpec return nil } +// The remote timeout bounds orphaned generators if the SSH connection is lost. +// Give export draining ten seconds of margin beyond the configured timeouts. +func generatorCommand(ctx context.Context, cfg Config, duration time.Duration, args ...string) (*exec.Cmd, context.CancelFunc) { + if cfg.GeneratorSSH == "" { + return exec.CommandContext(ctx, cfg.Telemetrygen, args...), func() {} + } + limit := duration + cfg.GeneratorTimeout + 10*time.Second + // Bound the SSH client too, allowing five seconds beyond the remote timeout. + ctx, cancel := context.WithTimeout(ctx, limit+5*time.Second) + remote := append([]string{"exec", "timeout", "--signal=TERM", strconv.FormatFloat(limit.Seconds(), 'f', -1, 64) + "s", cfg.Telemetrygen}, args...) + for i := range remote { + remote[i] = shellQuote(remote[i]) + } + port := cfg.GeneratorSSHPort + if port == 0 { + port = 22 + } + ssh := []string{"-p", strconv.Itoa(port), "-i", cfg.GeneratorSSHIdentity, + "-o", "ConnectTimeout=10", "-o", "ServerAliveInterval=15", "-o", "ServerAliveCountMax=3", + "-o", "BatchMode=yes", "-o", "StrictHostKeyChecking=accept-new", + "-o", "UserKnownHostsFile=" + cfg.GeneratorSSHKnownHosts, cfg.GeneratorSSH, "--", strings.Join(remote, " ")} + return exec.CommandContext(ctx, "ssh", ssh...), cancel +} + func generatorArgs(cfg Config, signal Signal, spec PhaseSpec) []string { args := []string{ string(signal), "--otlp-endpoint", cfg.Endpoint, @@ -566,8 +595,12 @@ func (r *Runner) buildManifest(ctx context.Context) Build { if b.TelemetrygenVersion == "" { cmdCtx, cancel := context.WithTimeout(ctx, 5*time.Second) defer cancel() - if output, err := exec.CommandContext(cmdCtx, r.Config.Telemetrygen, "--version").CombinedOutput(); err == nil { - b.TelemetrygenVersion = strings.TrimSpace(string(output)) + cmd, cancelGenerator := generatorCommand(cmdCtx, r.Config, 0, "--version") + defer cancelGenerator() + output := &boundedBuffer{limit: 16 << 10} + cmd.Stdout, cmd.Stderr = output, output + if err := cmd.Run(); err == nil { + b.TelemetrygenVersion = strings.TrimSpace(redact(output.String(), r.Config.Token, r.Config.QueryCookie, r.Config.MetricsToken)) } if b.TelemetrygenVersion == "" { b.TelemetrygenVersion = "unknown" diff --git a/internal/bench/runner_test.go b/internal/bench/runner_test.go index f2213cd..d38281f 100644 --- a/internal/bench/runner_test.go +++ b/internal/bench/runner_test.go @@ -1,12 +1,87 @@ package bench import ( + "context" "errors" + "os" "slices" "strings" "testing" + "time" ) +func TestGeneratorCommandWrapsSSHAndQuotesEveryArgument(t *testing.T) { + cfg := Config{Telemetrygen: "/stage/generator's binary", GeneratorSSH: "root@10.77.0.3", GeneratorSSHPort: 2222, GeneratorSSHIdentity: "/stage/key with spaces", GeneratorSSHKnownHosts: "/stage/known hosts", GeneratorTimeout: 30 * time.Second} + args := []string{"logs", "--otlp-header", "Authorization=\"Bearer token's $(touch /bad); value\""} + cmd, cancel := generatorCommand(context.Background(), cfg, 5*time.Second, args...) + defer cancel() + want := []string{"ssh", "-p", "2222", "-i", "/stage/key with spaces", "-o", "ConnectTimeout=10", "-o", "ServerAliveInterval=15", "-o", "ServerAliveCountMax=3", "-o", "BatchMode=yes", "-o", "StrictHostKeyChecking=accept-new", "-o", "UserKnownHostsFile=/stage/known hosts", "root@10.77.0.3", "--", + "'exec' 'timeout' '--signal=TERM' '45s' '/stage/generator'\\''s binary' 'logs' '--otlp-header' 'Authorization=\"Bearer token'\\''s $(touch /bad); value\"'"} + if !slices.Equal(cmd.Args, want) { + t.Fatalf("args=%q\nwant=%q", cmd.Args, want) + } + probe, cancelProbe := generatorCommand(context.Background(), cfg, 0, "--version") + defer cancelProbe() + if !slices.Equal(probe.Args[:len(probe.Args)-1], cmd.Args[:len(cmd.Args)-1]) { + t.Fatalf("version probe SSH options differ: %q", probe.Args) + } + if got := probe.Args[len(probe.Args)-1]; got != "'exec' 'timeout' '--signal=TERM' '40s' '/stage/generator'\\''s binary' '--version'" { + t.Fatalf("version command=%s", got) + } + cfg.GeneratorSSHPort = 0 + defaultPort, cancelDefault := generatorCommand(context.Background(), cfg, 0, "--version") + defer cancelDefault() + if defaultPort.Args[1] != "-p" || defaultPort.Args[2] != "22" { + t.Fatalf("default port missing: %q", defaultPort.Args) + } + cfg.GeneratorSSH = "" + local, cancelLocal := generatorCommand(context.Background(), cfg, 5*time.Second, args...) + defer cancelLocal() + if !slices.Equal(local.Args, append([]string{cfg.Telemetrygen}, args...)) { + t.Fatalf("same-host command changed: %q", local.Args) + } +} + +func TestGeneratorCommandClientDeadline(t *testing.T) { + if os.Getenv("FANOUT_BENCH_TEST_SILENT_CHILD") == "1" { + time.Sleep(time.Minute) + return + } + executable, err := os.Executable() + if err != nil { + t.Fatal(err) + } + for _, probe := range []bool{false, true} { + name := "launch" + if probe { + name = "version" + } + t.Run(name, func(t *testing.T) { + t.Parallel() + ctx, stop := context.WithTimeout(context.Background(), 20*time.Second) + defer stop() + args, duration := []string{"logs"}, time.Millisecond + if probe { + args, duration = []string{"--version"}, 0 + } + cmd, cancel := generatorCommand(ctx, Config{GeneratorSSH: "root@generator", Telemetrygen: "/telemetrygen"}, duration, args...) + defer cancel() + // Exercise the client's context with a silent local child; never run SSH. + cmd.Path = executable + cmd.Args = []string{executable, "-test.run=^TestGeneratorCommandClientDeadline$"} + cmd.Env = append(os.Environ(), "FANOUT_BENCH_TEST_SILENT_CHILD=1") + started := time.Now() + if err := cmd.Run(); err == nil { + t.Fatal("silent child escaped the client deadline") + } + elapsed := time.Since(started) + if elapsed < 14*time.Second || elapsed > 18*time.Second || ctx.Err() != nil { + t.Fatalf("client deadline took %s, parent error=%v; want remote limit + 5s", elapsed, ctx.Err()) + } + }) + } +} + func TestGeneratorArgsMatchPinnedTelemetrygenCLI(t *testing.T) { cfg := Config{Endpoint: "localhost:7520", Token: "token with space", GeneratorTimeout: 30, Insecure: true} spec := PhaseSpec{Duration: 5, RatePerWorker: 2500, WorkersPerProcess: 8, BatchSize: 1000} diff --git a/internal/bench/types.go b/internal/bench/types.go index 7aad1ed..c081594 100644 --- a/internal/bench/types.go +++ b/internal/bench/types.go @@ -16,38 +16,46 @@ const ( var Signals = []Signal{Traces, Logs, Metrics} type Config struct { - Host string - HostDescription string - SSHKnownHosts string - SSHIdentity string - SSHPort int - KeepRemote bool - Preset string - Endpoint string - MetricsURL string - Insecure bool - Telemetrygen string - Token string - MetricsToken string - OutputDir string - FanoutVersion string - FanoutCommit string - FanoutArtifact string - DataDir string - SampleInterval time.Duration - GeneratorTimeout time.Duration - Cooldown time.Duration - Repeat int - QueryURL string - QueryCookie string - QueryRate float64 - SameHost bool - ProcessOverride int - WorkerOverride int - RateOverride float64 - DurationOverride time.Duration - BatchSize int - TelemetrygenVersion string + Host string + HostDescription string + GeneratorHost string + GeneratorAddress string + FanoutListen string + GeneratorHostDescription string + GeneratorSSH string + GeneratorSSHPort int + GeneratorSSHIdentity string + GeneratorSSHKnownHosts string + SSHKnownHosts string + SSHIdentity string + SSHPort int + KeepRemote bool + Preset string + Endpoint string + MetricsURL string + Insecure bool + Telemetrygen string + Token string + MetricsToken string + OutputDir string + FanoutVersion string + FanoutCommit string + FanoutArtifact string + DataDir string + SampleInterval time.Duration + GeneratorTimeout time.Duration + Cooldown time.Duration + Repeat int + QueryURL string + QueryCookie string + QueryRate float64 + SameHost bool + ProcessOverride int + WorkerOverride int + RateOverride float64 + DurationOverride time.Duration + BatchSize int + TelemetrygenVersion string } type PhaseSpec struct { @@ -76,15 +84,16 @@ func (p PhaseSpec) OfferedPerSecond() float64 { } type Host struct { - Description string `json:"description,omitempty"` - OS string `json:"os"` - Architecture string `json:"architecture"` - CPUModel string `json:"cpu_model,omitempty"` - LogicalCPUs int `json:"logical_cpus"` - MemoryBytes uint64 `json:"memory_bytes,omitempty"` - GoVersion string `json:"go_version"` - Hostname string `json:"hostname,omitempty"` - GeneratorLocal bool `json:"generator_on_target_host"` + Description string `json:"description,omitempty"` + GeneratorDescription string `json:"generator_description,omitempty"` + OS string `json:"os"` + Architecture string `json:"architecture"` + CPUModel string `json:"cpu_model,omitempty"` + LogicalCPUs int `json:"logical_cpus"` + MemoryBytes uint64 `json:"memory_bytes,omitempty"` + GoVersion string `json:"go_version"` + Hostname string `json:"hostname,omitempty"` + GeneratorLocal bool `json:"generator_on_target_host"` } type Build struct { diff --git a/scripts/hetzner.sh b/scripts/hetzner.sh index 8e1f4dc..871c8db 100755 --- a/scripts/hetzner.sh +++ b/scripts/hetzner.sh @@ -3,7 +3,7 @@ set -euo pipefail if [[ $# -eq 0 ]]; then echo "Usage: scripts/hetzner.sh [-- ]…" >&2 - echo "Each nonempty group runs in order on one VM; SSH port and identity must agree across groups." >&2 + echo "Each nonempty group shares the Fanout and generator VMs; SSH port and identity must agree across groups." >&2 exit 2 fi @@ -73,6 +73,7 @@ validate_group "${group_args[@]}" : "${FANOUT_BENCH_HCLOUD_SSH_KEY:?Set FANOUT_BENCH_HCLOUD_SSH_KEY to a registered Hetzner SSH key name (or comma-separated names)}" server_type=${FANOUT_BENCH_HCLOUD_TYPE:-ccx33} +generator_type=${FANOUT_BENCH_HCLOUD_GENERATOR_TYPE:-ccx43} location=${FANOUT_BENCH_HCLOUD_LOCATION:-fsn1} image=${FANOUT_BENCH_HCLOUD_IMAGE:-ubuntu-24.04} @@ -101,37 +102,74 @@ run_dir=$(mktemp -d "${TMPDIR:-/tmp}/fanout-bench-hetzner.XXXXXX") known_hosts="$run_dir/known_hosts" touch "$known_hosts" nonce=$(od -An -N4 -tx1 /dev/urandom | tr -d ' \n') -server_name="fanout-bench-$(date -u +%Y%m%dT%H%M%SZ)-$nonce" -# Invoked indirectly by the EXIT trap. +name="fanout-bench-$(date -u +%Y%m%dT%H%M%SZ)-$nonce" +fanout_name="$name-fanout" +generator_name="$name-generator" +network_attempted=false +fanout_attempted=false +generator_attempted=false +# Deletion can race an interrupted create. Confirm absence rather than treating +# a missing resource as a cleanup failure; retry API and transient state errors. +# Invoked indirectly by the EXIT trap's cleanup function. # shellcheck disable=SC2329 -cleanup() { - local status=$? attempt deleted=false - trap - EXIT - trap '' INT TERM +delete_resource() { + local kind=$1 resource=$2 attempt names check absent for attempt in 1 2 3; do - if hcloud server delete "$server_name" >/dev/null; then - deleted=true - break + if hcloud "$kind" delete "$resource" >/dev/null; then + return 0 + fi + if names=$(hcloud "$kind" list --selector purpose=fanout-bench -o noheader -o columns=name) && + [[ $'\n'$names$'\n' != *$'\n'$resource$'\n'* ]]; then + absent=true + for ((check = 0; check < 5; check++)); do + sleep 2 + if ! names=$(hcloud "$kind" list --selector purpose=fanout-bench -o noheader -o columns=name) || + [[ $'\n'$names$'\n' == *$'\n'$resource$'\n'* ]]; then + absent=false + break + fi + done + if [[ $absent == true ]]; then + return 0 + fi fi if [[ $attempt -lt 3 ]]; then sleep 2 fi done - if [[ $deleted == false ]]; then - echo "Failed to delete Hetzner server $server_name after three attempts; delete it manually." >&2 - if [[ $status -eq 0 ]]; then - status=1 - fi + echo "Failed to delete Hetzner $kind $resource after three attempts; delete it manually." >&2 + return 1 +} +# Invoked indirectly by the EXIT trap. +# shellcheck disable=SC2329 +cleanup() { + local status=$? + trap - EXIT + trap '' INT TERM + if [[ $fanout_attempted == true ]]; then + delete_resource server "$fanout_name" || status=1 + fi + if [[ $generator_attempted == true ]]; then + delete_resource server "$generator_name" || status=1 + fi + if [[ $network_attempted == true ]]; then + delete_resource network "$name" || status=1 fi rm -rf -- "$run_dir" exit "$status" } # Arm cleanup before creation so an interrupted or failed create is covered too. benchmark_pid= +create_in_flight=false +interrupted_status= # Invoked indirectly by the signal traps. # shellcheck disable=SC2329 interrupt() { trap '' INT TERM + if [[ $create_in_flight == true ]]; then + interrupted_status=$2 + return + fi if [[ -n $benchmark_pid ]]; then kill -s "$1" "$benchmark_pid" 2>/dev/null || true wait "$benchmark_pid" || true @@ -141,12 +179,55 @@ interrupt() { trap cleanup EXIT trap 'interrupt INT 130' INT trap 'interrupt TERM 143' TERM -hcloud server create --name "$server_name" --type "$server_type" \ +# Keep long-running creates in children too, so traps run promptly while waiting. +run_child() { + "$@" <&0 & + benchmark_pid=$! + local status=0 + wait "$benchmark_pid" || status=$? + create_in_flight=false + if [[ -n $interrupted_status ]]; then + # The signal interrupted wait, not the create. Give the create two + # minutes to finish, then kill it; cleanup deletes resources by name. + local deadline=$((SECONDS + 120)) + while kill -0 "$benchmark_pid" 2>/dev/null && (( SECONDS < deadline )); do + sleep 1 + done + kill -s KILL "$benchmark_pid" 2>/dev/null || true + wait "$benchmark_pid" 2>/dev/null || true + exit "$interrupted_status" + fi + benchmark_pid= + return "$status" +} +run_create() { + create_in_flight=true + run_child "$@" +} +zone=$(hcloud location describe "$location" -o 'format={{.NetworkZone}}') +if [[ -z "$zone" ]]; then + echo "Hetzner location $location has no network zone" >&2 + exit 1 +fi +network_attempted=true +run_create hcloud network create --name "$name" --ip-range 10.77.0.0/16 \ + --label purpose=fanout-bench >/dev/null +run_create hcloud network add-subnet "$name" --type cloud --network-zone "$zone" \ + --ip-range 10.77.0.0/24 >/dev/null +fanout_attempted=true +run_create hcloud server create --name "$fanout_name" --type "$server_type" \ --location "$location" --image "$image" --label purpose=fanout-bench \ - "${key_args[@]}" >/dev/null -ip=$(hcloud server ip "$server_name") -if [[ -z "$ip" ]]; then - echo "Hetzner server $server_name has no IPv4 address" >&2 + --network "$name" "${key_args[@]}" >/dev/null +generator_attempted=true +run_create hcloud server create --name "$generator_name" --type "$generator_type" \ + --location "$location" --image "$image" --label purpose=fanout-bench \ + --network "$name" "${key_args[@]}" >/dev/null +fanout_ip=$(hcloud server ip "$fanout_name") +generator_ip=$(hcloud server ip "$generator_name") +fanout_private=$(hcloud server describe "$fanout_name" -o 'format={{(index .PrivateNet 0).IP}}') +generator_private=$(hcloud server describe "$generator_name" -o 'format={{(index .PrivateNet 0).IP}}') +if [[ -z "$fanout_ip" || -z "$generator_ip" || -z "$fanout_private" || -z "$generator_private" ]]; then + echo "Hetzner servers must have public and private IPv4 addresses" >&2 exit 1 fi @@ -157,13 +238,15 @@ ssh_args=(-o BatchMode=yes -o StrictHostKeyChecking=accept-new if [[ -n "$ssh_identity" ]]; then ssh_args+=(-i "$ssh_identity") fi -deadline=$((SECONDS + 300)) -until ssh "${ssh_args[@]}" "root@$ip" true 2>/dev/null; do - if (( SECONDS >= deadline )); then - echo "Timed out waiting for SSH on Hetzner server $server_name" >&2 - exit 1 - fi - sleep 2 +for ip in "$fanout_ip" "$generator_ip"; do + deadline=$((SECONDS + 300)) + until run_child ssh "${ssh_args[@]}" "root@$ip" true 2>/dev/null; do + if (( SECONDS >= deadline )); then + echo "Timed out waiting for SSH on Hetzner server at $ip" >&2 + exit 1 + fi + sleep 2 + done done statuses=() @@ -171,8 +254,11 @@ status=0 offset=1 for group_length in "${group_lengths[@]}"; do bin/fanout-bench run "${@:offset:group_length}" \ - --host "root@$ip" --ssh-known-hosts "$known_hosts" \ - --host-description "Hetzner Cloud $server_type, $location, $image" <&0 & + --host "root@$fanout_ip" --ssh-known-hosts "$known_hosts" \ + --generator-host "root@$generator_ip" --generator-address "$generator_private" \ + --fanout-listen "$fanout_private" \ + --host-description "Hetzner Cloud $server_type, $location, $image" \ + --generator-host-description "Hetzner Cloud $generator_type, $location, $image" <&0 & benchmark_pid=$! if wait "$benchmark_pid"; then statuses+=(0) From 7a8e89a3c37fd9476edb9354a9d699761a6ef077 Mon Sep 17 00:00:00 2001 From: Vishal Rana Date: Wed, 14 Oct 2026 21:38:51 -0700 Subject: [PATCH 4/4] fix: silence trap-only function warnings on older ShellCheck ShellCheck before 0.11 reports functions invoked only by traps as unreachable (SC2317); 0.11 uses SC2329. Disable both on those functions. --- scripts/hetzner.sh | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/scripts/hetzner.sh b/scripts/hetzner.sh index 871c8db..8a8e757 100755 --- a/scripts/hetzner.sh +++ b/scripts/hetzner.sh @@ -111,7 +111,7 @@ generator_attempted=false # Deletion can race an interrupted create. Confirm absence rather than treating # a missing resource as a cleanup failure; retry API and transient state errors. # Invoked indirectly by the EXIT trap's cleanup function. -# shellcheck disable=SC2329 +# shellcheck disable=SC2317,SC2329 delete_resource() { local kind=$1 resource=$2 attempt names check absent for attempt in 1 2 3; do @@ -141,7 +141,7 @@ delete_resource() { return 1 } # Invoked indirectly by the EXIT trap. -# shellcheck disable=SC2329 +# shellcheck disable=SC2317,SC2329 cleanup() { local status=$? trap - EXIT @@ -163,7 +163,7 @@ benchmark_pid= create_in_flight=false interrupted_status= # Invoked indirectly by the signal traps. -# shellcheck disable=SC2329 +# shellcheck disable=SC2317,SC2329 interrupt() { trap '' INT TERM if [[ $create_in_flight == true ]]; then