gocached: govern put-queue movers by throughput, not latency ratio

[bradfitz]

The mover governor compared each interval's median small-copy latency
to an "unloaded" baseline and only grew the limit while the ratio
stayed under 1.5x, backing off above 2x. In production that pinned it
at its floor of 8 within 30 seconds of starting: the baseline was
learned at 28ms during the first near-idle seconds, and at 8 concurrent
copies the median was 80 to 115ms, a ratio of 3 to 4. The governor read
that as saturation and, since a backlog kept demand constant, never
re-learned. Copy throughput stayed at the same 85/s the fixed 8 movers
had delivered, so the deploy changed nothing.

The premise was wrong. Per-copy latency on a network filesystem grows
with concurrency long before throughput stops growing, because requests
queue at the client and on the wire even when the server has capacity.
Latency is not the signal; throughput is.

Replace the controller with a hill-climber on measured copies/s. The
governor keeps a committed operating point (a limit and the throughput
measured there) and probes from it: raise the limit by 25% and keep the
raise only if copies/s rose by at least half the ideal linear gain;
lower it by 25% and keep that if copies/s held within 10%. A rejected
probe reverts and holds for 30s before trying again. In steady state
every fourth probe is downward so a limit that drifted above the knee
gets trimmed, and a 25% fall in throughput at the committed limit
triggers an immediate downward probe since the directory may have
gotten slower. Windows where the slots were less than 90% busy are
discarded: throughput was bounded by demand, not the limit, and says
nothing about capacity. A window also needs 512 completions before it
counts, which puts Poisson noise on a two-window comparison near 6%,
under the 12.5% a probe must show.

In simulation against a directory that scales linearly to a knee, the
limit reaches 90% of the ceiling in 40 to 110 seconds from the floor,
settles within a step of the knee, stays there under 15% throughput
jitter, trims within minutes when the ceiling drops, and climbs back
when it rises. Tests cover each of those, plus demand-limited and
sparse windows leaving the limit alone.

Also bound cleanup-intent deletions to the first minLimit mover
goroutines. Letting all 256 do them turned every metadata flush into a
burst of about 85 concurrent unlinks against the main directory, which
is the likely reason copy latency doubled under the first governor at
the same 8-copy concurrency; this returns the unlink load to what a
fixed-mover server produced.

The latency baseline gauges go away. New gauges export the committed
limit and the copies/s measured there, which together read as the main
directory's throughput ceiling whenever the committed limit is below
the maximum, and a count of rejected probes, which climbs steadily in
both directions once the governor has found the knee.

Updates tailscale/corp#47471

Migrated-from: bradfitz/go-tool-cache@13c5751

[bradfitz]

Deleted this all via #7 instead