Skip to content

gocached, cmd/gocached: replace put-queue mover governor with fixed 128 - #7

Merged
bradfitz merged 1 commit into
mainfrom
bradfitz/rm_putq_gov
Oct 8, 2026
Merged

bradfitz merged 1 commit into
mainfrom
bradfitz/rm_putq_gov

Conversation

@bradfitz

@bradfitz bradfitz commented Oct 8, 2026

Copy link
Copy Markdown
Member

The adaptive mover governor from f579d8a did not work in production.
In the week since it was deployed to ci-gocached-aws-1 its limit
averaged 8.0 to 8.4 per hour and peaked at 18. It made 617 increases
and 568 decreases, almost always in immediately reverted pairs: the
baseline it learned from idle intervals drifted down to 6-8ms while
the median small-copy latency under any real load was 50-120ms, 7 to
20 times that and far past its 2x backoff threshold. Copies/s stayed
at the same ~100/s ceiling as the old constant of 8, and the spooled
lane blocked about 190,000 PUTs on its byte cap during peaks.

A replacement governor driven by throughput instead of latency was
written, but it is another 300+ lines of control loop with its own
tuning constants and takes a minute or two to find the knee. The
point was to have no knob for operators to turn, and a fixed high
mover count gets there more simply.

High concurrency is cheap here. Each mover blocked on the NFS mount
costs one OS thread while the kernel queues its requests; it doesn't
cause errors. The governor week effectively tested this already, since
its cleanup unlinks ran across all 256 mover goroutines: the process
peaked at 269 threads and 5.9k fds (of 990k), there were zero copy
errors, and GETs barely touch the main directory (0.05/s vs 136/s
from the hot tier), so background copies can't starve them. By
Little's law, peak load (about 330 spooled PUTs/s at ~100ms per copy
plus unlinks) needs about 55 concurrent movers. 128 leaves 2x
headroom.

So delete the governor and run a fixed 128 movers, settable with
WithPutMovers or --put-movers=N. Keep the larger spooled-lane memory
budget from f579d8a, which is what stopped the count-cap blocking
(about 214,000 blocked PUTs in 9 days before, 1,800 in the 7 after).
Replace the governor's gauges with gocached_put_queue_movers (the
configured count); gocached_put_queue_movers_active keeps its name.

This is an alternative to the throughput-governor branch, not a
follow-up to it.

Updates tailscale/corp#47471

The adaptive mover governor from f579d8a did not work in production.
In the week since it was deployed to ci-gocached-aws-1 its limit
averaged 8.0 to 8.4 per hour and peaked at 18. It made 617 increases
and 568 decreases, almost always in immediately reverted pairs: the
baseline it learned from idle intervals drifted down to 6-8ms while
the median small-copy latency under any real load was 50-120ms, 7 to
20 times that and far past its 2x backoff threshold. Copies/s stayed
at the same ~100/s ceiling as the old constant of 8, and the spooled
lane blocked about 190,000 PUTs on its byte cap during peaks.

A replacement governor driven by throughput instead of latency was
written, but it is another 300+ lines of control loop with its own
tuning constants and takes a minute or two to find the knee. The
point was to have no knob for operators to turn, and a fixed high
mover count gets there more simply.

High concurrency is cheap here. Each mover blocked on the NFS mount
costs one OS thread while the kernel queues its requests; it doesn't
cause errors. The governor week effectively tested this already, since
its cleanup unlinks ran across all 256 mover goroutines: the process
peaked at 269 threads and 5.9k fds (of 990k), there were zero copy
errors, and GETs barely touch the main directory (0.05/s vs 136/s
from the hot tier), so background copies can't starve them. By
Little's law, peak load (about 330 spooled PUTs/s at ~100ms per copy
plus unlinks) needs about 55 concurrent movers. 128 leaves 2x
headroom.

So delete the governor and run a fixed 128 movers, settable with
WithPutMovers or --put-movers=N. Keep the larger spooled-lane memory
budget from f579d8a, which is what stopped the count-cap blocking
(about 214,000 blocked PUTs in 9 days before, 1,800 in the 7 after).
Replace the governor's gauges with gocached_put_queue_movers (the
configured count); gocached_put_queue_movers_active keeps its name.

This is an alternative to the throughput-governor branch, not a
follow-up to it.

Updates tailscale/corp#47471

Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I318d13b2505ccf65d2f0e41e764522726a23782e
@bradfitz
bradfitz requested a review from a team October 8, 2026 13:39
@bradfitz
bradfitz merged commit 6bd4181 into main Oct 8, 2026
3 checks passed

@winckler winckler left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, Less is more.

Once we confirm, we can always go back to a more elegant solution.

Please confirm that none of the prod deployment is setting the flag, as this changes semantics (I think all are using the default value)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants