Repository navigation
gocached, cmd/gocached: replace put-queue mover governor with fixed 128 - #7
Merged
Merged
Conversation
The adaptive mover governor from f579d8a did not work in production. In the week since it was deployed to ci-gocached-aws-1 its limit averaged 8.0 to 8.4 per hour and peaked at 18. It made 617 increases and 568 decreases, almost always in immediately reverted pairs: the baseline it learned from idle intervals drifted down to 6-8ms while the median small-copy latency under any real load was 50-120ms, 7 to 20 times that and far past its 2x backoff threshold. Copies/s stayed at the same ~100/s ceiling as the old constant of 8, and the spooled lane blocked about 190,000 PUTs on its byte cap during peaks. A replacement governor driven by throughput instead of latency was written, but it is another 300+ lines of control loop with its own tuning constants and takes a minute or two to find the knee. The point was to have no knob for operators to turn, and a fixed high mover count gets there more simply. High concurrency is cheap here. Each mover blocked on the NFS mount costs one OS thread while the kernel queues its requests; it doesn't cause errors. The governor week effectively tested this already, since its cleanup unlinks ran across all 256 mover goroutines: the process peaked at 269 threads and 5.9k fds (of 990k), there were zero copy errors, and GETs barely touch the main directory (0.05/s vs 136/s from the hot tier), so background copies can't starve them. By Little's law, peak load (about 330 spooled PUTs/s at ~100ms per copy plus unlinks) needs about 55 concurrent movers. 128 leaves 2x headroom. So delete the governor and run a fixed 128 movers, settable with WithPutMovers or --put-movers=N. Keep the larger spooled-lane memory budget from f579d8a, which is what stopped the count-cap blocking (about 214,000 blocked PUTs in 9 days before, 1,800 in the 7 after). Replace the governor's gauges with gocached_put_queue_movers (the configured count); gocached_put_queue_movers_active keeps its name. This is an alternative to the throughput-governor branch, not a follow-up to it. Updates tailscale/corp#47471 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com> Change-Id: I318d13b2505ccf65d2f0e41e764522726a23782e
tomhjp
approved these changes
Oct 8, 2026
winckler
approved these changes
Oct 8, 2026
winckler
left a comment
There was a problem hiding this comment.
Yes, Less is more.
Once we confirm, we can always go back to a more elegant solution.
Please confirm that none of the prod deployment is setting the flag, as this changes semantics (I think all are using the default value)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The adaptive mover governor from f579d8a did not work in production.
In the week since it was deployed to ci-gocached-aws-1 its limit
averaged 8.0 to 8.4 per hour and peaked at 18. It made 617 increases
and 568 decreases, almost always in immediately reverted pairs: the
baseline it learned from idle intervals drifted down to 6-8ms while
the median small-copy latency under any real load was 50-120ms, 7 to
20 times that and far past its 2x backoff threshold. Copies/s stayed
at the same ~100/s ceiling as the old constant of 8, and the spooled
lane blocked about 190,000 PUTs on its byte cap during peaks.
A replacement governor driven by throughput instead of latency was
written, but it is another 300+ lines of control loop with its own
tuning constants and takes a minute or two to find the knee. The
point was to have no knob for operators to turn, and a fixed high
mover count gets there more simply.
High concurrency is cheap here. Each mover blocked on the NFS mount
costs one OS thread while the kernel queues its requests; it doesn't
cause errors. The governor week effectively tested this already, since
its cleanup unlinks ran across all 256 mover goroutines: the process
peaked at 269 threads and 5.9k fds (of 990k), there were zero copy
errors, and GETs barely touch the main directory (0.05/s vs 136/s
from the hot tier), so background copies can't starve them. By
Little's law, peak load (about 330 spooled PUTs/s at ~100ms per copy
plus unlinks) needs about 55 concurrent movers. 128 leaves 2x
headroom.
So delete the governor and run a fixed 128 movers, settable with
WithPutMovers or --put-movers=N. Keep the larger spooled-lane memory
budget from f579d8a, which is what stopped the count-cap blocking
(about 214,000 blocked PUTs in 9 days before, 1,800 in the 7 after).
Replace the governor's gauges with gocached_put_queue_movers (the
configured count); gocached_put_queue_movers_active keeps its name.
This is an alternative to the throughput-governor branch, not a
follow-up to it.
Updates tailscale/corp#47471