gocached: shard put-queue cleanup intents and temp files by blob

[bradfitz]

On ci-gocached-aws-1's S3 Files NFS mount, the kernel serializes
creates, unlinks, and renames within a directory, holding the
directory's lock for each operation's server round trip of about 5ms.
A benchmark measured it directly: with 64 goroutines creating and
unlinking files in one directory, 62 were parked on the directory lock
and about one was in an NFS RPC, for 96 creates+unlinks a second at any
concurrency, while the same load over 256 directories reached 3,300 a
second with 47 RPCs in flight. That one-directory ceiling is the put
queue's: every copy into the main blob directory created and later
unlinked a cleanup intent in the single .cleanup directory, and created
its temp file in the main directory's root before renaming it out.

The change in #8 stops writing intents for the most common case, blobs
already present. For the blobs that are really copied, spread the
remaining mutations over the 256 shards the main directory already
uses:

• Intents go in .cleanup/<xx>/, xx being the first two hex digits of
  the blob's name, rather than directly in .cleanup.
• The temp file is created in the blob's own shard directory instead
  of the main directory's root, so its rename into place stays within
  that one directory and locks nothing else.

Both directories are created on demand: a create is tried first and
the directory made only if it fails with ENOENT, as renameCreatingDir
did for the rename, which no longer needs that since the temp file now
lives in the target's directory.

The sweeper lists .cleanup and then each shard subdirectory in it,
sweeping intents in both the new layout and, for upgrades, the old flat
one. That's 257 directory listings a minute instead of one, each
locking only its own subdirectory, and only briefly.

Nothing else lists the main directory's shard directories (usage stats
come from SQLite, and GETs and eviction use exact names), so a temp
file appearing in one while a copy is in flight changes nothing.

Rolling back to an older version after this one has run would leave
intents in shard subdirectories unswept, and the old sweeper would log
each subdirectory as a malformed intent once a minute.

Updates tailscale/corp#47471

Signed-off-by: Brad Fitzpatrick bradfitz@tailscale.com
Change-Id: Icadb72b048c11f6e58c3332903a50438c561a283

[runewake2]

LGTM