feat(search): use stable node IDs for split affinity to keep disk cache warm when searcher pods restart

[congx4]

This is independent of the split-range disk cache stack. It is useful for cache locality after a same-node restart, but it does not require that cache.

Summary

• Search job placement (and related rendezvous uses: KV replication, report_splits) now hashes the cluster node_id / QW_NODE_ID instead of the gRPC SocketAddr.
• The searcher pool stays keyed by address, so RPC dialing and DataFusion worker URLs are unchanged. A restart that keeps the same node ID keeps the same split affinity even if the listen address changes.
• node_id docs note that searchers should keep this ID stable (for example a StatefulSet pod name) if the same splits should keep landing on the same node.

What this does not fix

• One-time remap: deploying this changes every split's preferred searcher once, so any address-based warm cache goes cold for that rollout.
• Unstable IDs: if QW_NODE_ID is a random Deployment hostname, affinity is still unstable. The default remains the short hostname.
• Generation is not hashed: Chitchat's generation is a per-process incarnation (restart counter). Hashing it would move splits on every restart, which is the opposite of this PR. A higher generation keeps the same affinity as long as node_id is unchanged.
• Scroll KV during mixed-version rollout: put_kv/get_kv share this ranking. While old and new coordinators coexist, an in-flight scroll can miss a live replica (possible at 8+ searchers). Homogeneous fleet and replica restarts are unchanged from today; MiniKV is still in-memory.

Test plan

• cargo test -p quickwit-search --all-features --lib
• cargo test -p quickwit-serve --all-features --lib -- setup_searcher datafusion_worker
• cargo test -p quickwit-janitor --all-features --lib
• cargo test -p quickwit-datafusion --all-features
• cargo clippy -p quickwit-search -p quickwit-serve -p quickwit-datafusion --all-features --tests -- -D warnings
• Confirm a two-searcher restart with a stable QW_NODE_ID and a changed gRPC port keeps the same split assignment

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 057b3f7124

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".