fix(storage): stabilize benchmark ALTS handshakes and static IRQ affinity

[shradhakatyal]

Summary

In benchmarks-cloudbuild.yaml, the standing benchmark VM (gcs-benchmark-runner-us-west4-a) is cold-started immediately before executing run_benchmark_tests.sh with 48 concurrent worker processes (PROCESSES=48). Two cold-start effects reduce the 3-round average read throughput:

1. ALTS Metadata Server Handshake Storm: By default in gRPC, GRPC_ALTS_MAX_CONCURRENT_HANDSHAKES is 100 per process (48 * 100 = 4,800 potential concurrent streams). When all 48 processes simultaneously initiate DirectPath ALTS handshakes against the local GCE metadata server (169.254.169.254:8080, ~100 stream buffer limit), the metadata server drops streams with UNAVAILABLE: GOAWAY received; Error code: 11; Debug Text: Buffers full {http2_error:11}, stalling Round 1 from 35.3s (5s warmup + 30s measurement) to 41.5s–49.0s.
2. Dynamic IRQ Migration on Cold Boot: When _worker_init() initializes the 48-worker pool, it reads /proc/irq/*/smp_affinity_list once via get_irq_affinity() and pins workers to non-IRQ vCPUs. On a freshly booted VM, the continuous background irqbalance daemon dynamically migrates gVNIC interrupt queues every 10 seconds as traffic ramps up, colliding with pinned worker vCPUs.

Change

• Configure and export GRPC_ALTS_MAX_CONCURRENT_HANDSHAKES="${GRPC_ALTS_MAX_CONCURRENT_HANDSHAKES:-2}" in packages/google-cloud-storage/cloudbuild/run_benchmark_tests.sh so 48 worker processes never exceed 96 concurrent ALTS handshakes against the local metadata server while still handshaking multiple DirectPath subchannels concurrently.
• Stop the continuous background irqbalance daemon and execute a single one-shot pass (sudo irqbalance --oneshot) before starting pytest.

[shradhakatyal]

/gcbrun (512d4e7)

[gemini-code-assist]

Code Review

This pull request enhances the GCS DirectPath read benchmark by adding median and maximum throughput columns to the results display, and optimizing the benchmark execution script. Key script improvements include limiting concurrent ALTS handshakes, preferring modern Python versions (3.11/3.12), pre-compiling bytecode, and pre-warming gRPC channels. Feedback on these changes highlights alignment issues in the output table borders, potential type conversion errors when parsing throughput values, risks of bypassing dependency resolution with --no-deps during pip installation, and the suppression of stderr during the pre-warming step which hinders debugging.

[shradhakatyal]

/gcbrun (7d52fce)

[shradhakatyal]

/gcbrun (d78d379)

[shradhakatyal]

/gcbrun (2701e2e)

[shradhakatyal]

/gcbrun (0137b2e)

[shradhakatyal]

/gcbrun 5cf9e92

[shradhakatyal]

/gcbrun (00ff4d5)

[shradhakatyal]

/gcbrun (28bd981)