[storage] libcurl 8.5.0 skips WriteVector rewind on HTTP/2 retry when writebytecount==0, silently corrupting resumable uploads

[seankungrubrik]

Does this issue affect the google-cloud-cpp project?

Yes. The bug manifests through google/cloud/storage's Client::WriteObject / ObjectWriteStream interface. The root cause is in libcurl 8.5.0's Curl_retry_request (fixed in 8.7.0 by commit 14bcea074), but the SDK documentation declares libcurl 8.5.0 as a supported transport and relies on libcurl correctly invoking the seek callback on retry — which it silently fails to do under the conditions described. The result is that ObjectWriteStream delivers a corrupted upload to the server with no diagnosis of the cause.

---

What component of google-cloud-cpp is this related to?

google/cloud/storage — specifically the resumable upload path via Client::WriteObject with UploadContentLength.

---

Describe the bug

When using Client::WriteObject over HTTP/2 with libcurl 8.5.0, if the server sends a GOAWAY (or RST_STREAM) while libcurl has consumed data from its internal 64 KiB upload buffer (UPLOADBUFFER_DEFAULT) into the WriteVector read callback but has sent zero bytes on the wire (writebytecount == 0, which occurs when the HTTP/2 connection-level flow-control window is zero), libcurl's Curl_retry_request skips the rewind of the WriteVector. On retry, the client declares its original Content-Length: N but can only deliver N − 65536 bytes. If the server accepts the short body (GCS does, for resumable uploads), the stored object contains corrupted data: the first 65536 bytes of the source are lost and all subsequent bytes are shifted earlier by exactly 65536 bytes.

When CRC checking is enabled, ObjectWriteStream correctly surfaces this as a hash mismatch error (computed_hash vs received_hash). When disabled, the upload returns successfully with valid-looking metadata and the corruption is completely silent. In either case the SDK does not diagnose the cause as a mispositioned WriteVector or prevent the retry from proceeding with a short body.

The libcurl bug is in lib/transfer.c around line 1817: Curl_retry_request gates the rewind on writebytecount (bytes acknowledged sent) rather than bytes consumed from the read callback. When writebytecount == 0 it concludes no rewind is needed, even though the read callback already advanced the WriteVector position. Fixed upstream in commit 14bcea074, released in libcurl 8.7.0.

---

To Reproduce

The reproduction requires no GCS credentials and no network access beyond localhost. Two components: a hostile HTTP/2 server and a C++ driver using Client::WriteObject directly:

// Client pointed at localhost:8443 with anonymous credentials and local CA
auto client = gcs::Client(options);

// Fixed-size writes then Close(), mirroring the SDK's resumable upload pattern
auto wstream = client.WriteObject(bucket, object);
while (offset < payload.size()) {
    wstream.write(payload.data() + offset, write_buffer_size);
    offset += write_buffer_size;
}
wstream.Close();
// With CRC disabled: wstream.metadata() returns OK despite corrupted body.
// With CRC enabled:  wstream.bad() is set, hash mismatch error returned.

The HTTP/2 server:

1. Completes the resumable-upload POST (returns a session URI).
2. On the first PUT: advertises SETTINGS_INITIAL_WINDOW_SIZE = 0, ACKs the PUT headers, holds the zero window until libcurl's 64 KiB UPLOADBUFFER_DEFAULT has been drained from the read callback, then sends GOAWAY(last_stream_id=0) while writebytecount == 0.
3. On the retry PUT: accepts the incoming body.

Observable result on the server: declared 8388608 bytes, received 8323072, short by 65536 == UPLOADBUFFER_DEFAULT. Diffing the received bytes against the intended payload shows 100% shift-by-65536 displacement across the damaged span.

---

Expected behavior

Client::WriteObject should either: (a) detect that libcurl is about to retry with an un-rewound WriteVector and force the seek before allowing the retry; (b) return a clear error identifying the cause rather than an opaque hash mismatch; or (c) document that libcurl ≥ 8.7.0 is required for safe HTTP/2 uploads and enforce it at build or runtime.

---

Operating system

Ubuntu 24.04.

---

What compiler and version are you using?

g++ (Ubuntu 13.x).

---

What version of google-cloud-cpp are you using?

2.26 (static archive, linked from prebuilt .a files).

---

Additional context

• The libcurl fix: commit 14bcea074 in curl/curl, released in 8.7.0 (2024-03-27).
• The condition is reliably triggered in production by a congested GCS connection: the HTTP/2 connection-level flow-control window reaches zero, libcurl drains the UPLOADBUFFER_DEFAULT from WriteVector, and the connection then dies with writebytecount still 0. GOAWAY is routine server behavior — GCS sends it during rolling restarts, connection rebalancing, and max-age cycling — so this is reachable by normal server lifecycle events on a degraded network path, not only adversarial conditions.
• The SDK's CRC32C check detects the corruption when enabled: the SDK accumulates CRC on the bytes written via wstream.write() (correct source data) while GCS accumulates CRC on what it actually received (the shifted body); the mismatch sets wstream.bad().
• A standalone hostile server implementation and the C++ driver (using Client::WriteObject with no external dependencies) are available to share if helpful.