From ef26b4033a22df1ee401d1b5da17e25fa16eea8c Mon Sep 17 00:00:00 2001 From: Feng Ruohang Date: Mon, 24 Aug 2026 21:35:03 +0800 Subject: [PATCH] docs: expand CopyObject checksum design Signed-off-by: Feng Ruohang --- .../blog/design/copyobject-checksum/index.md | 362 ++++++++++++++++-- .../design/copyobject-checksum/index.zh.md | 362 ++++++++++++++++-- 2 files changed, 650 insertions(+), 74 deletions(-) diff --git a/content/blog/design/copyobject-checksum/index.md b/content/blog/design/copyobject-checksum/index.md index 3bdb59f3..b07d80a6 100644 --- a/content/blog/design/copyobject-checksum/index.md +++ b/content/blog/design/copyobject-checksum/index.md @@ -4,71 +4,359 @@ date: 2026-08-24 lastmod: 2026-08-24 author: "Ruohang Feng" summary: > - When destination compression was enabled, SILO could persist a CopyObject checksum of the S2 storage stream instead of the logical S3 object. This record explains the plaintext-reader invariant, verification boundary, related fixes, and remediation of older objects. + Destination compression made it possible for SILO to persist a checksum of the S2 storage stream instead of the logical CopyObject result. This design record reconstructs the failure, compares the rejected fixes, defines the plaintext-reader invariant, proves why the selected solution works, and records rollout and remediation boundaries. tags: [Design, S3, Compatibility, Checksum] weight: 10 draft: false url: "/blog/design/copyobject-checksum/" --- -This is the design and verification record for [SILO #63](https://github.com/pgsty/silo/issues/63). +This is the final design and verification record for [SILO issue #63](https://github.com/pgsty/silo/issues/63). -**Status:** the checksum-domain fix was merged through [PR #66](https://github.com/pgsty/silo/pull/66); public release pending. -**Related fixes:** metadata-only transform state [#67](https://github.com/pgsty/silo/issues/67) through [PR #69](https://github.com/pgsty/silo/pull/69), and CopyObjectResult checksum fields [#68](https://github.com/pgsty/silo/issues/68) through [PR #70](https://github.com/pgsty/silo/pull/70); both merged, public release pending. -**Upstream client:** [minio-go #2295](https://github.com/minio/minio-go/pull/2295). -**Release boundary:** a merge does not prove that a release artifact, package, or container image already contains the fix. +**Decision:** compute every server-generated CopyObject checksum over the logical destination object before compression or encryption, retain that reader separately from the storage reader, and refuse to publish the object if the expected checksum is unavailable at EOF.
+**Implementation:** [PR #66](https://github.com/pgsty/silo/pull/66), merged as commit c0e715977.
+**Related repairs:** transform-state preservation [#67](https://github.com/pgsty/silo/issues/67) / [PR #69](https://github.com/pgsty/silo/pull/69), and CopyObjectResult checksum fields [#68](https://github.com/pgsty/silo/issues/68) / [PR #70](https://github.com/pgsty/silo/pull/70).
+**Upstream client:** [minio-go PR #2295](https://github.com/minio/minio-go/pull/2295).
+**Release boundary:** these changes are merged into source, but no statement on this page implies that a particular release tag, RPM, DEB, APK, archive, or container image already contains them. -## The defect {#defect} +## Decision in one sentence {#decision} -CopyObject reads the source as logical object data, then may compress and encrypt the destination storage stream. The old handler installed a requested server-side checksum on a reader that already represented compressed bytes: +A checksum is not merely a digest produced somewhere along the write path. It is a function of a precisely defined byte sequence. For S3 CopyObject, that sequence is the logical object returned to a client, not SILO's compressed or encrypted representation of that object. - logical object -> S2 compression -> checksum -> optional encryption -> storage +The accepted pipeline is therefore: -The digest was valid but covered the wrong byte domain. A client downloading and independently hashing the object therefore obtained a different value. The API reproduction was deterministic: +~~~text +logical source bytes + -> server-side S3 checksum + -> optional S2 compression + -> storage-stream hash and ETag delegation + -> optional server-side encryption + -> erasure coding + -> EOF checksum validation + -> atomic data and metadata commit +~~~ - stored CRC32 before the fix: hN7ytg== - logical object CRC32: 1WxbLg== +Everything else in this design follows from preserving that ordering. -All five algorithms implemented by this SILO baseline were affected: CRC32, CRC32C, CRC64NVME, SHA1, and SHA256. Compression combined with encryption made the wrong value nondeterministic because encrypted-stream S2 padding is randomized. +## Background: one object, several integrity domains {#background} -## Accepted invariant {#invariant} +SILO handles several values that are all casually called a checksum, but they protect different contracts. -The logical checksum reader is now separate from storage transformation readers: +| Value | Byte domain | Purpose | +| --- | --- | --- | +| Additional S3 checksum | Logical object bytes | Client-visible end-to-end integrity through HEAD, GET, attributes, and copy responses | +| ETag | Logical content in the ordinary single-part, unencrypted-compatible case; otherwise protocol-specific | Object identity and conditional request compatibility | +| Storage reader accounting | Compressed or encrypted write stream | Carry size, stream, and ETag delegation through the write path | +| Erasure bitrot checksum | Stored erasure shards | Detect corruption of SILO's physical representation | +| Encryption authentication | Ciphertext framing and keys | Detect tampering and authenticate encrypted storage | +| Compression index | S2 storage stream offsets | Support efficient reads of large compressed objects | - logical object - -> server-side checksum - -> optional S2 compression - -> storage hash - -> optional server-side encryption - -> erasure coding and commit +These values may be computed during one streaming write, but they are not interchangeable. In particular, a storage-stream checksum can be perfectly valid while being completely wrong as an S3 object checksum. -The handler installs the hasher before starting the compression goroutine. PutObjReader retains the logical reader even when its active storage reader is replaced. At EOF, the object layer requires the checksum to exist, be valid, and match the expected base algorithm before committing metadata. +Amazon S3 documents that CopyObject produces a destination checksum and that a multipart source copied in one operation becomes a full-object checksum. The algorithm may be selected by the request, inherited from the source, or defaulted when the source has no checksum. The result describes the copied object, not a provider's private storage encoding. -This reuses the checksumReader contract introduced for multipart upload. It adds no second abstraction, no second object read, and no new on-disk representation. +## How #46 exposed #63 {#discovery} -## Verification boundary {#verification} +The bug was found while repairing multipart checksum compatibility in [#46](https://github.com/pgsty/silo/issues/46). -The permanent API suite covers all five algorithms and default CRC64NVME; uncompressed, compressed, encrypted-only, and compressed-plus-encrypted destinations; SSE-C and SSE-S3; encrypted and compressed sources; versioned buckets; full and multipart-composite source checksums; in-place copy; zero-length and threshold data; indexed S2 streams; ETag; body round trip; HEAD/GET checksum mode; and internal invariant failures. +That work established three internal rules for UploadPart and UploadPartCopy: -The regression is red on the unfixed baseline and green on the repaired tree. Focused race tests, shuffled repeated runs, full cmd tests, the CGO-disabled kqueue/dev CI shape, lint, vet, cross-compilation, compatibility guards, and remote CI were also required before merge. +1. Keep a dedicated reader for logical plaintext checksum calculation. +2. Install a server fallback hasher in the handler, before compression or encryption can consume the stream. +3. Let the object layer validate and persist the completed result, rather than deciding the byte domain there. -## Adjacent defects kept separate {#adjacent} +The resulting private field, checksumReader, deliberately remained separate from the active Reader and the historical rawReader. WithEncryption may replace the active storage reader, but must not replace the logical checksum reader. -Adversarial review found two inherited defects in nearby code: +Reviewing ordinary CopyObject after #46 showed the same conceptual hazard in a different handler. The code created newS2CompressReader, wrapped its output as srcInfo.Reader, and only later called AddServerSideChecksumHasher on that reader. At that point the name srcInfo.Reader concealed an important fact: it represented the storage stream, not necessarily the S3 object stream. -1. A metadata/reference-only self-copy could change compression markers without rewriting referenced data. Versioned SSE-C key rotation could also fall into an invalid rewrite. This is isolated in [#67](https://github.com/pgsty/silo/issues/67). -2. Successful CopyObject XML omitted checksum elements after the checksum was committed. The server response fix is [#68](https://github.com/pgsty/silo/issues/68); minio-go also discarded those fields and is followed in [#2295](https://github.com/minio/minio-go/pull/2295). +#46 intentionally did not change CopyObject. Keeping #63 separate meant that the P0 multipart repair could be reviewed, released, or rolled back without bundling another API and another test matrix. -Legacy federated UploadPartCopy checksum recovery is a different API and remains [#64](https://github.com/pgsty/silo/issues/64). +## Failure model {#failure-model} -The archived upstream minio/minio tree retains the original placement. silo-pkg does not own this reader chain. MCLI switches from server-side copy to download/upload when --checksum is requested, and SILO Console only passes CopyObject through minio-go, so neither required a duplicate server fix. +### The old ordering -## Existing objects {#existing-objects} +The relevant old flow was: -The repair affects future CopyObject operations. It does not scan or rewrite checksum metadata already stored by an affected version. +~~~text +GetObject logical reader + -> start S2 compressor goroutine + -> wrap compressed output in hash.Reader + -> later choose destination checksum algorithm + -> attach server-side hasher to compressed hash.Reader + -> persist that result as the object's S3 checksum +~~~ -Objects are candidates for verification when they were created by CopyObject, destination compression matched their key or content type, and they carry an additional S3 checksum. Retrieve the object with checksum mode enabled, independently hash the downloaded logical bytes with the reported algorithm, and compare the Base64 values. +The checksum did not cover missing or corrupt data from the storage writer's point of view. It covered the wrong, complete stream. -To repair an object, copy it to a new key while explicitly selecting the checksum algorithm. An in-place copy is possible with x-amz-metadata-directive: REPLACE, but it rewrites the object and replaces the current value on an unversioned bucket; a versioned bucket receives a new version. Validate retention, legal hold, metadata, tags, encryption keys, free capacity, and rollback requirements before bulk remediation. +### Static hypothesis versus dynamic result -SILO does not perform automatic online backfill because that would read and rewrite user data outside an explicit S3 operation. +The original issue described two possible failures: + +- the hasher could cover compressed data; +- the compression goroutine could consume logical input before the hasher was attached, causing a prefix to be missed. + +The API reproduction confirmed the first and did not confirm the second. The hasher was attached to the compressor's output reader, so it observed the complete transformed stream from that reader's beginning. Bytes consumed on the compressor's input side were not bytes consumed from the output-side hash reader. + +This distinction matters. The root cause is not an intermittent race that merely needs a lock. It is a deterministic data-domain error. + +### Concrete reproduction + +For the permanent test payload, the unfixed tree stored: + +~~~text +CRC32 of S2 storage bytes: hN7ytg== +CRC32 of logical bytes: 1WxbLg== +~~~ + +The stored value was a legitimate CRC32, which is why ordinary metadata validation did not catch it. Only an independent checksum of the downloaded logical object exposed the mismatch. + +CRC32, CRC32C, CRC64NVME, SHA1, and SHA256 all failed for compressed destinations. When compression and destination encryption were combined, S2 used randomized padding for the encrypted stream. The checksum was then not only wrong but nondeterministic across identical logical copies. + +## Requirements and non-goals {#requirements} + +The repair had to satisfy all of the following: + +1. **Correct byte domain.** Server-generated checksums cover exactly the logical destination bytes. +2. **Single pass.** CopyObject must remain streaming; no second object read. +3. **Transformation independence.** Compression and encryption cannot change the logical checksum. +4. **Client compatibility.** Existing client-supplied checksum validation and algorithm selection remain unchanged. +5. **Multipart-source correctness.** A composite checksum from a multipart source is recomputed as a full-object checksum for the single-operation destination. +6. **Default behavior.** A source without a checksum still gives the destination the configured S3-compatible default, CRC64NVME in this baseline. +7. **ETag preservation.** Moving the checksum reader cannot silently change the CopyObject ETag contract. +8. **Fail closed.** If an internal caller asks for a server checksum but fails to produce it, SILO must not return success with missing integrity metadata. +9. **Format compatibility.** Persist results in the existing checksum metadata representation. +10. **Small rollback boundary.** Do not mix response-schema, federation, or metadata-only transform bugs into the core placement fix. + +The following were explicit non-goals for #63: + +- adding new checksum algorithms; +- changing the on-disk checksum encoding; +- scanning or backfilling old objects; +- fixing legacy federated UploadPartCopy; +- adding CopyObjectResult XML fields; +- changing MCLI or Console behavior. + +## Alternatives considered {#alternatives} + +| Option | Attraction | Why it was rejected | +| --- | --- | --- | +| Attach the hasher in the object layer | One centralized fallback for every caller | The object layer receives a storage-oriented reader after handler transformations. It cannot reliably reconstruct the logical byte domain, and attachment may be too late | +| Hash the S2 output | Minimal code movement | This is the demonstrated bug: it protects storage bytes, not S3 object bytes | +| Hash ciphertext | Convenient after encryption setup | Encryption IVs, framing, and authentication make the value provider-specific and often nondeterministic | +| Read the completed object a second time | Easy to reason about | Doubles I/O, breaks the single-pass streaming goal, delays responses, and is expensive for large or tiered objects | +| Buffer the whole object before transforming it | Simple sequencing | CopyObject supports large objects; whole-object buffering creates unacceptable memory and latency costs | +| Always copy the source checksum value | Avoids computation | Fails when the request selects another algorithm, when the source has no checksum, and when a multipart composite source must become a full-object checksum | +| Add a second CopyObject-only checksum abstraction | Keeps code local | Duplicates the invariant already created by #46 and gives future paths two subtly different contracts | +| Reuse the logical checksumReader before transformations | One streaming pass, existing metadata format, common invariant | Selected | + +The selected option is not simply the one with the fewest changed lines. It is the smallest option that makes the byte-domain contract explicit and reusable. + +## Final design {#design} + +### 1. Construct the logical reader first + +CopyObject obtains a source GetObjectReader that already yields the logical source object: stored compression has been decoded and source encryption has been removed using the authorized source options. + +SILO wraps that stream in a logical hash.Reader with the known actual object size. For compressed destinations this also tightens the old unlimited length into a hard logical-size bound before compression. + +At this point no compression goroutine has started. + +### 2. Choose the destination checksum policy + +The existing policy remains intact: + +1. If the request supplies x-amz-checksum-algorithm, compute that base algorithm. +2. Otherwise inspect the source checksum. +3. A source full-object checksum can be retained because the logical bytes are unchanged. +4. A source multipart composite checksum must be recomputed with its base algorithm because CopyObject creates a single-operation full object. +5. A source without checksum metadata receives the default CRC64NVME checksum. + +Only branches that require computation call AddServerSideChecksumHasher. + +### 3. Start compression after hasher installation + +For a compressed destination, the logical reader is captured as checksumReader and passed as the input to newS2CompressReader. The compressor therefore cannot obtain one byte without that byte first passing through the logical hasher. + +The compressed output receives its own storage hash.Reader. A new PutObjReader is built around that storage reader, then setChecksumReader restores the logical reader reference. + +~~~text +PutObjReader.Reader = compressed or encrypted storage stream +PutObjReader.rawReader = stream used for the historical ETag path +PutObjReader.checksumReader = logical plaintext stream +~~~ + +No exported method or new package-level abstraction is required. + +### 4. Preserve the separation through encryption + +Destination encryption wraps the compressed storage reader and may replace PutObjReader.Reader through WithEncryption. It does not modify checksumReader. + +The destination checksum is therefore identical for: + +- plaintext storage; +- compressed storage; +- encrypted storage; +- compressed and encrypted storage. + +If checksum metadata itself must be protected, the existing metadata encryption function encrypts the serialized checksum after calculation. That protects metadata at rest without changing what bytes were hashed. + +### 5. Finalize at EOF and fail closed + +The internal hash reader sets ServerSideChecksumResult only after its source returns EOF. In the compressed path, io.Copy drains the logical checksum reader before the compressor closes the pipe. The object writer cannot observe the compressed stream's EOF before the logical reader has finalized its checksum. + +The object layer then validates: + +- the result is present; +- the result is structurally valid; +- its base algorithm matches WantServerSideChecksumType. + +Failure logs an internal invariant violation and aborts the write. Deferred erasure cleanup removes temporary shards before unique metadata is published. Returning HTTP 200 without a requested or default checksum would be a silent correctness failure and is therefore not an acceptable fallback. + +### 6. Persist without a format change + +The validated checksum is appended to the same FileInfo.Checksum representation already used by existing objects. Encrypted destinations reuse the existing metadata encrypter. HEAD, GET, GetObjectAttributes, replication metadata, and later readers continue to consume the same representation. + +## Why the design is correct {#proof} + +### Byte-domain proof + +Every byte accepted by the compressor is read from checksumReader. The hasher is installed before the compressor is constructed. Therefore the digest input is exactly the compressor's logical input, not its output. + +### Completeness proof + +The compressor closes its output only after draining the logical input and closing the S2 writer. The object writer must read that output to EOF before completing the write. The checksum result is finalized on the logical input EOF, which precedes the observable storage EOF. + +This creates a natural happens-before relationship through the pipe; no separate mutex or out-of-band signal is necessary. Targeted race tests and repeated shuffled executions confirm the implementation. + +### ETag proof + +The compressed reader is wrapped with the logical reader as its ETag delegate. Moving the S3 checksum hasher does not move ETag calculation onto S2 bytes. Permanent tests independently compare the final ETag with the logical object's MD5 in the compatible plaintext cases. + +### Storage-integrity proof + +The storage-side reader remains after compression for physical stream accounting and ETag delegation, while the erasure layer still writes its own bitrot protection for stored shards. Neither mechanism is replaced by the S3 logical checksum, and the S3 checksum is not presented as shard integrity. + +### Encryption proof + +The encryption reader consumes the storage stream after logical checksum calculation. Random encryption or padding cannot influence the checksum. SSE-C and SSE-S3 tests cover encrypted-only and compressed-plus-encrypted destinations, and an encrypted source verifies that source decryption also precedes hashing. + +### Compatibility proof + +For an uncompressed, unencrypted destination, NewPutObjReader initializes checksumReader and rawReader to the same reader, so the accessor change is behaviorally neutral. + +The patch introduced no new server API and no new storage marker. The production portion was limited to three files and about 30 additions / 15 deletions. The larger test file reflects the compatibility matrix, not runtime complexity. + +## Adversarial findings kept in separate fixes {#adjacent} + +The review intentionally tried to break the solution around its boundaries. It found two real inherited defects, both independent of the checksum placement. + +### Metadata-only transform state: #67 + +CopyObject derived destination compression metadata from current configuration before it knew whether object bytes would be rewritten. A metadata/reference-only self-copy could therefore add a compression marker to uncompressed data or remove the marker from compressed data. + +Versioned copies exposed a deeper edge: an unresolved source VersionID could make a metadata-only operation fall through to PutObject. Versioned SSE-C key rotation then wrote plaintext while preserving encryption metadata, producing sio: unsupported version. + +[PR #69](https://github.com/pgsty/silo/pull/69) fixed this separately by: + +- preserving source transform metadata for metadata/reference-only updates; +- changing compression markers only when bytes are actually rewritten; +- passing the resolved source version into versioned reference copies. + +Keeping this separate preserved #63's rollback boundary and prevented an apparently simple three-line guard from hiding the versioned corruption case. + +### CopyObjectResult checksum response: #68 + +After #63, the object stored and returned the correct checksum through HEAD and GET, but the successful CopyObject XML still contained only LastModified and ETag. + +[PR #70](https://github.com/pgsty/silo/pull/70) added the five checksum fields supported by this server plus ChecksumType, populated them from the committed destination ObjectInfo, and registered the exported fields in the compatibility baseline. + +Active minio-go already had checksum fields on UploadInfo but discarded CopyObjectResult values. [Upstream PR #2295](https://github.com/minio/minio-go/pull/2295) connects those existing fields without adding public API. + +## Verification evidence {#verification} + +The permanent suite covers: + +- CRC32, CRC32C, CRC64NVME, SHA1, and SHA256; +- explicit algorithms and default CRC64NVME; +- plain, compressed, encrypted-only, and compressed-plus-encrypted destinations; +- SSE-C and SSE-S3; +- encrypted and compressed sources; +- unversioned and versioned buckets; +- source full-object checksum preservation; +- multipart composite source conversion to a full-object checksum; +- in-place self-copy; +- empty data, exactly 4096 bytes, and 4097 bytes; +- the S2 compression-index path above 8 MiB; +- logical ETag and byte-for-byte body round trip; +- HEAD and GET with checksum mode enabled; +- missing and mismatched internal checksum results; +- absence of a published object after invariant failure. + +Validation gates included: + +~~~text +focused API tests +focused race tests +10 shuffled race iterations with GOMAXPROCS=8 +full go test ./cmd +CGO-disabled kqueue,dev cmd tests +go vet +golangci-lint +compatibility and rebrand guards +cross compilation +vulnerability analysis +release-pipeline snapshot, SBOM, provenance, package, and image validation +~~~ + +The regression is red on the unfixed baseline and green on the repaired tree. + +## Operational and rollout considerations {#operations} + +### Mixed server versions + +The metadata representation is unchanged, so an older node can read an object written with the corrected checksum. However, behavior during a rolling upgrade is request-node dependent: a CopyObject handled by an old node can still write the wrong value while a new node writes the correct value. + +Upgrade all API-serving nodes before treating CopyObject checksum behavior as stable. A successful local build or one upgraded node is not sufficient release evidence. + +### Existing objects + +The fix affects future copies. SILO does not automatically scan or rewrite historical checksum metadata because doing so would read and rewrite user data outside an explicit S3 operation. + +An object is a candidate for verification when: + +- it was created by CopyObject on an affected server; +- destination compression matched its key or content type; +- it carries an additional S3 checksum. + +Retrieve the checksum with checksum mode enabled, download the logical object, independently compute the named algorithm, and compare the Base64 value. + +For remediation, prefer copying to a new key with an explicit destination checksum algorithm and verifying the result before replacing the original. An in-place copy with x-amz-metadata-directive: REPLACE also rewrites the object, but replaces the current value in an unversioned bucket and creates a new version in a versioned bucket. Review retention, legal hold, tags, user metadata, encryption keys, capacity, replication, and rollback requirements before a bulk rewrite. + +### Release versus merge + +The server fixes and this design record are merged and the document is deployed. That does not identify the first released binary containing the changes. Release notes must name the eventual tag and independently verify archives, packages, container manifests, checksums, signatures, SBOMs, and provenance. + +## Cross-repository impact {#cross-repo} + +| Repository | Decision | +| --- | --- | +| pgsty/silo | Owns the handler, reader chain, object-layer invariant, response schema, and tests | +| minio/minio | Archived upstream retains the original defect; no normal upstream server PR is possible | +| minio/minio-go | PR #2295 returns CopyObject checksum fields through existing UploadInfo fields; maintainer merge pending | +| pgsty/silo-pkg | No change: it does not own ObjectInfo, PutObjReader, or CopyObjectHandler | +| pgsty/mc | No change: requesting --checksum deliberately disables server-side copy and uses download/upload | +| pgsty/silo-console | No direct change: it passes CopyObject through minio-go and does not interpret the checksum result | +| silo.pgsty.com | Owns this bilingual design, release boundary, and historical-object guidance | + +Legacy federated UploadPartCopy checksum recovery remains [issue #64](https://github.com/pgsty/silo/issues/64). It is a different API, response contract, and deployment topology and must not be presented as solved by this work. + +## Final outcome {#outcome} + +The repair is small because it does not invent a new checksum system. It makes an existing distinction explicit: + +~~~text +S3 checksum reader = logical object contract +storage reader = physical representation contract +~~~ + +Once those responsibilities are separated, compression, encryption, ETag, erasure coding, and metadata persistence can remain streaming and independently testable. That is why the solution fixes the demonstrated bug without trading it for extra I/O, unbounded buffering, a new disk format, or a second internal abstraction. diff --git a/content/blog/design/copyobject-checksum/index.zh.md b/content/blog/design/copyobject-checksum/index.zh.md index 8bcd56c9..93dfb5fb 100644 --- a/content/blog/design/copyobject-checksum/index.zh.md +++ b/content/blog/design/copyobject-checksum/index.zh.md @@ -4,71 +4,359 @@ date: 2026-08-24 lastmod: 2026-08-24 author: "冯若航" summary: > - 目标端启用压缩时，SILO 可能把 S2 存储流的 checksum 当成 CopyObject 逻辑对象 checksum 持久化。本文记录明文 reader 不变量、验证边界、独立后续修复与旧对象处理方法。 + 目标端启用压缩后，SILO 可能把 S2 存储流的 checksum 当成 CopyObject 逻辑对象 checksum 持久化。本文重建故障机理，比较被否决的修复方案，定义明文 reader 不变量，证明最终方案为什么成立，并记录发布与存量对象处置边界。 tags: [设计, S3, 兼容性, Checksum] weight: 10 draft: false url: "/zh/blog/design/copyobject-checksum/" --- -本文是 [SILO #63](https://github.com/pgsty/silo/issues/63) 的设计与验证归档。 +本文是 [SILO #63](https://github.com/pgsty/silo/issues/63) 的最终设计与验证归档。 -**状态：** checksum 数据域修复已通过 [PR #66](https://github.com/pgsty/silo/pull/66) 合并；公开发布待完成。 -**相关修复：** metadata-only transform state [#67](https://github.com/pgsty/silo/issues/67) 已通过 [PR #69](https://github.com/pgsty/silo/pull/69) 合并，CopyObjectResult checksum 字段 [#68](https://github.com/pgsty/silo/issues/68) 已通过 [PR #70](https://github.com/pgsty/silo/pull/70) 合并；公开发布仍待完成。 -**上游客户端：** [minio-go #2295](https://github.com/minio/minio-go/pull/2295)。 -**发布边界：** 合并不代表公开 release、软件包或容器镜像已经包含修复。 +**最终决策：** 所有由服务器生成的 CopyObject checksum 都必须在压缩和加密之前，基于目标对象的逻辑字节计算；逻辑 reader 与存储 reader 分离保存；如果读到 EOF 后仍拿不到预期 checksum，则拒绝发布对象。
+**实现：** [PR #66](https://github.com/pgsty/silo/pull/66)，合并提交为 c0e715977。
+**相关修复：** transform state 保真 [#67](https://github.com/pgsty/silo/issues/67) / [PR #69](https://github.com/pgsty/silo/pull/69)，CopyObjectResult checksum 字段 [#68](https://github.com/pgsty/silo/issues/68) / [PR #70](https://github.com/pgsty/silo/pull/70)。
+**上游客户端：** [minio-go PR #2295](https://github.com/minio/minio-go/pull/2295)。
+**发布边界：** 这些修改已经合并到源码，但本文不声称任何具体 release tag、RPM、DEB、APK、归档或容器镜像已经包含它们。 -## 缺陷本质 {#defect} +## 一句话决策 {#decision} -CopyObject 先把源对象读成逻辑数据，再按目标配置压缩并加密存储流。旧处理器把 server-side checksum 挂在了已经代表压缩字节的 reader 上： +checksum 不是“在写入链路上随便找个位置算出的 digest”。它一定是某段明确定义字节序列的函数。对 S3 CopyObject 而言，这段字节必须是客户端最终读到的逻辑对象，而不是 SILO 私有的压缩或加密表示。 - 逻辑对象 -> S2 压缩 -> checksum -> 可选加密 -> 存储 +因此最终采用的流水线是： -digest 在数学上有效，却覆盖了错误的数据域。客户端下载对象后对逻辑字节独立计算，结果自然不同。API 级复现是确定性的： +~~~text +逻辑源对象字节 + -> server-side S3 checksum + -> 可选 S2 压缩 + -> 存储流 hash 与 ETag 委托 + -> 可选服务端加密 + -> 纠删码 + -> EOF checksum 验证 + -> 数据与 metadata 原子提交 +~~~ - 修复前持久化 CRC32：hN7ytg== - 逻辑对象 CRC32： 1WxbLg== +整份设计都可以从这个顺序推出。 -当前基线实现的 CRC32、CRC32C、CRC64NVME、SHA1、SHA256 都会受影响。压缩叠加加密时，S2 加密流填充包含随机值，错误 checksum 甚至会变成非确定值。 +## 背景：一个对象存在多个完整性域 {#background} -## 采用的不变量 {#invariant} +SILO 中有若干值都会被口语化地称作 checksum，但它们保护的契约并不相同。 -逻辑 checksum reader 现在与存储变换 reader 分离： +| 值 | 字节域 | 用途 | +| --- | --- | --- | +| S3 additional checksum | 逻辑对象字节 | 通过 HEAD、GET、attributes 和复制响应提供客户端可见的端到端完整性 | +| ETag | 普通单 part、兼容未加密场景下代表逻辑内容；其他场景遵循各自协议语义 | 对象身份与条件请求兼容性 | +| 存储 reader 记账 | 压缩或加密后的写入流 | 在写入路径中传递 size、stream 与 ETag 委托 | +| 纠删码 bitrot checksum | 盘上纠删码 shard | 检测 SILO 物理表示损坏 | +| 加密认证 | 密文 framing 与密钥 | 检测篡改并认证加密存储 | +| 压缩 index | S2 存储流偏移 | 支持大压缩对象的高效读取 | - 逻辑对象 - -> server-side checksum - -> 可选 S2 压缩 - -> 存储流 hash - -> 可选服务端加密 - -> 纠删码与提交 +它们可以在同一遍流式写入中计算，却绝不能相互替代。存储流 checksum 完全可能在数学上正确，同时作为 S3 对象 checksum 完全错误。 -处理器必须在压缩 goroutine 启动前安装 hasher。即使压缩或加密替换活动存储 reader，PutObjReader 仍保存逻辑 reader。读到 EOF 后，对象层要求 checksum 存在、有效并与预期 base algorithm 一致，随后才允许提交 metadata。 +Amazon S3 明确规定 CopyObject 会产生目标 checksum；multipart 来源在一次 CopyObject 后会成为 full-object checksum。算法可能由请求显式选择、从来源继承，或在来源没有 checksum 时使用默认算法。无论哪一种，结果描述的都是复制后的对象，而不是供应商私有的存储编码。 -该设计直接复用 multipart checksum 的 checksumReader 契约，不创建第二套抽象，不重读对象，也不改变盘上格式。 +## #46 如何暴露 #63 {#discovery} -## 验证边界 {#verification} +这个缺陷是在修复 multipart checksum 兼容性 [#46](https://github.com/pgsty/silo/issues/46) 时发现的。 -永久 API 测试覆盖五种算法与默认 CRC64NVME；不压缩、压缩、仅加密和压缩加密；SSE-C 与 SSE-S3；加密源和压缩源；版本化桶；full 与 multipart-composite 来源；原地复制；零长度、压缩阈值与带索引 S2 流；ETag、正文 round trip、HEAD/GET checksum mode，以及内部不变量失败。 +#46 为 UploadPart 与 UploadPartCopy 建立了三条内部规则： -同一条回归在未修复基线上失败，在修复树上通过。合并前还要求定向 race、随机顺序重复运行、全量 cmd、禁用 CGO 的 kqueue/dev CI 形态、lint、vet、交叉编译、兼容性 guard 与远端 CI 全部通过。 +1. 为逻辑明文 checksum 保留专用 reader。 +2. fallback server hasher 必须由 handler 在压缩或加密消费数据前安装。 +3. 对象层负责验证并持久化完成后的结果，但不在对象层决定字节域。 -## 刻意拆开的邻接缺陷 {#adjacent} +由此引入的私有字段 checksumReader 刻意与活动 Reader、历史 rawReader 分开。WithEncryption 可以替换活动存储 reader，却不能替换逻辑 checksum reader。 -对抗审查又发现两个继承缺陷： +沿着普通 CopyObject 检查后，我们发现同一个概念风险出现在另一条 handler 中：代码先创建 newS2CompressReader，把它的输出包装成 srcInfo.Reader，之后才对这个 reader 调用 AddServerSideChecksumHasher。变量名掩盖了关键事实：此时的 srcInfo.Reader 已经代表存储流，不一定代表 S3 对象流。 -1. metadata/reference-only self-copy 可能在没有重写引用数据时改变压缩标记；版本化 SSE-C 密钥轮换还可能落入非法重写。该问题独立收敛在 [#67](https://github.com/pgsty/silo/issues/67)。 -2. 目标 checksum 已提交后，成功的 CopyObject XML 仍不返回 checksum 元素。服务端由 [#68](https://github.com/pgsty/silo/issues/68) 跟踪；minio-go 还会丢弃字段，对应 [#2295](https://github.com/minio/minio-go/pull/2295)。 +#46 刻意不修改 CopyObject。把 #63 单独拆出，意味着 P0 multipart 修复可以独立审查、发布或回滚，不必绑定另一套 API 与测试矩阵。 -旧 federation 的 UploadPartCopy checksum 恢复属于另一个 API，继续由 [#64](https://github.com/pgsty/silo/issues/64) 跟踪。 +## 故障模型 {#failure-model} -归档的上游 minio/minio 仍保留原始 reader 放置方式。silo-pkg 不拥有该 reader 链。MCLI 在指定 --checksum 时会从 server-side copy 切换为下载再上传，SILO Console 只通过 minio-go 透传 CopyObject，因此两者不需要复制一份服务端修复。 +### 旧顺序 -## 已存在对象 {#existing-objects} +旧流程的关键部分是： -修复只影响此后的 CopyObject，不会自动扫描或重写旧版本已经保存的 checksum metadata。 +~~~text +GetObject 逻辑 reader + -> 启动 S2 压缩 goroutine + -> 把压缩输出包装为 hash.Reader + -> 随后决定目标 checksum algorithm + -> 把 server-side hasher 挂到压缩 hash.Reader + -> 把结果当成对象 S3 checksum 持久化 +~~~ -由 CopyObject 创建、当时目标 key 或内容类型命中压缩配置、并带有额外 S3 checksum 的对象值得核验。使用 checksum mode 取回对象与 checksum，再用同一算法独立计算下载后的逻辑字节并比较 Base64 值。 +从存储 writer 的角度看，这个 checksum 并没有漏数据，也不一定损坏；它只是覆盖了错误但完整的流。 -修复时可显式指定 checksum algorithm，把对象复制到新 key。也可以带 x-amz-metadata-directive: REPLACE 做原地复制，但这会重写对象：未版本化桶替换当前值，版本化桶创建新版本。批量处理前必须确认 retention、legal hold、metadata、tag、加密密钥、剩余空间和回滚要求。 +### 静态假设与动态结果 -SILO 不做自动在线回填，因为那意味着在没有显式 S3 操作的情况下读取并重写用户数据。 +最初 Issue 提出了两种可能： + +- hasher 覆盖压缩数据； +- 压缩 goroutine 在 hasher 安装前已经消费部分逻辑输入，导致漏掉前缀。 + +API 复现确认了第一种，没有确认第二种。hasher 实际安装在压缩器输出侧 reader 上，因此会从该输出 reader 的起点观察完整 transformed stream。压缩输入侧提前消费的字节，并不是输出 hash reader 已经消费的字节。 + +这个区别很重要：根因不是“偶发 race，因此加一把锁即可”，而是确定性的数据域错误。 + +### 可重复复现 + +对于永久测试使用的同一份 payload，未修复树保存的是： + +~~~text +S2 存储字节 CRC32：hN7ytg== +逻辑对象 CRC32： 1WxbLg== +~~~ + +前者是合法 CRC32，所以普通 metadata 合法性检查无法发现。只有对下载后的逻辑对象独立计算，才会暴露不一致。 + +CRC32、CRC32C、CRC64NVME、SHA1、SHA256 在压缩目标上都会失败。当压缩与目标加密组合时，S2 会为加密流加入随机 padding，错误 checksum 不但错误，而且同一份逻辑复制多次可能得到不同结果。 + +## 需求与非目标 {#requirements} + +修复必须同时满足： + +1. **字节域正确。** 服务器生成的 checksum 精确覆盖目标逻辑字节。 +2. **单遍流式处理。** CopyObject 不能增加第二遍对象读取。 +3. **与变换无关。** 压缩和加密不能改变逻辑 checksum。 +4. **客户端兼容。** 既有客户端 checksum 校验与算法选择语义不变。 +5. **multipart 来源正确。** multipart composite 来源复制为单对象后，必须按 base algorithm 重算 full-object checksum。 +6. **默认行为正确。** 来源没有 checksum 时，目标仍获得当前基线的 S3 兼容默认 CRC64NVME。 +7. **ETag 不回归。** 移动 checksum reader 不能悄悄改变 CopyObject ETag 契约。 +8. **fail closed。** 内部调用者声明要生成 checksum 却没有生成时，不能返回成功并缺少完整性 metadata。 +9. **格式兼容。** 继续使用既有 checksum metadata 表示。 +10. **回滚边界小。** 不把响应 schema、federation 或 metadata-only transform 缺陷混入核心放置修复。 + +#63 明确不负责： + +- 新增 checksum 算法； +- 修改盘上 checksum 编码； +- 扫描或回填旧对象； +- 修复 legacy federated UploadPartCopy； +- 增加 CopyObjectResult XML 字段； +- 修改 MCLI 或 Console 行为。 + +## 备选方案与权衡 {#alternatives} + +| 方案 | 吸引力 | 否决原因 | +| --- | --- | --- | +| 在对象层安装 hasher | 所有调用者集中 fallback | 对象层拿到的是 handler 变换后的存储 reader，无法可靠重建逻辑字节域，而且安装时点可能过晚 | +| 对 S2 输出做 hash | 代码移动最少 | 这就是已经复现的缺陷：保护存储字节，不保护 S3 对象字节 | +| 对密文做 hash | 加密设置完成后最方便 | IV、framing、认证与 padding 让结果变成供应商私有值，且往往非确定 | +| 写完后重新读取对象 | 推理简单 | I/O 翻倍，破坏单遍流式目标，增加大对象和分层对象延迟 | +| 变换前缓存整个对象 | 顺序直观 | CopyObject 可处理大对象；整对象缓冲带来不可接受的内存与延迟 | +| 永远复制来源 checksum 值 | 避免计算 | 请求可能指定不同算法；来源可能没有 checksum；multipart composite 来源必须转为 full-object | +| 新建 CopyObject 专用 checksum 抽象 | 改动局部 | 重复 #46 已建立的不变量，未来形成两套略有差异的内部契约 | +| 在变换前复用逻辑 checksumReader | 单遍、复用既有格式、共享内部不变量 | 采用 | + +选择方案不只是“改动行数最少”，而是它用最少机制把字节域契约显式化，并且能够复用。 + +## 最终设计 {#design} + +### 1. 先构造逻辑 reader + +CopyObject 取得的源 GetObjectReader 已经输出逻辑源对象：盘上压缩已经解码，来源加密也已经通过授权的 source options 解开。 + +SILO 用已知逻辑对象大小把它包装成逻辑 hash.Reader。对压缩目标而言，这还把过去的无限长度收紧为压缩前的逻辑长度硬上限。 + +此时压缩 goroutine 尚未启动。 + +### 2. 决定目标 checksum 策略 + +既有策略保持不变： + +1. 请求包含 x-amz-checksum-algorithm 时，计算对应 base algorithm。 +2. 否则检查来源 checksum。 +3. 来源是 full-object checksum 时，因为逻辑字节不变，可以保留该值。 +4. 来源是 multipart composite 时，因为 CopyObject 产生单次 full object，必须用 base algorithm 重算。 +5. 来源没有 checksum metadata 时，目标获得默认 CRC64NVME。 + +只有真正需要计算的分支才调用 AddServerSideChecksumHasher。 + +### 3. hasher 安装后再启动压缩 + +目标需要压缩时，逻辑 reader 被捕获为 checksumReader，并作为 newS2CompressReader 的输入。压缩器无法得到任何一个字节，除非该字节先通过逻辑 hasher。 + +压缩输出拥有独立的存储 hash.Reader。新的 PutObjReader 以存储 reader 为活动 reader，再通过 setChecksumReader 保存逻辑 reader 引用。 + +~~~text +PutObjReader.Reader = 压缩或加密后的存储流 +PutObjReader.rawReader = 历史 ETag 路径使用的流 +PutObjReader.checksumReader = 逻辑明文流 +~~~ + +不需要新增导出方法或 package 级抽象。 + +### 4. 加密阶段继续保持分离 + +目标加密会包装压缩存储 reader，并可能通过 WithEncryption 替换 PutObjReader.Reader，但不会修改 checksumReader。 + +因此以下四种存储方式必须得到同一逻辑 checksum： + +- 明文； +- 压缩； +- 加密； +- 压缩加密。 + +checksum metadata 本身需要保护时，既有 metadata encryption function 会在计算完成后加密序列化结果。这保护的是 metadata at rest，不会改变被 hash 的字节。 + +### 5. EOF 后完成并 fail closed + +内部 hash reader 只在来源返回 EOF 后设置 ServerSideChecksumResult。压缩路径中，io.Copy 必须先排空逻辑 checksum reader，随后压缩器才能关闭 pipe。对象 writer 不可能在逻辑 reader 完成 checksum 前观察到压缩流 EOF。 + +对象层随后验证： + +- 结果存在； +- 结果结构有效； +- base algorithm 与 WantServerSideChecksumType 一致。 + +失败时记录内部不变量错误并中止写入。延迟纠删码清理会在唯一 metadata 发布前删除临时 shard。显式或默认要求 checksum 的操作若返回 HTTP 200 却没有 checksum，是静默正确性损失，不能作为 fallback。 + +### 6. 不改变盘上格式 + +验证后的 checksum 继续追加到既有 FileInfo.Checksum 表示。加密目标复用现有 metadata encrypter。HEAD、GET、GetObjectAttributes、复制 metadata 与后续 reader 仍消费同一种表示。 + +## 为什么这个设计一定成立 {#proof} + +### 字节域证明 + +压缩器接受的每一个字节都来自 checksumReader。hasher 在压缩器构造前安装，所以 digest 输入精确等于压缩器的逻辑输入，而不是输出。 + +### 完整性证明 + +压缩器只有在排空逻辑输入并关闭 S2 writer 后，才能关闭输出。对象 writer 必须把输出读到 EOF 才能完成写入。checksum 在逻辑输入 EOF 时完成，而这一时刻先于存储侧可见 EOF。 + +pipe 自然建立 happens-before 关系，不需要额外 mutex 或旁路信号。定向 race 与随机顺序重复执行验证了实现。 + +### ETag 证明 + +压缩 reader 把逻辑 reader 作为 ETag delegate。移动 S3 checksum hasher 不会把 ETag 计算迁到 S2 字节上。永久测试在兼容的未加密场景中，把最终 ETag 与逻辑对象 MD5 独立比较。 + +### 存储完整性证明 + +压缩后仍保留存储侧 reader，用于物理流记账与 ETag 委托；纠删码层继续为盘上 shard 写入自己的 bitrot protection。二者都没有被 S3 逻辑 checksum 取代，S3 checksum 也不会被冒充成 shard 完整性机制。 + +### 加密证明 + +加密 reader 在逻辑 checksum 之后消费存储流。随机 IV、framing 或 padding 无法影响 checksum。SSE-C 与 SSE-S3 测试同时覆盖仅加密、压缩加密目标；加密来源测试证明 source decryption 也发生在 hash 之前。 + +### 兼容性证明 + +对不压缩、不加密的目标，NewPutObjReader 会把 checksumReader 与 rawReader 初始化为同一个 reader，因此 accessor 调整在行为上不变。 + +补丁没有新增服务端 API，也没有新增存储 marker。生产修改只涉及三个文件，约 30 行新增、15 行删除。较大的测试文件反映兼容性矩阵，不是运行时复杂度。 + +## 对抗审查拆出的独立修复 {#adjacent} + +审查刻意尝试从边界击穿方案，发现了两个真实继承缺陷，但都不属于 checksum 放置本身。 + +### Metadata-only transform state：#67 + +CopyObject 在确认是否重写对象字节前，就按当前目标配置推导压缩 metadata。metadata/reference-only self-copy 因而可能给未压缩数据增加压缩标记，或从压缩数据删除标记。 + +版本化路径还暴露了更深边界：来源 VersionID 未解析时，metadata-only 操作可能落入 PutObject。版本化 SSE-C 密钥轮换随即会写入明文却保留加密 metadata，后续 GET 报 sio: unsupported version。 + +[PR #69](https://github.com/pgsty/silo/pull/69) 单独修复： + +- metadata/reference-only 更新保留来源 transform metadata； +- 只有真实重写字节时才改变压缩标记； +- 版本化 reference copy 使用已经解析的来源版本。 + +这个问题保持独立，既保护 #63 回滚边界，也避免用一个表面上的三行 guard 掩盖版本化损坏。 + +### CopyObjectResult checksum 响应：#68 + +#63 完成后，对象通过 HEAD 与 GET 返回正确 checksum，但成功 CopyObject XML 仍只有 LastModified 与 ETag。 + +[PR #70](https://github.com/pgsty/silo/pull/70) 增加当前服务端支持的五种 checksum 字段与 ChecksumType，从已提交的目标 ObjectInfo 填充，并把新增导出字段登记到 compatibility baseline。 + +活跃的 minio-go 已经在 UploadInfo 上拥有 checksum 字段，却会丢弃 CopyObjectResult 值。[上游 PR #2295](https://github.com/minio/minio-go/pull/2295) 连接这些已有字段，不新增公共 API。 + +## 验证证据 {#verification} + +永久测试覆盖： + +- CRC32、CRC32C、CRC64NVME、SHA1、SHA256； +- 显式算法与默认 CRC64NVME； +- 明文、压缩、仅加密、压缩加密目标； +- SSE-C 与 SSE-S3； +- 加密来源与压缩来源； +- 未版本化与版本化桶； +- 来源 full-object checksum 保留； +- multipart composite 来源转换为 full-object checksum； +- 原地 self-copy； +- 空数据、精确 4096 字节、4097 字节； +- 超过 8 MiB 的 S2 compression-index 路径； +- 逻辑 ETag 与逐字节正文 round trip； +- 启用 checksum mode 的 HEAD 与 GET； +- 内部 checksum 缺失与算法失配； +- 不变量失败后对象没有发布。 + +验证门禁包括： + +~~~text +聚焦 API 测试 +聚焦 race 测试 +GOMAXPROCS=8 下 10 轮随机顺序 race +全量 go test ./cmd +禁用 CGO 的 kqueue,dev cmd 测试 +go vet +golangci-lint +compatibility 与 rebrand guard +交叉编译 +漏洞分析 +release-pipeline 快照、SBOM、provenance、包与镜像验证 +~~~ + +同一条回归在未修复基线上为红，在修复树上为绿。 + +## 运维与发布考虑 {#operations} + +### 混合服务器版本 + +metadata 表示没有变化，所以旧节点能够读取新节点写入的正确 checksum。但滚动升级期间，CopyObject 行为取决于实际处理请求的节点：旧节点仍可能写入错误值，新节点写入正确值。 + +所有承载 API 的节点都升级后，才能把 CopyObject checksum 行为视为稳定。一次本地构建成功或只升级一个节点，都不构成发布证据。 + +### 已存在对象 + +修复只影响此后的复制。SILO 不自动扫描或重写历史 checksum metadata，因为那意味着在没有显式 S3 操作的情况下读取并重写用户数据。 + +同时满足以下条件的对象值得核验： + +- 由受影响版本的 CopyObject 创建； +- 当时目标 key 或内容类型命中压缩配置； +- 对象带有额外 S3 checksum。 + +使用 checksum mode 取回 checksum，下载逻辑对象，用同一算法独立计算，再比较 Base64 值。 + +修复时优先把对象复制到新 key，显式指定目标 checksum algorithm，验证完成后再替换原对象。也可以带 x-amz-metadata-directive: REPLACE 做原地复制，但未版本化桶会替换当前值，版本化桶会创建新版本。批量重写前必须确认 retention、legal hold、tag、用户 metadata、加密密钥、容量、复制与回滚要求。 + +### 合并不等于发布 + +服务端修复与本文已经合并，文档也已部署，但这仍不能回答“第一个包含修复的二进制版本是什么”。最终 release note 必须明确具体 tag，并独立验证归档、软件包、容器 manifest、checksum、签名、SBOM 与 provenance。 + +## 跨仓库影响 {#cross-repo} + +| 仓库 | 决策 | +| --- | --- | +| pgsty/silo | 拥有 handler、reader 链、对象层不变量、响应 schema 与测试 | +| minio/minio | 已归档上游仍保留原缺陷，无法正常提交服务端上游 PR | +| minio/minio-go | PR #2295 通过已有 UploadInfo 字段返回 CopyObject checksum；等待 maintainer 合并 | +| pgsty/silo-pkg | 无需修改：不拥有 ObjectInfo、PutObjReader 或 CopyObjectHandler | +| pgsty/mc | 无需修改：指定 --checksum 时会主动禁用 server-side copy，改用下载再上传 | +| pgsty/silo-console | 无需直接修改：通过 minio-go 透传 CopyObject，且不解释 checksum 结果 | +| silo.pgsty.com | 负责本文双语设计、发布边界与历史对象指导 | + +legacy federated UploadPartCopy checksum 恢复仍由 [#64](https://github.com/pgsty/silo/issues/64) 跟踪。它属于不同 API、响应契约与部署拓扑，绝不能声称已由本次工作解决。 + +## 最终结果 {#outcome} + +修复之所以小，不是因为它省略了问题，而是因为它没有发明新的 checksum 系统，只把原本隐含的职责差异显式化： + +~~~text +S3 checksum reader = 逻辑对象契约 +storage reader = 物理表示契约 +~~~ + +一旦二者分离，压缩、加密、ETag、纠删码与 metadata 持久化都可以继续保持流式、独立验证。这就是该方案能够解决已复现缺陷，同时不换来额外 I/O、无界缓冲、新盘上格式或第二套内部抽象的原因。