feat: implement DCUtR hole punching, file chunking, and resumable transfers - #7
Open
adarshxsh wants to merge 25 commits into
Open
feat: implement DCUtR hole punching, file chunking, and resumable transfers#7adarshxsh wants to merge 25 commits into
adarshxsh wants to merge 25 commits into
Conversation
added 22 commits
July 7, 2026 18:53
…O_RESERVATION error
- Establish decoupled pipeline architecture for Content Engine - Implement stream splitting (chunking) based on dynamic size - Implement XChaCha20-Poly1305 for independent chunk encryption - Implement SHA-256 Digest for content hashing and integrity checks - Create local FS ChunkSource and ChunkSink - Generate immutable Manifest containing chunk IDs and metadata - Create cmd/content-test for manual end-to-end testing - Update documentation and test matrix
Content Engine : Chunking and Encryption
This commit officially bridges the decoupled Content Engine to the libp2p network layer via the new /cipher/chunk/1.0.0 protocol. Key Highlights: - Implemented a unified, symmetric Message envelope (Version, Type, Payload) to eliminate silent stream failures. - Stream Handler: Implemented sequential stream routing that queries the local ContentEngine for Manifests and Chunks. - Client Downloader: Enforces strict Verify-Then-Store semantics (SHA-256 verification of ciphertext) before committing any chunk to the CAS store and issuing an ACK. - Decoupled Crypto: Reassembly tests confirm that network transmission (Manifest) is successfully decoupled from decryption rights (Key). - Benchmarking: Base transport pipeline benchmarks at ~490 MB/s over local loopback using strict sequential streaming. - CLI: Updated cmd/peer to support new '-ingest', '-fetch', '-key', and '-reassemble' flags. Deprecated the legacy /cipher/filetransfer/1.0.0 protocol. - Docs: Frozen protocol/chunk.proto.md spec and migrated to internal/protocol/chunk/. Updated architecture and testing matrices.
* feat(transport): Milestone 9 resumable transfers * docs: update architecture and testing matrices for Milestone 9
Member
|
adarshxsh
force-pushed
the
main
branch
5 times, most recently
from
July 17, 2026 15:35
f488993 to
1709f19
Compare
- Added libp2p Circuit Relay v2 support and connection protection - Enabled AutoNAT service and forced Private Reachability for hole punching - Added chaos testing flags - Updated testing matrix with long-distance Azure VM hole punching validation
added 2 commits
July 17, 2026 22:22
- Reduced chunk size from 256KB to 32KB for better streaming and deduplication - Replaced XChaCha20 (random 24-byte nonces) with ChaCha20 (deterministic 12-byte nonces) - Derived nonces from chunk index to remove random generation overhead - Updated ChunkHeader and manifest CryptoDescriptor accordingly
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This pull request introduces the core public milestone release of the CIPHER P2P Transport module, completely overhauling the codebase to support robust peer-to-peer connection establishment and resilient, large-scale file transfers.
At the connection layer, this PR implements Direct Connection Upgrade through Relay (DCUtR) and abstracts the transport layer. This enables peers to successfully dial each other and perform hole punching through NATs, securely falling back to relay nodes when direct connections are not immediately possible.
At the data transfer layer, this PR introduces a robust Content Engine featuring file chunking, CAS (Content-Addressable Storage) sharding, and resumable downloads. Transfers are now resilient to network interruptions; if a connection drops, the download can seamlessly resume from the last verified chunk. Additionally, a CLI download progress bar has been added for improved user experience.
Fixes # (issue)
Type of change
How Has This Been Tested?
Extensive manual testing and simulated network configurations were used to verify connection behavior and data integrity:
Checklist: