Skip to content

feat: add P2P model checkpoint sharing over IPFS - #1461

Open
AaryanVerma17 wants to merge 8 commits into
libp2p:mainfrom
AaryanVerma17:issue-23-ipfs-checkpoints
Open

feat: add P2P model checkpoint sharing over IPFS#1461
AaryanVerma17 wants to merge 8 commits into
libp2p:mainfrom
AaryanVerma17:issue-23-ipfs-checkpoints

Conversation

@AaryanVerma17

Copy link
Copy Markdown

Description

This PR implements Issue #23: P2P Model Checkpoint Sharing over IPFS + libp2p.

The implementation provides persistent model checkpoint storage and retrieval through IPFS while using libp2p for peer-to-peer communication and CID exchange. This enables peers to synchronize and restore model checkpoints even after being temporarily offline.

Features

  • Model checkpoint creation and persistence
  • Local checkpoint save/load functionality
  • Upload model checkpoints to IPFS
  • Retrieve checkpoints from IPFS using CIDs
  • CID exchange between peers using py-libp2p
  • P2P checkpoint synchronization
  • Model restoration from downloaded checkpoints
  • Checkpoint version/round tracking
  • Offline peer recovery and synchronization
  • Integration with the federated learning workflow
  • Tests for checkpoint storage, retrieval, and synchronization
  • Documentation and example usage

Workflow

Peer A
   │
   ▼
Train Model
   │
   ▼
Create Checkpoint
   │
   ▼
Upload Checkpoint to IPFS
   │
   ▼
Generate CID
   │
   ▼
Share CID via libp2p
   │
   ▼
Peer B
   │
   ▼
Receive CID
   │
   ▼
Retrieve Checkpoint from IPFS
   │
   ▼
Load & Restore Model

Offline Peer Synchronization

A peer that was temporarily offline can rejoin the network and retrieve an available model checkpoint using its CID, allowing it to synchronize with the latest checkpoint.

Testing

The implementation includes tests covering:

  • Checkpoint creation
  • Checkpoint persistence
  • IPFS storage and retrieval
  • CID handling
  • Model restoration
  • P2P checkpoint synchronization
  • Offline peer recovery

Related Issue

Implements #23

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants