gRPC Large File Streaming with Go

A demonstration of how to efficiently stream large files over gRPC using Go. This project shows a practical implementation of client-streaming RPCs for file upload, addressing the challenges of transferring large files in a network-friendly way.

Why Stream Large Files?

When uploading large files, sending the entire file as a single request can be inefficient and risky:

• Memory pressure: Loading a multi-GB file into memory before sending causes memory spikes
• Network resilience: A single failed request means restarting the entire upload
• Poor user experience: No progress feedback or cancellation capability
• Server scalability: Keeping many large payloads in memory limits concurrent uploads

Streaming breaks the file into smaller chunks sent over a persistent connection, allowing:

• Constant memory usage regardless of file size
• Chunk-level error handling and retry logic
• Real-time progress tracking
• Better resource utilization on both client and server

How It Works

This project implements client-streaming RPC — the client opens a bidirectional stream with the server and sends multiple messages over a single connection. The server accumulates all chunks and responds once the stream ends.

Client                                  Server
  |                                        |
  |--- UploadRequest (chunk 1) ----------->|
  |                                        |
  |--- UploadRequest (chunk 2) ----------->|
  |                                        |
  |--- UploadRequest (chunk 3) ----------->|
  |                                    (accumulate)
  |--- (close stream) -------EOF-------->|
  |                                        |
  |<-------- UploadResponse (result) ------|
  |

The .proto file defines the streaming contract, and generated Go code handles all the gRPC machinery automatically.

Project Structure

.
├── proto/
│   ├── file.proto                 # Protocol buffer definition
│   ├── file.pb.go                 # Generated message code (protoc-gen-go)
│   └── file_grpc.pb.go            # Generated gRPC service code (protoc-gen-go-grpc)
├── server/
│   └── server.go                  # gRPC server implementation
├── client/
│   └── client.go                  # gRPC client implementation
├── api/
│   └── main.go                    # REST API reference implementation (optional)
├── go.mod                         # Module dependencies
└── go.sum                         # Dependency checksums

Prerequisites

• Go 1.26.5 or later
• protoc (protocol buffer compiler)
• protoc-gen-go (Go code generator plugin)
• protoc-gen-go-grpc (gRPC code generator plugin)

Install protoc and plugins

macOS (using Homebrew):

brew install protobuf
go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest

Linux (Ubuntu/Debian):

sudo apt-get install protobuf-compiler
go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest

Windows: Download the latest protoc release and add it to your PATH, then:

go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest

Setup

1. Clone the repository

git clone https://github.com/niloy104/Go-gRPC-Large-File-Streaming.git
cd Go-gRPC-Large-File-Streaming

2. Generate gRPC code from .proto file

This step only needs to be repeated if you modify proto/file.proto:

protoc --go_out=. --go-grpc_out=. proto/file.proto

This command generates:

• proto/file.pb.go — Message definitions (UploadRequest, UploadResponse)
• proto/file_grpc.pb.go — Service interfaces and client/server stubs

3. Install dependencies

go mod download

Running the Demo

Start the gRPC server

In one terminal:

go run server/server.go

The server listens on localhost:9000.

Start the gRPC client

In another terminal, first create a test file:

# Create a test file (e.g., 100 MB)
dd if=/dev/zero of=test.bin bs=1M count=100

Then run the client:

go run client/client.go

The client will:

1. Open a stream to the server
2. Read the test file in 2 MB chunks
3. Send each chunk via gRPC
4. Print progress for each batch
5. Receive a final response with total bytes transferred and status

Expected output (client side):

2024/08/15 14:22:01 Sent - batch #1 - size -2097152
2024/08/15 14:22:01 Sent - batch #2 - size -2097152
2024/08/15 14:22:01 Sent - batch #3 - size -2097152
...
2.234567s
2024/08/15 14:22:03 Sent - 104857600 bytes - File written successfully

The server writes the streamed file to ./demo.bin.

Understanding the Code

Protocol Buffer Definition (proto/file.proto)

message UploadRequest {
    string FilePath = 1;
    bytes chunks = 2;
}

message UploadResponse {
    int64 FileSize = 1;
    string message = 2;
}

service StreamUpload {
    rpc Upload(stream UploadRequest) returns (UploadResponse);
}

• stream UploadRequest: Client sends multiple UploadRequest messages
• UploadResponse: Server sends a single response after the stream closes
• FilePath: Original file path (useful for multi-file uploads)
• chunks: Binary data of each chunk
• FileSize: Total bytes received by the server
• message: Status message from server

Server Implementation (server/server.go)

func (s server) Upload(stream proto.StreamUpload_UploadServer) error {
    var fileBytes []byte
    var fileSize int64 = 0

    // Receive chunks until client closes the stream
    for {
        req, err := stream.Recv()
        if err == io.EOF {
            break  // Stream closed by client
        }
        
        chunks := req.GetChunks()
        fileBytes = append(fileBytes, chunks...)  // Accumulate chunks
        fileSize += int64(len(chunks))
    }

    // Write accumulated data to disk
    f, err := os.Create("./demo.bin")
    if err != nil {
        return err
    }
    defer f.Close()
    
    _, err = f.Write(fileBytes)
    if err != nil {
        return err
    }

    // Send response and close stream
    return stream.SendAndClose(&proto.UploadResponse{
        FileSize: fileSize,
        Message: "File written successfully",
    })
}

Key points:

• stream.Recv() blocks until the client sends a chunk (or closes the stream)
• io.EOF indicates the client closed their send side
• stream.SendAndClose() sends the response and closes the server's send side
• All errors are propagated to the client via gRPC status codes

Client Implementation (client/client.go)

func uploadStreamFile(path string, batchsize int) {
    file, err := os.Open(path)
    if err != nil {
        panic(err)
    }

    buff := make([]byte, batchsize)  // 2 MB chunks
    stream, err := client.Upload(context.TODO())
    if err != nil {
        panic(err)
    }

    for {
        num, err := file.Read(buff)
        if err == io.EOF {
            break  // Finished reading file
        }
        
        chunk := buff[:num]
        
        // Send chunk to server
        if err := stream.Send(&proto.UploadRequest{
            FilePath: path,
            Chunks:   chunk,
        }); err != nil {
            return
        }
        
        log.Printf("Sent - batch #%v - size -%v\n", batchNumber, len(chunk))
    }

    // Close client send side and wait for server response
    res, err := stream.CloseAndRecv()
    if err != nil {
        return
    }

    log.Printf("Sent - %v bytes - %s\n", res.GetFileSize(), res.GetMessage())
}

Key points:

• client.Upload(ctx) opens a new client stream
• stream.Send() sends a chunk (non-blocking)
• stream.CloseAndRecv() closes the client's send side and waits for the server's response
• Error handling at the chunk level allows partial retries

REST API Reference (api/main.go)

For comparison, this package includes a simple REST API using Gin that accepts file uploads via multipart form data. It listens on port 9001 and saves uploaded files to the uploads/ directory. This demonstrates the traditional REST approach that this gRPC implementation improves upon.

Large Files and .gitignore

The .gitignore file excludes *.bin files to avoid committing large test files to Git. When you run the demo:

• You create test.bin locally (outside of version control)
• The server writes demo.bin locally (outside of version control)

If you need to test with different file sizes, generate them with dd:

# 100 MB
dd if=/dev/zero of=test.bin bs=1M count=100

# 1 GB
dd if=/dev/zero of=test.bin bs=1G count=1

# 5 GB
dd if=/dev/zero of=test.bin bs=1G count=5 bs=1M

Note: Memory usage remains constant (~2 MB for the buffer) regardless of file size, thanks to streaming.

What I Learned

This project demonstrates several important gRPC patterns:

1. Client-Streaming RPC: One-way message flow from client to server, with a final server response. Useful for bulk operations.

2. Memory Efficiency: Streaming keeps memory usage low and predictable. Compare to loading the entire file in memory at once.

3. Error Handling: gRPC status codes allow the server to reject uploads mid-stream, and clients can handle errors per chunk.

4. Protocol Buffers: The .proto file is the contract between client and server. Protoc generates all serialization/deserialization code automatically.

5. Chunk Size Trade-off: Larger chunks (2 MB here) reduce overhead but increase latency if a retry is needed. Smaller chunks increase overhead but enable finer-grained progress.

6. Graceful Shutdown: The server keeps receiving until io.EOF, then processes the accumulated data. The client uses CloseAndRecv() to ensure the server responds.

Possible Improvements

• Resumable uploads: Track chunk indices so failed transfers can resume from a specific point instead of restarting.
• Chunk verification: Include MD5 or SHA256 checksums of chunks so the server can detect corruption.
• Disk streaming: Instead of accumulating all bytes in memory, write chunks directly to disk as they arrive.
• Dynamic chunk sizing: Adjust batch size based on network conditions (RTT, packet loss).
• Bidirectional feedback: Use streaming in both directions for progress updates and cancellation signals.
• Timeout and deadline handling: Add context deadlines and cancellation to prevent hanging connections.
• Concurrency: Handle multiple simultaneous uploads with goroutines.
• Compression: Use gRPC's built-in compression (gzip) to reduce bandwidth.

References

• gRPC Go documentation
• Protocol Buffers language guide
• gRPC streaming patterns
• Go io package