Skip to content

Efficiency: quantize stored embeddings to int8 and shrink the index #175

Description

@CryptoJones

The derived index is ~19 MB for 5,678 chunks, almost all of it float32 vectors.

int8 with a per-vector scale is standard practice among the low-storage retrievers surveyed: ~4x smaller file, faster matmul, negligible ranking loss at this scale. Add a residual-norm tiebreaker for near-equal cosines.

Backlog: BACKLOG.md → Open → Efficiency.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions