Skip to content

[roadmap] Streaming query for large JSON files #4

Description

@Cryptoteep

Goal

Currently query() loads the entire document into memory. For very large JSON files (logs, exports), we should support a streaming query mode that parses incrementally and yields matches as they are found.

Proposed API

import { queryStream } from 'jsoncraft';
import { createReadStream } from 'node:fs';

for await (const match of queryStream(createReadStream('big.json'), '$..events[?(@.level == "error")]')) {
  console.log(match);
}

Implementation notes

  • A streaming JSON parser (SAX-style) would be needed. To preserve the zero-dependency core, this could live in a separate jsoncraft/stream entry point.
  • The query engine already walks the tree recursively; adapting it to emit matches from a streaming parser is the main work.

Acceptance

  • queryStream() works on Node.js readable streams
  • Memory usage stays flat regardless of file size
  • Supports the same JSONPath subset as query()

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requesthelp wantedExtra attention is needed

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions