Skip to content

Add option to read more than one line at a time #13

Description

@mifrandir

Currently, the program is bottlenecked by the fact that we are only reading one line at a time.

I might lead to serious performance gains if we limited the number of I/O operations (especially writes) by reading a variable number of lines. This number could be

  1. specified by the user,
  2. inferred from a given memory target and a line length heuristic, or
  3. inferred from a memory limit and actual line lengths and therefore dynamically change per batch.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions