fix(ingestion): write the raw zone as parquet - #851
Merged
Conversation
The first live materialization landed .jsonl.gz. dlt's filesystem destination writes gzipped JSONL unless given a loader_file_format, and it never was -- while the plan, the source doc, and Mode B replay all assume parquet. Replay reads the raw zone filtered on event time. A columnar format with real types lets it read a window without decompressing and parsing every record, and it round-trips the difference between a null and a missing field more reliably than JSONL. pyarrow was already a dependency, added for this. Objects already written stay JSONL. dlt reads both, so they need no migration, but a replay spanning the boundary reads two formats -- noted in the source doc rather than left to be discovered. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
Coverage✅ 79.83% total — gate is 75%. Coverage for the Python files changed in this PR
|
Contributor
|
Your pull request is automatically being deployed to Dagster Cloud.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The first live materialization worked — data landed date-partitioned in the raw
zone — but as
.jsonl.gz, not parquet:dlt's filesystem destination writes gzipped JSONL unless given a
loader_file_format, and it never was — while the plan,docs/sources/san_acacia.md,and Mode B replay all assume parquet.
pyarrowwas already a dependency, addedfor exactly this.
Why it matters
Replay reads the raw zone filtered on event time. A columnar format with real
types lets it read a window without decompressing and parsing every record, and
it round-trips the distinction between a null and a missing field more reliably
than JSONL.
Existing objects
Stay JSONL. dlt reads both formats, so they need no migration — but a replay
spanning the boundary reads two formats, which is now noted in the source doc
rather than left to be discovered by whoever writes 4.3.
81 tests pass.
🤖 Generated with Claude Code