Repository navigation
feat(cli): explore, a data-first view of a file for business users - #608
Merged
Merged
Conversation
ByteSize stopped at MB, so a 2.3 GB file read as "2179.1 MB" in inspect, tui, import, export and explore output. Rather than add one more tier, derive the unit from the count's highest set bit (each unit is 10 more bits), with units up to EB: a long tops out just under 8 EiB, so no byte count can outgrow the list again. Units stay binary, like the KB and MB ones: that file now prints as 2.1 GB, and Long.MAX_VALUE as 8.0 EB. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SwhK4W5cgj7y7s4ZPCif5x
Each chunk's embedded array stats carry what a vortex.zoned zone map does not: exact string min/max (the zone map only bounds them) and, where the writer records it, a sum. Tooling outside the reader (the explore statistics table) needs them, so the method becomes public. A dictionary chunk now reports only its values pool's min/max. The pool's sum and null count describe the dictionary, not the chunk's rows; no caller read them before, but a public method must not hand them out as the chunk's. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SwhK4W5cgj7y7s4ZPCif5x
`inspect` answers how a file is stored; `explore` answers what is in it. explore [--serve] <file|url> serves a page on 127.0.0.1, under a random path (as Jupyter does), and opens it in the browser. It leads with one row per column, folded from the statistics the file already records, so nothing is decoded and a 2.3 GB, 100M-row file opens in well under a second: null %, exact min/max, mean where the file records a sum, and a sparkline of each zone's min-max band over the rows (sorted, clustered or random at a glance). Analyze on a column reads its values once for the count, null %, total, mean, distinct count, string byte lengths, and its spread: every value with its share up to 30 distinct values, else a 30-bin histogram for a numeric, date or timestamp column (bins from the recorded min/max, or a min/max pass when the file has none), else the 10 most frequent values, and nothing past 10,000 distinct. Totals reuse the fused aggregate behind Calcite's SUM push-down and value counts its GROUP BY fold, both working off dictionary codes. Columns are analyzed in parallel. A reader's arena is confined to its thread, so the file is opened once per core, each on its own IoWorker, and the server answers on virtual threads. Analyze all on that 100M-row file takes 2.2 s on 10 cores against 9-11 s one column at a time. explore --html out.html <file|url> analyzes every column up front and writes the page as one self-contained file to publish: 1.5 s and 111 KB for the January 2024 NYC taxi file (3M rows, 19 columns). The data page lives in ExplorePage, apart from the storage-first HtmlReport, which is unchanged (inspect --html keeps its output). vortex-jni cross-checks: a Rust-written file's profile, total and histogram match the data written, and a dictionary column's value counts are exact. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SwhK4W5cgj7y7s4ZPCif5x
dfa1
force-pushed
the
feat/vortex-explore
branch
from
October 25, 2026 07:40
5993415 to
b456d5e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
inspectanswers how a file is stored;exploreanswers what is in it.What it shows
How
SUMpush-down and value counts itsGROUP BYfold, both off dictionary codes.IoWorker, and the server answers on virtual threads. Analyze all on the 100M-row file: 2.2 s on 10 cores vs 9-11 s one column at a time.--servebinds127.0.0.1only, under a random path (as Jupyter does).--html: January 2024 NYC taxi (3M rows, 19 columns) in 1.5 s, 111 KB.ExplorePage;HtmlReport(inspect --html) is unchanged.Commits
fix(inspector): sizes past 1 GiB printed as e.g.2179.1 MB; units now derive from the highest set bit, up to EB (Long.MAX_VALUE→8.0 EB).feat(reader):ScanIterator.chunkStatspublic; a dict chunk reports only its pool's min/max.feat(cli):explore.Tests
Unit tests for folding, chart choice, escaping and the server's paths/concurrency; vortex-jni cross-checks: a Rust-written file's profile, total and histogram match the data written, and a dict column's value counts are exact.
🤖 Generated with Claude Code
https://claude.ai/code/session_01SwhK4W5cgj7y7s4ZPCif5x