Skip to content

Commit 181b9fa

Browse files
author
Paul Meng
committed
Document measured milestones and add reproducible architecture experiments
1 parent b926318 commit 181b9fa

12 files changed

Lines changed: 1433 additions & 10 deletions

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@
88
- Keep SQL constants out of the source-field namespace, apply FROM aliases to stdin consistently with files, and fix duplicate-alias full-sort/Top-N binding and a Top-N panic. Accept following clauses after IN, postfix and CAST expressions.
99
- Add a same-engine fresh/reused-plan diagnostic and a bounded fixed-schema external-sort example with temporary-disk quotas and exact-output validation. Retain both as experiments: measurements do not justify a production session API, and production spilling still needs lossless dynamic-value storage and shared query accounting.
1010
- Add generated cross-path correctness checks and a generated conformance index checked in CI. Runtime numbers remain Int32/Float32.
11+
- Publish reproducible CLI, lifecycle, kernel and external-sort runners with independent answer checks, immutable manifests and archived source/binary identities. Record the completed plan, [measured adoption decisions](docs/performance-execution-2026-09-06.md), and a proposed [Int64/Float64 compatibility contract](docs/numeric-migration.md).
1112

1213
## Unreleased — correctness, performance, and documentation audit (2026-09-06)
1314

‎README.md‎

Lines changed: 13 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,8 @@ logq's streaming queries used 8–9 MiB peak RSS in that run. Subsequent executi
2121
changes are measured in the September 5 [performance results](docs/performance-2026-09-05.md),
2222
[expanded workloads](docs/performance-expansion-2026-09-05.md), and
2323
[operator and file controls](docs/performance-next-milestones-2026-09-05.md).
24+
The September 6 [execution milestones](docs/performance-execution-2026-09-06.md)
25+
cover nested pruning, batch output, arithmetic, plan reuse, and external sorting.
2426
See the [benchmark guide](docs/benchmarks.md) for versions, methodology, and limitations.
2527

2628
## Supported Log Formats
@@ -247,10 +249,17 @@ from it
247249
select approx_count_distinct(user_agent) from it
248250
```
249251

250-
`HAVING` currently refers to aggregate output aliases: use `having cnt > 10`
251-
with `count(*) as cnt`. Writing the aggregate call again in HAVING is unsupported.
252-
`GROUP BY` also requires at least one aggregate expression; use `SELECT DISTINCT`
253-
when only distinct keys are needed.
252+
`HAVING` accepts aggregate aliases and aggregate calls, including aggregates not
253+
returned by SELECT. For example, `select category from it group by category
254+
having sum(score) > 25` filters groups without adding the sum to the output.
255+
Without an explicit GROUP BY, SELECT must itself contain an aggregate:
256+
`select count(*) as n from it having count(*) > 0` is supported.
257+
`GROUP BY` requires an aggregate in SELECT or HAVING; use `SELECT DISTINCT`
258+
when only distinct keys are needed. See the [executable SQL examples](tests/conformance/README.md).
259+
260+
Numeric equality currently distinguishes integers from floats. SUM returns a
261+
float, so use `having sum(score) = 25.0` for equality; ordered comparisons such
262+
as `sum(score) > 25` compare across numeric types.
254263

255264
### ORDER BY and LIMIT
256265

‎docs/architecture.md‎

Lines changed: 32 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -80,7 +80,14 @@ Non-correlated subqueries are planned recursively with the same source registry.
8080

8181
The logical layer distinguishes value-producing `Expression` from boolean
8282
`Formula`. Physical conversion in `logical/types.rs` builds execution nodes and
83-
places generated constant bindings in a `Variables` map.
83+
keeps literals as constant values, so JSON fields cannot shadow SQL constants.
84+
85+
`logical/having.rs` binds aggregate calls in HAVING to aggregate outputs before
86+
ordinary planning. It reuses unambiguous SELECT aggregates or adds collision-free
87+
hidden aggregates, then removes hidden outputs after HAVING and before
88+
DISTINCT/ORDER/LIMIT. Nested aggregate calls are rejected. HAVING-only aggregates
89+
require explicit GROUP BY when SELECT contains no aggregate, matching the current
90+
grouped-projection subset rather than silently discarding SELECT expressions.
8491

8592
### 4. Pipeline Selection (`execution/types.rs`)
8693

@@ -90,8 +97,11 @@ tree. Eligible subtrees use columnar operators. Unsupported parents can still
9097
consume a batch child through `BatchToRowAdapter`.
9198

9299
Fixed-format files support typed batch scans. JSONL batching requires a known set
93-
of required root fields, which can be empty for `count(*)`; nested selected
94-
values use dynamic `Value` storage. Bare scans, stdin, table bindings, regex
100+
of required root fields, which can be empty for `count(*)`. Required object paths
101+
prune unreferenced nested siblings while preserving full consumed-line validation.
102+
Whole-object references override narrower masks; array/index/wildcard and scoped
103+
shapes retain their safe full-value fallback. Nested selected values still use
104+
dynamic `Value` storage. Bare scans, stdin, table bindings, regex
95105
formats, joins, set operations, and some expression/path shapes use row
96106
operators. Prefix LIMIT paths preserve demand-driven parsing and expression
97107
evaluation so they need not read later invalid rows.
@@ -125,6 +135,19 @@ to avoid converting rejected rows back into records. Supported expressions bind
125135
function handles for repeated evaluation. Typed kernels operate on contiguous
126136
numeric data and validity bitmaps; mixed values retain scalar semantics.
127137

138+
JSON/NDJSON consume the selected batch stream directly, borrowing string and
139+
mixed-value cells instead of rebuilding a map per output row. The row API, CSV
140+
and table consumers still use `BatchToRowAdapter` as needed. Both result interfaces
141+
share pipeline selection, LIMIT guards and the query memory tracker. Duplicate
142+
output names use the last occurrence consistently in projection, predicates,
143+
sorting and serialization.
144+
145+
Trusted Float32 arithmetic can produce a typed output column directly for a
146+
left-associated chain of addition by constants/root Float32 columns and
147+
multiplication by Float32 constants. Every operation retains f32 rounding;
148+
custom functions, integer overflow-aware arithmetic and unsupported trees use
149+
the original evaluation path.
150+
128151
| Operation | Retained state |
129152
| --- | --- |
130153
| Scan/filter/project/limit | Current input buffers and batches; queued batches for parallel scans |
@@ -203,6 +226,12 @@ floats and may lose precision. Large identifiers should be stored as strings.
203226
Integer arithmetic uses checked operations so overflow is an error in both
204227
debug and release builds.
205228

229+
Equality distinguishes Int32 from Float32, while ordered comparisons compare
230+
across numeric types. For example, SUM returns a float and `SUM(x) = 5.0`
231+
differs from `SUM(x) = 5`. The [numeric migration proposal](numeric-migration.md)
232+
specifies the compatibility work required for a future Int64/Float64 runtime;
233+
it does not change the current representation.
234+
206235
NULL means an unknown value; MISSING means an absent field. Both participate in
207236
three-valued arithmetic/comparison logic, and `IS NULL`/`IS MISSING` distinguish
208237
them. Boolean logic includes rules such as `NULL AND FALSE = FALSE` and

0 commit comments

Comments
 (0)