Repository navigation
feat(core-spec): add Layer 3 wide table query interface - #653
Open
christianeu-db wants to merge 1 commit into
Open
christianeu-db wants to merge 1 commit into
christianeu-db wants to merge 1 commit into
Conversation
Add core-spec/wide_table_semantics.md, the Layer 3 specification of the layered query interface. A Layer 3 query names dimensions, metrics, and one filter. The engine lowers the query to a Layer 2 query that evaluates each metric with AGG(). This version covers the shared-dimension case: it joins only along many-to-one relationships, and it raises an error for ambiguous or unsupported cases. Update the pointers to the Layer 3 specification: - spec_layers.md: link the new file in place of apache#246, and name metrics, a filter, and datasets in the Layer 3 summary. - relational_semantics.md: link the new file in place of apache#246, and link Layers 1 and 3 from the conformance section. Co-authored-by: Isaac <no-reply@databricks.com>
christianeu-db
force-pushed
the
wide-table-query-interface
branch
from
October 28, 2026 06:31
855afbe to
6868df5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds core-spec/wide_table_semantics.md, specifying Layer 3, the Wide Table Query Interface, of the layered query interface in spec_layers.md. A Layer 3 query names dimensions, metrics, and an optional filter. The engine chooses the joins, lowers the query to Layer 2 (#354), and returns the result.
This builds directly on #246 by willpugh and the expressions working group working group. #246 established the core guarantees for querying an Ossie model: preserving fact rows, avoiding double counting, combining facts across shared dimensions, and rejecting queries that cannot be evaluated safely.
This PR preserves those guarantees for a smaller initial subset, while building on the Layer 2 query semantics and existing core spec.
What carries forward from #246
Query shape (§5.1.1). Queries specify dimensions, metrics (#246's Measures), and a pre-aggregation filter (Where). Results group by all dimensions, with one row for grand totals.
Correctness guarantees (§6.1). The five user-visible guarantees carry forward as S1–S5 in this spec: fact preservation, no fan-out double counting, preservation of multi-fact groups, no unsafe re-aggregation, and no silently wrong answers. S6 adds that filtering cannot introduce new result groups.
Join and stitching semantics (§6.6, §6.8). LEFT JOIN preserves unmatched fact rows in NULL groups. Facts aggregate independently and combine on shared dimensions using NULL-safe equality. Grand totals combine through a CROSS JOIN of one-row results. Missing groups use SQL empty-input aggregate values (§6.11).
Path resolution (§6.9). Ambiguous paths raise E_AMBIGUOUS_PATH; the engine never silently chooses a path. The initial spec retains this rule for many-to-one traversal.
Filtering and errors (§6.2, Appendix A). Filters apply before aggregation to every metric. The spec retains the relevant error vocabulary and the principle of returning an error rather than an incorrect result.
Key differences from #246
Measure evaluation belongs to Layer 2. #246 defines its own fan-out and chasm-trap rewrites, including pre-aggregation (§6.7), bridge de-duplication (§6.8.1), and stitching (§6.8.2). This spec instead evaluates measures through Layer 2's AGG() (#354). Layer 3 chooses paths, joins, filter placement, and how independently aggregated results combine. It introduces no separate measure-evaluation rules.
Model definitions come from the core spec. #246 §4 defines datasets, fields, relationships, metrics, grain, namespaces, and cardinality inference. This spec reuses the existing core model, including dataset-scoped and model-scoped metrics (#591). Relationship direction establishes many-to-one cardinality without requiring primary or unique key declarations.
A smaller initial query surface. This version supports many-to-one traversal through unambiguous paths, with dimensions and filters reachable from every metric. It defers one-to-many and many-to-many traversal, including bridges; HAVING, sorting and limits; scalar queries; window functions; and more complex filtering. Consumers needing richer query behavior can use Layer 2 directly, selecting their own joins and filters while retaining measure-aware aggregation.
Intent
The goal is a minimal, interoperable Layer 3 baseline for star schemas and multiple fact tables sharing dimensions.
Unsupported or ambiguous cases raise errors rather than relying on heuristics. Appendix A identifies possible extensions, including per-metric overrides, explicit relationship selection, one-to-many traversal, and richer query operations.
These can be standardized incrementally. Their defaults must preserve the results of queries accepted by this version, following #246's additive-extension principle.
NB
The query interface and binding are intentionally non-normative, except that Layer 3 must not present itself as a SQL relation exposing measure columns or AGG(), since its result domain can depend on the requested metrics.
Related Issues
Relates to #452, #354. Derived from #246 (by wpugh & earlier working group discussions)
Checklist
Specification
core-spec/and follow the existing structureOntology
ontology/are consistent with spec changesConverters
Validation
validation/are updated if the spec changedDocumentation
docs/is updated to reflect any user-facing changesCONTRIBUTING.mdis updated if the contribution process changedExamples
examples/are added or updated for any new spec constructsTests
pytest/ CI green)Compliance
NOTICEand/orLICENSEhave been updated per ASF policyAI disclosure
Per the ASF Generative Tooling Guidance, this contribution was prepared with AI assistance. I have reviewed and verified every change.