Skip to content

feat(core-spec): add Layer 3 wide table query interface - #653

Open
christianeu-db wants to merge 1 commit into
apache:mainfrom
christianeu-db:wide-table-query-interface
Open

christianeu-db wants to merge 1 commit into
apache:mainfrom
christianeu-db:wide-table-query-interface

Conversation

@christianeu-db

@christianeu-db christianeu-db commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Summary

This PR adds core-spec/wide_table_semantics.md, specifying Layer 3, the Wide Table Query Interface, of the layered query interface in spec_layers.md. A Layer 3 query names dimensions, metrics, and an optional filter. The engine chooses the joins, lowers the query to Layer 2 (#354), and returns the result.

This builds directly on #246 by willpugh and the expressions working group working group. #246 established the core guarantees for querying an Ossie model: preserving fact rows, avoiding double counting, combining facts across shared dimensions, and rejecting queries that cannot be evaluated safely.

This PR preserves those guarantees for a smaller initial subset, while building on the Layer 2 query semantics and existing core spec.

What carries forward from #246

Query shape (§5.1.1). Queries specify dimensions, metrics (#246's Measures), and a pre-aggregation filter (Where). Results group by all dimensions, with one row for grand totals.

Correctness guarantees (§6.1). The five user-visible guarantees carry forward as S1–S5 in this spec: fact preservation, no fan-out double counting, preservation of multi-fact groups, no unsafe re-aggregation, and no silently wrong answers. S6 adds that filtering cannot introduce new result groups.

Join and stitching semantics (§6.6, §6.8). LEFT JOIN preserves unmatched fact rows in NULL groups. Facts aggregate independently and combine on shared dimensions using NULL-safe equality. Grand totals combine through a CROSS JOIN of one-row results. Missing groups use SQL empty-input aggregate values (§6.11).

Path resolution (§6.9). Ambiguous paths raise E_AMBIGUOUS_PATH; the engine never silently chooses a path. The initial spec retains this rule for many-to-one traversal.

Filtering and errors (§6.2, Appendix A). Filters apply before aggregation to every metric. The spec retains the relevant error vocabulary and the principle of returning an error rather than an incorrect result.

Key differences from #246

Measure evaluation belongs to Layer 2. #246 defines its own fan-out and chasm-trap rewrites, including pre-aggregation (§6.7), bridge de-duplication (§6.8.1), and stitching (§6.8.2). This spec instead evaluates measures through Layer 2's AGG() (#354). Layer 3 chooses paths, joins, filter placement, and how independently aggregated results combine. It introduces no separate measure-evaluation rules.

Model definitions come from the core spec. #246 §4 defines datasets, fields, relationships, metrics, grain, namespaces, and cardinality inference. This spec reuses the existing core model, including dataset-scoped and model-scoped metrics (#591). Relationship direction establishes many-to-one cardinality without requiring primary or unique key declarations.

A smaller initial query surface. This version supports many-to-one traversal through unambiguous paths, with dimensions and filters reachable from every metric. It defers one-to-many and many-to-many traversal, including bridges; HAVING, sorting and limits; scalar queries; window functions; and more complex filtering. Consumers needing richer query behavior can use Layer 2 directly, selecting their own joins and filters while retaining measure-aware aggregation.

Intent

The goal is a minimal, interoperable Layer 3 baseline for star schemas and multiple fact tables sharing dimensions.

Unsupported or ambiguous cases raise errors rather than relying on heuristics. Appendix A identifies possible extensions, including per-metric overrides, explicit relationship selection, one-to-many traversal, and richer query operations.

These can be standardized incrementally. Their defaults must preserve the results of queries accepted by this version, following #246's additive-extension principle.

NB
The query interface and binding are intentionally non-normative, except that Layer 3 must not present itself as a SQL relation exposing measure columns or AGG(), since its result domain can depend on the requested metrics.

Related Issues

Relates to #452, #354. Derived from #246 (by wpugh & earlier working group discussions)

Checklist

Specification

Ontology

  • Ontology changes in ontology/ are consistent with spec changes
  • New or modified terms are defined and documented

Converters

Validation

  • Validation rules in validation/ are updated if the spec changed
  • New validation cases are covered by tests

Documentation

  • docs/ is updated to reflect any user-facing changes
  • New features or behaviors are documented with examples where appropriate
  • CONTRIBUTING.md is updated if the contribution process changed

Examples

  • examples/ are added or updated for any new spec constructs

Tests

  • All existing tests pass (pytest / CI green)
  • New functionality is covered by tests

Compliance

  • ASF license headers are present on all new source files
  • No third-party dependencies are added without PMC/IPMC approval
  • If third-party source code is vendored/copied (not just declared as a dependency), NOTICE and/or LICENSE have been updated per ASF policy

AI disclosure

Per the ASF Generative Tooling Guidance, this contribution was prepared with AI assistance. I have reviewed and verified every change.

@github-actions github-actions Bot added the spec label Oct 28, 2026
Add core-spec/wide_table_semantics.md, the Layer 3 specification of the
layered query interface. A Layer 3 query names dimensions, metrics, and
one filter. The engine lowers the query to a Layer 2 query that
evaluates each metric with AGG(). This version covers the
shared-dimension case: it joins only along many-to-one relationships,
and it raises an error for ambiguous or unsupported cases.

Update the pointers to the Layer 3 specification:
- spec_layers.md: link the new file in place of apache#246, and
  name metrics, a filter, and datasets in the Layer 3 summary.
- relational_semantics.md: link the new file in place of
  apache#246, and link Layers 1 and 3 from the conformance
  section.

Co-authored-by: Isaac <no-reply@databricks.com>
@christianeu-db
christianeu-db force-pushed the wide-table-query-interface branch from 855afbe to 6868df5 Compare October 28, 2026 06:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant