[christianeu-db]

Summary

Context

Discussion for this PR is in this thread: https://lists.apache.org/thread.html/vmgv5ffryw70wxbvtwnohhhz2zzx0ocf

Ossie is a vendor-neutral specification. Its extensibility model already reflects
this: a custom_extensions entry pairs a free-form vendor_name with a single
opaque data string. A vendor or an individual can attach any amount of extra
state, but that state is cordoned into one opaque value rather than promoted to
top-level spec fields. The core stays small and vendor-neutral, and everything
vendor-specific is opaque payload that the core carries but does not interpret.

Expressions are the one place where this principle is not yet applied. This PR
extends the same pattern to expressions and dialects. It is a spec-level change;
see the mailing-list threads under Related Issues.

1. Make the portable Ossie expression the primary field (backward-compatible)

Today an expression is a list of dialects, each a {dialect, expression} pair.
Ossie defines one portable dialect, OSSIE_SQL_2026
(core-spec/expression_language.md), but in practice it sits as just another
entry in the dialects list, peer to vendor dialects.
This is the only place in the spec where an Ossie concept and a vendor
concept appear at the same level, and the portable form is rarely authored as a
result (there are some workstreams in flight to close this gap).

This PR adds an optional top-level sql field to the expression object. It
is the primary, portable OSSIE_SQL_2026. The dialects list
remains for vendor-specific variants. When both the top-level sql and an
OSSIE_SQL_2026 entry in dialects are present, the top-level field is
authoritative. For portable-only expressions, this leads to a more concise representation
since the dialect sections / dialect type string can be dropped.

The field is optional and additive, so the change is backward-compatible: existing
documents remain valid.

Note: a prior iteration had the nesting as expression: expression: to stay consistent with the (dialect, expression) pair but the feedback on the thread from @jochenchrist pointed to the double-key nesting feeling wrong

2. Make dialect a free-form string, not an enum

dialect is currently a closed enum in core-spec/ossie-schema.json. Adding a
dialect requires a change to the core spec and a DISCUSS thread, and a dialect
outside the enum fails validation by default, so an outside party cannot register
its own dialect without a full spec change. Yet these dialects are explicitly
opaque: an implementation MAY ignore any non-OSSIE_SQL_2026 dialect, and
registering a dialect does not imply Ossie can query it.

vendor_name is already a free-form string for exactly these reasons. This PR
applies the same decoupling to dialect: enum to free-form string. One could argue
dialects belong entirely in custom_extensions, but expressions are fundamental
enough to keep as a first-class field; only the closed enumeration is removed.

Examples: expression portability before / after this PR

#	Job to be done	Before	After
1	Publish a measure that is already fully portable	✅ Portable form must be nested as a dialect entry. expression:   dialects:     - dialect: OSSIE_SQL_2026       expression: SUM(ss_sales_price)	✅ Portable form is a first-class top-level field. expression:   sql: SUM(ss_sales_price)
2	Keep vendor overrides alongside a portable form	✅ Portable and vendor forms are peers; nothing marks the primary. expression:   dialects:     - dialect: OSSIE_SQL_2026       expression: SUM(ss_sales_price)     - dialect: MAQL       expression: "SELECT SUM({fact/store_sales.ss_sales_price})"	✅ Portable form is primary; vendor forms are optional overrides. expression:   sql: SUM(ss_sales_price)   dialects:     - dialect: MAQL       expression: "SELECT SUM({fact/store_sales.ss_sales_price})"
3	Model an expression that cannot be made portable	✅ Vendor-only dialects, no portable form. expression:   dialects:     - dialect: DATABRICKS       expression: approx_count_distinct(session_id)     - dialect: SNOWFLAKE       expression: APPROX_COUNT_DISTINCT(session_id)	✅ Identical — omitting the top-level sql means "not portable." expression:   dialects:     - dialect: DATABRICKS       expression: approx_count_distinct(session_id)     - dialect: SNOWFLAKE       expression: APPROX_COUNT_DISTINCT(session_id)
4	Support an engine Ossie does not enumerate yet (FakeDB)	❌ Fails validation — FakeDB is not in the enum; valid only after editing core-spec/spec.yaml and regenerating the SDK types. expression:   dialects:     - dialect: FakeDB       expression: quantile_cont(latency_ms, 0.95)	✅ Valid — unknown dialect is opaque; optionally register it in extensions/dialects.md (docs only). expression:   dialects:     - dialect: FakeDB       expression: quantile_cont(latency_ms, 0.95)

Rows 1–3 are additive: every existing document stays valid and dialects
keeps working. Row 4 is the new capability, mirroring how vendor_name
already works for custom_extensions.

Why this matters

Together, the future shape is an expression whose primary, portable form is defined
once at the top level, with dialects as optional overrides. The Hex converter calls
out the variety of expression types: not every dialect is a SQL dialect - some are richer
expression languages. More broadly, there is no reason an expression could not be a
code snippet or any other evaluation detail. A closed enum is too restrictive for
that range; a free-form string is not.

Extensions registry

The dialect and vendor example constants move out of the normative spec into a new
top-level extensions/ folder (README.md, dialects.md, vendors.md). This is a
community registry of well-known tokens: a place to align on their spelling and
meaning without making any token fundamental to the spec. Presence in a list does
not imply endorsement, and new tokens can be added without a full spec review.

Changes in this PR

• core-spec/ossie-schema.json: Dialect becomes type: string with illustrative
  examples and no enum; Vendor examples are genericized; the Expression
  object gains the optional (Ossie) sql field with its precedence rule.
• core-spec/spec.md, spec.yaml, expression_language.md: document the free-form
  dialect, the portable sql field, and the link to extensions/; the
  opaque / declaration-only text is retained.
• extensions/: new README.md, dialects.md, vendors.md.
• scripts/generate-spec-types.py: no longer generates a dialects enumeration
  (dialects are free-form, like vendors); it still syncs datatypes.
• python/ SDK: OssieDialectExpression.dialect is typed str; OssieExpression
  gains the optional sql; OssieDialect / OssieVendor remain as
  convenience constants.
• validation/validate.py: an unregistered dialect is treated as opaque and skipped (with a warning)
  rather than parsed as ANSI SQL; the portable sql is validated as
  OSSIE_SQL_2026.
• validation/validate.py: cleanup on validation to extract dialect logic as an iterator / incorporate feat(core-spec): support dataset-scoped metrics #591.
• examples/tpcds_semantic_model.yaml: demonstrates the portable ossie sql field.

Scope and follow-ups

The scope is deliberately minimal. The following refinements are left for follow-up
PRs:

• Converters: this will be a separate change in ossie-converters
• The vendor and dialect constant code and validation should move further from the
  core — into the converters or a shared mixins package — rather than living
  alongside the core models. This can be done incrementally.
• The OSSIE_SQL_2026 reading logic should be generalized to bias toward the primary
  top-level sql when it is defined. This PR documents that precedence in the
  spec and the reference validator honors it, but the converters are not yet updated
  to read the field.
• Pulling the Ossie -> vendor mappings from expressions will be a follow up

Related Issues

• Relates to Add measurements for model portability across converters ossie-converters#24 (finding that expressions were not ported in practice)
• Mailing-list discussion:
  • https://lists.apache.org/thread.html/0tql9g8myg255hhonhbzo906rdpgstcf (listing all the spots vendor-specific details have entered the spec)
  • https://lists.apache.org/thread.html/xh6o9mzhmrdpnjzxsn7kotjsg18yksvb (DISCUSS for adding an enum)
  • https://lists.apache.org/thread.html/46h1cr804mn6m4r35vrqn1tkdhrjxm7m (DISCUSS for adding 2 other enums)
  • https://lists.apache.org/thread.html/msbk9hml117ttyoxgqqz1h59s1rdpf3b (DISCUSS finding that listing the ossie dialect separately increases context size / vendor extensions are a huge part of payloads in practice)

Checklist

Specification

• Spec changes are included in core-spec/ and follow the existing structure
• Spec changes have been discussed on the mailing list or in a linked issue: https://lists.apache.org/thread.html/vmgv5ffryw70wxbvtwnohhhz2zzx0ocf
• Breaking changes to the spec are clearly called out in the summary (none — the change is additive and relaxes validation; the SDK dialect field type changed from an enum to str)

Ontology

• Ontology changes in ontology/ are consistent with spec changes (N/A — no ontology changes)
• New or modified terms are defined and documented (N/A)

Converters

• Converter change in f8d3508 will be opened in a new issue / PR in ossie-converters once this merges

Validation

• Validation rules in validation/ are updated if the spec changed
• New validation cases are covered by tests

Documentation

• docs/ is updated to reflect any user-facing changes (follow-up; see Scope)
• New features or behaviors are documented with examples where appropriate (in core-spec/ and examples/)
• CONTRIBUTING.md is updated if the contribution process changed (N/A)

Examples

• examples/ are added or updated for any new spec constructs or converter support. Examples in the spec and updated a store sample

Tests

• All existing tests pass (pytest / CI green)
• New functionality is covered by tests

Compliance

• ASF license headers are present on all new source files
• No third-party dependencies are added without PMC/IPMC approval
• If third-party source code is vendored/copied (not just declared as a dependency), NOTICE and/or LICENSE have been updated per ASF policy (N/A — no vendored third-party source)

[kayemkim]

I read this alongside the list thread and merged it locally onto current main (b42ce09), no conflicts. Three things on the code; the design points are on the list.

The hex check job fails at the head: ty check reports invalid-argument-type at load_ossie_dialect_expression.py:79, where _to_sqlglot_dialect(ossie_dialect: OssieDialect) now receives the str that OssieDialectExpression.dialect became. The runtime side matters more: that function ends in assert_never, so a document with a dialect outside the old enum, which this PR makes valid, raises AssertionError: Expected code to be unreachable, but got: 'DB_DIALECT' on export. I ran it directly with DB_DIALECT and SNOWFLKE; both raise. The wisdom test you added covers this case; hex needs the same, with the final branch returning None for anything unregistered and the parameter typed str.

Expression still lists dialects under required with minItems: 1, and OssieExpression.dialects keeps Field(..., min_length=1), so a portable-only document fails both the schema and the SDK:

[Schema] datasets -> 0 -> fields -> 0 -> expression: 'dialects' is a required property

A required of one of expression or dialects (anyOf) in the schema and a model validator in the SDK would match the behaviour the summary describes.

With #612 cherry-picked on top, the two tests that read $defs.Dialect.enum fail with KeyError (validation test_every_schema_dialect_is_mapped_or_skipped, Holistics test_every_schema_dialect_is_parsed_or_marked_not_sql); the table cross-check and the sqlglot reader tests pass. I can point those two at the table in extensions/dialects.md instead, in whichever order the two PRs land.

[christianeu-db]

I read this alongside the list thread and merged it locally onto current main (b42ce09), no conflicts. Three things on the code; the design points are on the list.

The hex check job fails at the head: ty check reports invalid-argument-type at load_ossie_dialect_expression.py:79, where _to_sqlglot_dialect(ossie_dialect: OssieDialect) now receives the str that OssieDialectExpression.dialect became. The runtime side matters more: that function ends in assert_never, so a document with a dialect outside the old enum, which this PR makes valid, raises AssertionError: Expected code to be unreachable, but got: 'DB_DIALECT' on export. I ran it directly with DB_DIALECT and SNOWFLKE; both raise. The wisdom test you added covers this case; hex needs the same, with the final branch returning None for anything unregistered and the parameter typed str.

Expression still lists dialects under required with minItems: 1, and OssieExpression.dialects keeps Field(..., min_length=1), so a portable-only document fails both the schema and the SDK:

[Schema] datasets -> 0 -> fields -> 0 -> expression: 'dialects' is a required property

A required of one of expression or dialects (anyOf) in the schema and a model validator in the SDK would match the behaviour the summary describes.

With #612 cherry-picked on top, the two tests that read $defs.Dialect.enum fail with KeyError (validation test_every_schema_dialect_is_mapped_or_skipped, Holistics test_every_schema_dialect_is_parsed_or_marked_not_sql); the table cross-check and the sqlglot reader tests pass. I can point those two at the table in extensions/dialects.md instead, in whichever order the two PRs land.

Hi @kayemkim I've addressed the validator concern. The converter change will be addressed separately in ossie-converters. As I called out in the list, I decided to update the PR to incorporate your feedback around and expression iter (this had better ROI after merging in #591)

[abolfazlmadanii]

Thanks for this, the top-level sql reads much better than burying the portable form as one more dialect entry. It also lines up with what I saw when measuring payloads for #616: on portable-only expressions you drop the dialects: wrapper and the dialect: line every time, which adds up fast in big models.

A few small things from reading the latest head:

1. Both forms present and different. If a doc has a top-level sql and also an OSSIE_SQL_2026 entry in dialects with different text, the validator checks both and says nothing. Since the spec says the top-level one wins, the other is effectively dead text. A warning when they differ (or even just when both exist) would catch drift early.

2. Empty string. The new sql property has no minLength: 1, so sql: "" passes the schema and the SDK (Optional[str]), and only the SQL check catches it. Adding minLength: 1 would keep the schema, the SDK and the validator in agreement.

3. Converters. The hex assert_never crash kayemkim found lives in ossie-converters now, so it's probably worth listing in the converter follow-up issue so it doesn't get lost when dialects go free-form.

4. Nice to have. The unknown-dialect warning could suggest the closest known name (difflib.get_close_matches against DIALECT_MAP), so SNOWFLKE says "did you mean SNOWFLAKE?".

None of these block the direction, which I'm +1 on.

[kayemkim]

Re-ran the same checks on bd29692 merged onto current main (7205e15): validation 261 passed, python 217, the four workflows CI would run here all pass locally, spec.yaml in sync, and the examples come out as on main. Of the three things above, the portable-only document now passes both the schema and the SDK, and the hex crash left with the converters, so abolfazlmadanii's note to carry it into the ossie-converters issue covers it. #612 still fails on top of this branch, now for two reasons: the enum read and the DIALECT_MAP rename. No need to repoint it on your side. I will rebase #612 after this lands and pin SQLGLOT_DIALECTS and NON_SQL_DIALECTS to the table in extensions/dialects.md, which today lists exactly the same twelve tokens. #595 and #608 move onto iter_expressions at the same time.

Two small things from probing the new head. dialect: "" passes the schema and the SDK and comes out as "dialect '' is not supported by this validator". A minLength of 1 on Dialect, the same as abolfazlmadanii suggested for sql, closes it. And dialect: snowflake is now a different token from SNOWFLAKE and gets the same warning. Nothing in extensions/README.md says the tokens are matched exactly, so one sentence there would save the first lowercase registration. On his first point, a document with sql and a different OSSIE_SQL_2026 entry passes silently here too, and the superseded flag already marks that entry, so the warning is one comparison in the loop.