Skip to content

feat: add extended metadata fields for fields and metrics - #287

Open
jklahr wants to merge 12 commits into
apache:mainfrom
jklahr:extended-metadata
Open

jklahr wants to merge 12 commits into
apache:mainfrom
jklahr:extended-metadata

Conversation

@jklahr

@jklahr jklahr commented Jul 29, 2026 •

Copy link
Copy Markdown
Contributor

Summary

This PR proposes a new extended metadata layer for the Apache Ossie core spec — a standardized, optional set of non-executional, interpretability metadata fields for fields and metrics.

A companion announcement and discussion will be sent to dev@ossie.apache.org.

Motivation

OSI/Ossie defines what data is. There is currently limited standardization for how data should be interpreted and presented. As a result, consumers (BI tools, AI agents, developers) must repeatedly infer or redefine units, display conventions, aggregation behavior, KPI polarity, and field visibility.

This proposal addresses those gaps through optional metadata that does not affect query execution.

Extended proposal document (design rationale, field-by-field detail, open questions): https://docs.google.com/document/d/1DLuteF0WwlVbpD5kmHhcqa6vItqNFL4MoAQrImy6dKo

Changes

Three files in core-spec/ are updated:

  • spec.yaml — adds extended metadata fields to fields and metrics
  • osi-schema.json — adds new $defs: SemanticType, Measurement, DefaultSort, SemanticMapping, DesiredDirection, DefaultAggregation, DefaultTimeGranularity, UnitSystem
  • spec.md — documents all new fields with tables, type descriptions, and examples

New fields

Field Applies to Purpose
display_label fields, metrics Human-readable display name
semantic_type fields, metrics Token or URI classification (e.g., monetary, or https://www.iso20022.org/glossary/LEI)
measurement fields, metrics Unit/quantity metadata (quantity_kind, unit ISO 4217, unit_system)
display_format fields, metrics Excel-compatible format string (e.g., $#,##0.00)
default_sort fields, metrics Default sort direction, null handling, and optional sort-by-field
glossary_references fields, metrics References to concepts in external glossaries or ontologies via a SKOS-based open predicate vocabulary (target, predicate, provenance)
hidden fields, metrics Hide from consumer UIs while remaining available in expressions
group_label fields, metrics Organizational grouping for UI presentation
desired_direction metrics only KPI polarity: higher_is_better, lower_is_better, neutral
default_aggregation fields only Default aggregation when field is used as a measure
default_time_granularity fields only Default time bucket for temporal fields: day, week, month, quarter, year

Design principles

  • Non-executional: no field affects query execution
  • Optional and backward compatible: all existing models remain valid
  • Consumer-focused: designed for BI tools, AI agents, and developers

Community input incorporated

Dragos Crintea (maintainer of semantido, a SQLAlchemy-based semantic layer used in capital-markets/regulatory-reporting contexts) provided substantive feedback that has been incorporated into this PR:

  1. glossary_references uses an open predicate vocabulary (SKOS baseline: exactMatch, closeMatch, broadMatch, narrowMatch, relatedMatch) rather than a closed enum, allowing non-positive assertions like DISTINCT_FROM via extensions. This addresses the regulatory homonym problem — e.g., "Counterparty" under EMIR and "Counterparty" under MiFIR are distinct concepts that should not be conflated.
  2. semantic_type accepts either a well-known token or a URI, keeping simple cases simple while supporting governed external type systems (ISO 20022, FIBO, SKOS vocabularies, LEI, ISIN, etc.).

Dragos has also offered to contribute a reference implementation in the semantido-OSI converter and benchmark data showing accuracy improvements from this class of metadata. Those contributions will follow in subsequent PRs.

Naming

glossary_references was originally proposed as semantic_mappings; renamed per review input from Khushboo. The name describes what the field carries more directly — pointers to concepts in an external glossary, ontology, or standard — and avoids confusion with semantic_type. The structure is unchanged.

Impact on existing implementations

None — all additions are optional. The JSON schema uses additionalProperties: false per existing convention; all new fields are added explicitly to the Field and Metric definitions.

Related discussions

AI disclosure

This contribution was developed with AI assistance (Snowflake Cortex Code). The author (Josh Klahr) has reviewed all changes and is personally responsible for the content per the ASF Generative Tooling Guidance.

.... Generated with Cortex Code

Co-Authored-By: Cortex Code noreply@snowflake.com

@djwaldo

djwaldo commented Aug 20, 2026

Copy link
Copy Markdown
Contributor
  1. Have we considered locale settings in regards to display_format? I.e. for EN_AU i might want my currency to be displayed as $#,##0.00 however for SE_SE i might want the display to be $# ##0,00

  2. Have we considered multiple group_labels? A column may be present in multiple group sections. I.e. i could define groups for HR Team, Sales Team then also KPIs, Attributes. So 'Sales Amount' could be in three groups.

@jklahr

jklahr commented Aug 20, 2026

Copy link
Copy Markdown
Contributor Author

djwaldo good question

  • for the first one I wonder if we want to introduce the concept of locale files (basically extra files per local that include the ability to create locale specific metadata). I could imagine wanting to do this for descriptions, synonyms, etc.

  • for groups, yes I think we might want to do this, I can update

Josh Klahr and others added 4 commits August 27, 2026 10:50
Introduces optional, non-executional interpretability metadata to the
core spec. New fields enable consumers (BI tools, AI agents, developers)
to correctly interpret, render, and present semantic model data.

New field-level attributes: display_label, semantic_type, measurement,
display_format, default_aggregation, default_sort, default_time_granularity,
semantic_mappings, hidden, group_label.

New metric-level attributes: display_label, semantic_type, measurement,
display_format, desired_direction, default_sort, semantic_mappings,
hidden, group_label.

All additions are optional and backward compatible. Existing models
remain valid without modification.

.... Generated with [Cortex Code](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code)

Co-Authored-By: Cortex Code <noreply@snowflake.com>
…c_type

Incorporate suggestions from semantido maintainer:

- Restructure semantic_mappings: replace source+identifier with target
  (URI) + predicate (open vocabulary, SKOS baseline) + optional provenance.
  The open predicate vocabulary allows non-positive assertions such as
  DISTINCT_FROM to be expressed via extensions, addressing the regulatory
  homonym problem (e.g. EMIR vs MiFIR Counterparty disambiguation).

- Change semantic_type from a closed enum to token-or-URI. Well-known
  tokens (categorical, monetary, temporal, etc.) continue to work as-is;
  a URI value allows governed external type systems (ISO 20022, FIBO,
  SKOS vocabularies) to carry domain-specific types without requiring
  spec changes.

.... Generated with [Cortex Code](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code)

Co-Authored-By: Cortex Code <noreply@snowflake.com>
Allow a field or metric to belong to more than one organizational
group. Renamed from singular `group_label` (string) to `group_labels`
(array of strings).

.... Generated with [Cortex Code](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code)

Co-Authored-By: Cortex Code <noreply@snowflake.com>
The token-or-URI form of semantic_type was encoded as oneOf over an enum
branch and a branch using "format": "uri". JSON Schema treats format as an
annotation rather than an assertion unless a validator opts in, so a plain
token such as "monetary" matched both branches and oneOf rejected it. Every
well-known token was therefore invalid, including the values used throughout
the examples in spec.md.

Replaces oneOf with anyOf and asserts the URI branch with a pattern
requiring a scheme (^[A-Za-z][A-Za-z0-9+.-]*:). This makes the two branches
disjoint without relying on format, and adds validation the original design
would not have provided even had it worked: a value that is neither a known
token nor a scheme-qualified URI, such as a misspelled "monetory", is now
rejected instead of being silently accepted as an opaque type reference.

spec.md and spec.yaml are updated to state that the URI form requires a
scheme.

Found by validating a document that exercises every extended metadata field.
The existing checks did not catch it because neither the TPC-DS example nor
the Complete Example in spec.md sets semantic_type.

.... Generated with [Cortex Code](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code)

Co-Authored-By: Cortex Code <noreply@snowflake.com>
@jklahr
jklahr force-pushed the extended-metadata branch from f0507d9 to 070a81c Compare August 27, 2026 18:09
Renames the field on both fields and metrics, and the corresponding schema
definition from SemanticMapping to GlossaryReference, per review input from
Khushboo.

"Glossary references" describes what the field carries more directly: pointers
to concepts in an external glossary, ontology, or standard. "Semantic mappings"
was easy to confuse with semantic_type, and read as though it described a
transformation rather than a reference.

No structural change. The target/predicate/provenance shape, the SKOS baseline
predicates, and the open predicate vocabulary are unchanged.

Also updates the ROADMAP entry for this proposal, which listed the old field
name.

.... Generated with [Cortex Code](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code)

Co-Authored-By: Cortex Code <noreply@snowflake.com>
@jklahr jklahr mentioned this pull request Aug 29, 2026
15 of 17 tasks
@djwaldo

djwaldo commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Thanks for picking up the grouping point,group_labels as an array with "A field may belong to more than one group" is exactly it.

On the locale question, let me offer what ThoughtSpot actually does, since we shipped this and the shape may be useful input. Short version: we came to the same "locale files" conclusion you floated, but the key turned out not to be locale.

How it works. Localized names live in a separate document, a column_alias object, not inline on the model, and each record is keyed by a four-tuple:

(column, locale, org, group)  ->  alias, description

So the same column can read "Revenue" for one audience, "Chiffre d'affaires" for a French locale, and "Net Bookings (AUD)" for a particular tenant, all at once, resolved at read time against the viewer's locale and their org and group membership.

Three things fell out of that which might be worth folding into the design:

  1. Locale alone was not enough a key. We started there and had to widen it. In practice the same question, "what should this column be called for this reader", is answered by tenant and audience at least as often as by language, and once you have two mechanisms for renaming a column you have a precedence problem. One keyed record handles all three.

  2. It carries description, not just the label. A localized display name over an English description is a half-translated model, and users notice. If display_label becomes localizable, description almost certainly has to travel with it.

  3. It is a separate document because it does not fit inline. This is the practical part. Our TML import has size limits where a wide model across several locales and a few dozen tenants gets there faster than you would guess. Anything that puts the full locale matrix inline in the core document will hit it. We do have delta strategies to mitigate this.

On display_format specifically — and this is a genuine gap on our side, not advice. Our alias record carries alias and description and not format, so ThoughtSpot has no per-column-per-locale format string; number and currency presentation is resolved from instance and user settings instead. That works until a model is genuinely multi-region, and then $#,##0.00 versus $# ##0,00 is not a preference, it is wrong output.

So I would gently push back on display_format being a single Excel-compatible string on the field. Either it is understood to be locale-neutral, in which case the spec should say what a consumer does with it in a non-en_US locale, or it belongs in whatever locale mechanism eventually lands, alongside display_label and description. What I would most want to avoid is it being the one presentation attribute that cannot be localized, since it is arguably the one where getting it wrong is most visible.

Happy to write up the alias schema in more detail if a locale proposal gets going — including the parts we would do differently.

@djwaldo

djwaldo commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Two concrete mismatches from mapping ThoughtSpot onto this PR's fields. Both are cheap to change now and awkward once the enum ships, so raising them while it is open.

1. default_time_granularity has no equivalent of "let the engine decide."

ThoughtSpot's per-column default grain is DAILY | WEEKLY | MONTHLY | QUARTERLY | YEARLY | AUTO. The five named values map onto day | week | month | quarter | year one-for-one. AUTO does not, and it is not an edge case, it means "pick the grain from the time span the query actually covers", so a five-year query buckets by month and a one-week query by day, from the same column definition.

Under the enum as drafted, a model using it either loses the setting or has to pick a fixed grain that is wrong at most zoom levels. Adding auto costs one enum value; a consumer that does not implement it can fall back to whatever it does today. Worth noting the value is genuinely different in kind from the other five, it is a deferral rather than a grain, so if that distinction is unwelcome in the enum, a separate boolean would also solve it.

2. Is "Excel-compatible" the canonical vocabulary for display_format, or the example?

The field is specified as an "Excel-compatible display format string (e.g. $#,##0.00)". ThoughtSpot has a native format-pattern vocabulary of its own, and I would guess most tools with a formatting layer do. That makes this a translation on the way in and out rather than a passthrough, which is fine, but it needs saying, because translation raises questions the spec is currently silent on:

  • What should a converter do with a native pattern that has no Excel equivalent? Drop it, approximate it, or refuse?
  • Is a consumer expected to render Excel format strings, or to translate them back into its own vocabulary?
  • Round-tripping native -> Excel -> native will not be lossless for every pattern. Is that understood and accepted, or should the original be preserved alongside?

One option worth considering: allow the field to carry the format plus an optional vocabulary tag, defaulting to Excel. That keeps the simple case simple and gives a lossless path for tools that have their own. But mainly I want to flag that "Excel-compatible" is doing a lot of load-bearing work in one adjective, and the round-trip semantics are the part that will bite implementers.

Neither of these blocks the PR from my side, the extended-metadata layer is a clear improvement and I would rather see it land than see it perfect.

djwaldo added a commit to thoughtspot/thoughtspot-agent-skills that referenced this pull request Aug 31, 2026
PR #287 (extended metadata for fields and metrics, jklahr, open) proposes
eleven optional non-executional fields. Five are things the construct mapping
currently stashes under vendor_name: THOUGHTSPOT precisely because the standard
had nowhere to put them.

Recorded as a watch item, not a plan. #287 is OPEN, so nothing here is spec and
the converter must not emit these as first-class fields yet — a display_format
on a Field fails schema validation today exactly as a THOUGHTSPOT dialect does.

New section 'Pending upstream' maps all five:

- data_panel_column_groups -> group_labels. Clean; our map values are always
  empty strings so it degrades to an array. The array shape is downstream of
  ThoughtSpot feedback on the PR (2026-08-20).
- properties.is_hidden -> hidden. Clean, with our own caveat that inbound
  hidden: true stays an issue rather than a silent write.
- default_date_bucket -> default_time_granularity. MISMATCH: AUTO has no target.
- format_pattern -> display_format. MISMATCH: Excel-compatible vs ThoughtSpot's
  own vocabulary, so a translation rather than a passthrough.
- currency_type -> measurement. Partial: iso_code maps, the column and
  is_browser forms describe a per-row unit and have no target.

A5 re-graded: display_label answers it against our reading rather than for it,
so re-grade rather than re-post, and the residual question belongs on #287 not
on the dormant discussion #37. G2 narrows to its identifier half.

Also corrects two venues found dead while checking: A12 -> issue #290
(Cube.dev, opened the day before our mapping; our scan only covered
discussions, so an issue was structurally invisible) and A3 -> discussion #342
(6 comments in two days) rather than #5, which has one comment ever, ours.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
- measurement must contain at least one property (minProperties: 1)
- by_field is a name, not an expression; document it on metric default_sort,
  which the shared DefaultSort schema already permits
- display_format is a format pattern, not an expression or named preset
- state explicitly that default_* properties are hints that do not change
  query results, and that display labels are not referenceable names

Generated with [Snowflake CoCo](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code)

Co-authored-by: Snowflake CoCo <noreply@snowflake.com>
@github-actions github-actions Bot added the spec label Oct 20, 2026
Josh Klahr and others added 3 commits October 20, 2026 07:41
Mirror the extended metadata properties in the Python SDK so documents
that use them load instead of failing as unknown keys, and register the
new Measurement, DefaultSort and GlossaryReference nodes in the
schema/model parity check.

Generated with [Snowflake CoCo](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code)

Co-authored-by: Snowflake CoCo <noreply@snowflake.com>
- display_format is locale-invariant; '.' and ',' are separator placeholders
- currency symbol derives from measurement.unit, not a literal in the pattern
- localized rendering must not change a value's currency
- base strings are in the authoring language; variants are out of scope

Generated with [Snowflake CoCo](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code)

Co-authored-by: Snowflake CoCo <noreply@snowflake.com>
@jklahr

jklahr commented Oct 21, 2026

Copy link
Copy Markdown
Contributor Author

@djwaldo thanks, this is exactly the input the locale question needed, and the ThoughtSpot write-up changed my view of where it belongs. I've pushed an update that settles what can be settled in this PR, and I'd like to take the rest (localized and audience-specific names) to a separate proposal. A sketch of it is below so you can tear into it before it's written up.

What changed in this PR

Spec wording only. No new properties, and the schema changes are description text.

1. display_format is locale-invariant. In a pattern, . and , are the decimal and grouping separator placeholders, not literal characters. Consumers render them with the viewer's locale, so #,##0.00 is 1,234.56 for en-AU and 1 234,56 for sv-SE from the same pattern. That's the LDML/ICU convention, and it's how Excel number formats behave across regional settings, so "Excel-compatible" and "localizable" stop being in tension. It answers your question about what a consumer does outside en-US: it applies the viewer's separators. A separator meant literally is quoted or escaped, as in Excel.

2. Currency comes from measurement.unit, not from the pattern. A symbol written into the pattern ($#,##0.00) is a literal and renders as $ everywhere, which is the wrong output you described. When quantity_kind is currency, unit (ISO 4217) is now the authoritative currency, and consumers derive the symbol and its placement from it plus the viewer's locale. The examples now use #,##0.00 with unit: usd.

3. Localization never changes the unit. New normative sentence: a consumer MUST NOT present a value in a currency other than its measurement.unit. "Net Bookings (AUD)" for one tenant over USD data isn't a relabel; it's a different value, and it belongs in a separate metric (or a conversion), not in presentation metadata. I think this is the line any locale mechanism has to hold, so I wanted it in the base spec first.

4. Base strings are monolingual. display_label, description, ai_context and group_labels are in the model's authoring language, and this version doesn't define variants. That keeps the door open for the proposal below without anything here needing to change.

Your other two points, auto for default_time_granularity and an optional vocabulary tag on display_format for lossless round-trips, are separate from localization, and I agree they're cheaper now than later. I'll come back to them in a follow-up comment rather than mix them in here.

Proposed separately: Ossie presentation overlays

Your three lessons are the design constraints: (1) locale alone isn't the key, (2) the description travels with the label, (3) it has to live outside the core document. Here's the shape I have in mind.

A separate document that targets a model and overrides presentation properties when a selector matches:

# sales.overlays.yaml
version: 0.2.0.dev0
overlay_for: sales            # name of the base model
overlays:
  - when: { locale: fr }
    metrics:
      total_revenue:
        display_label: Chiffre d'affaires
        description: Chiffre d'affaires total des commandes terminées
        ai_context: { synonyms: [CA, ventes] }
    datasets:
      orders:
        fields:
          order_date:
            display_label: Date de commande

  - when: { audience: finance }
    metrics:
      total_revenue:
        display_label: Net Bookings
        ai_context: { synonyms: [bookings, net bookings] }

  - when: { audience: finance, locale: fr }
    metrics:
      total_revenue:
        display_label: Réservations nettes

Selectors. locale (BCP 47) and audience (an opaque string) to start, with an open set of keys. Ossie doesn't define who belongs to an audience; the consumer resolves it from user, group, org or tenant. Your (column, locale, org, group) maps onto this as the target plus a selector of {locale, audience}, or {locale, org, group} if keeping org and group apart turns out to matter. That's one of the questions below.

Precedence.

  1. The most specific match wins: audience + locale beats audience alone, which beats locale alone, which beats the base model.
  2. Locale falls back progressively: fr-CA → fr → base.
  3. Two equally specific overlays setting the same property on the same target is a validation error, so the result is never ambiguous.

Only presentation properties can be overridden.

  • Allowed: display_label, description, ai_context (including synonyms), group_labels, hidden, and display_format. display_format is for genuine exceptions, such as fewer decimals for an executive audience; separators and currency already localize without it.
  • Not allowed: name, expression, datatype, measurement, semantic_type, relationships. An overlay can change what a metric is called, never what it is, which is also what keeps the currency rule above enforceable.

Label and description together. A partial overlay is allowed, but validation should warn when an overlay localizes display_label without description for the same target. Your "half-translated model" lesson, made checkable.

The base model declares its language. An optional locale on the core document (e.g. en-US), so a consumer knows whether a fr overlay needs to apply at all.

Why a separate document.

  • Your size point.
  • Translations and audience vocabularies are usually owned by different people than the model, on a different cadence.
  • A consumer that ignores overlays still reads a complete, valid model.

Questions I'd value your view on

  1. Selector granularity. Did keeping org and group as separate keys earn its keep at ThoughtSpot, or would a single audience string have been enough?
  2. Precedence in practice. When a user matched several groups with different aliases, how did you break the tie? A fixed order, the first match, or something the admin configured?
  3. Synonyms. Did localized synonyms (for search and NL query) end up in the alias record, or somewhere else?
  4. Deltas. You mentioned delta strategies for size. Is that something the overlay format should support directly, such as one file per locale merged at load, or is it an import-pipeline concern?
  5. What you'd do differently. You offered this, and I'd take it. That's the part most worth getting into the proposal before it's written.

If this direction looks right to you, I'll open it as its own issue and discussion thread and link it here, so this PR can land on the base metadata without waiting on it.

@djwaldo

djwaldo commented Oct 21, 2026

Copy link
Copy Markdown
Contributor

1 - Selector granularity. Did keeping org and group as separate keys earn its keep at ThoughtSpot, or would a single audience string have been enough?

Damian >> I think an single audience is probably enough. The concept of Org and Group is more related to ThoughtSpot architecture. I.e. within a single cluster we can setup multi-tenancy with separate Orgs. Within an Org users can be members of Groups, where these Groups could represent a tenant. We built our aliasing to support clients that are setup in either manner.

The only wrinkle to this is where a combination of Org->Group->Local = En-AU results in a different definition. This is unlikely an probably unnecessary for language aliasing. E.g. Sofa vs Couch. These are both valid in en-au but different groups use different terms. I do not think we need this level of definition.

  1. Precedence in practice. When a user matched several groups with different aliases, how did you break the tie? A fixed order, the first match, or something the admin configured?

Damian >> In ThoughtSpot a user can only have a single locale at any one time. Therefore we did not have a precedence issue. The issue is how to debug the issue. For an adjacent area of the product if a single area could not be resolved we returned the base string. In this way we could quickly identify that the user could have multiple definitions.

Personally I lean to a fixed order of definition. This assumes that you can change this order. This gives a use the control.

  1. Synonyms. Did localized synonyms (for search and NL query) end up in the alias record, or somewhere else?

Damian >> we parked synonyms for the initial release. This just added additional work for limited value. However I feel that synonyms are just an array of values that live in the alias file. I.e. these are valid synonyms for the current alias.

  1. Deltas. You mentioned delta strategies for size. Is that something the overlay format should support directly, such as one file per locale merged at load, or is it an import-pipeline concern?

Damian >> So the quickest path to manage change is a complete replacement. However as the number and subsequent size of the alias files grow this becomes problematic. I.e. I think we have capped files size of objects at 25MB. This is more than enough however I could see large alias files being above this if everything was updated. Also then all the back ground meta data services need to update everything.

So the concern is more about import-pipeline.

  1. What you'd do differently. You offered this, and I'd take it. That's the part most worth getting into the proposal before it's written.

Damian >> i like the proposed updates in the PR. For me the key is thinking about this as a foundational layer so we get it correct now rather than have to retrofit it later.

@eisber eisber left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few suggestions based on experience with Power BI, in case they help make the semantics clearer across tools.

Comment on lines +161 to +165
"DefaultAggregation": {
"type": "string",
"enum": ["sum", "avg", "min", "max", "count", "count_distinct"],
"description": "Default aggregation function to apply when this field is used as a measure"
},

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Based on experience with Power BI, would it be useful to include none, distinct from leaving this property unset? Its SummarizeBy property distinguishes a default choice from an explicit “do not summarize” preference—for example, on a numeric customer identifier.

Perhaps omission could mean “no preference specified,” while none means “do not implicitly aggregate.” That would preserve the author’s intent without preventing a consumer from explicitly requesting an aggregation.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, and added. none is now in the enum, with the distinction you described:

  • Omitted: no preference; the consumer applies its own default.
  • none: don't aggregate implicitly, e.g. a numeric customer_id. A consumer SHOULD NOT add an aggregate when the field is placed in a query, but MAY still apply one the user explicitly asks for.

That keeps the author's intent without blocking an explicit request, which I think is the part of SummarizeBy worth carrying over.

Comment thread core-spec/spec.md Outdated
Comment on lines +603 to +607
The `display_format` string follows Excel-compatible custom number format conventions. Consumers MAY support a subset, but interoperability is improved when adhering to common patterns.

The value is a format pattern. It is not an expression, and named presets such as `currency` or `percent` are not part of the vocabulary.

**The pattern is locale-invariant.** In a pattern, `.` denotes the decimal separator and `,` the grouping separator; they are placeholders, not literal characters. Consumers render them with the viewer's locale, so one pattern serves every locale: `#,##0.00` displays as `1,234.56` for `en-AU` and `1 234,56` for `sv-SE`. This is the convention of LDML/ICU number patterns, and how Excel number formats behave across regional settings. A separator meant literally is quoted or escaped, as in Excel.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Excel-style patterns seem a practical starting point. Could we make the reference more precise by citing the SpreadsheetML number-format definitions and Microsoft’s MS-OI29500 grammar, then identifying the subset Ossie intends to support?

That reference describes Office behavior and includes locale considerations, so Ossie would still need to state its locale-invariant convention explicitly. A few examples covering quoting, positive/negative sections and percentage scaling would help different implementations agree. It would also be useful to say how unsupported patterns are preserved, rather than silently approximated.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. The section now cites the SpreadsheetML number-format grammar (ECMA-376 Part 1, numFmt) and MS-OI29500, and says that where Office's behavior is locale-dependent, Ossie's locale-invariant reading applies instead.

  • Core subset:
    • digit placeholders 0 # ?
    • . and , separator placeholders, including trailing-comma scaling by 1,000
    • % (×100)
    • E+ / E-
    • ; sections (positive;negative;zero)
    • quoted or escaped literals
    • date/time parts
  • Outside the core: bracketed codes (colors, conditions, locale/currency tags, elapsed time) and padding/fill.
  • New examples: a zero section with a quoted literal, percent scaling, and thousands scaling with a suffix.
  • Preservation: a consumer that can't render a pattern falls back to its own formatting and MUST NOT rewrite the stored pattern. A converter whose source format has no core equivalent omits display_format and keeps the native format in custom_extensions, rather than approximating it, so the round trip stays lossless.

Comment thread core-spec/spec.md Outdated
|-------|------|-------------|
| `direction` | string | Sort direction: `asc`, `desc` |
| `nulls` | string | Null positioning: `first`, `last` |
| `by_field` | string | Name of a different field or metric to sort by (e.g., sort month names by month number). A name, not an expression. When omitted, the sort is on the owning field or metric. |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One distinction that has been useful in Power BI is between a column’s sort key—such as MonthName ordered using MonthNumber—and the ordering of a particular result, such as revenue descending. Could we clarify whether this property represents both, or distinguish them?

For the sort-key case, it would help to define reference scope and the handling of missing targets or cycles. The month example also has a data constraint: each month-name value needs one corresponding month-number value. Making that explicit would give consumers a common interpretation rather than leaving them to infer one.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good distinction. The section now names the two explicitly: direction/nulls are the default result ordering, and by_field is the sort key. For the sort key:

  • Scope: on a field, by_field names another field of the same dataset. On a metric, it names another metric in the same scope (model-scoped → model-scoped, dataset-scoped → same dataset).
  • Resolution: a by_field that names nothing, or names its owner, is invalid, and so is a cycle.
  • Data constraint: each value of the owner must correspond to exactly one sort-key value. That's documented as a property of the data rather than validated, and the ordering is undefined if it doesn't hold.

I kept it as one object rather than splitting it, since both are presentation defaults for the same field. Happy to split if you think the combination will confuse implementers. Enforcing resolution and cycles in validate.py is a follow-up; this PR states the rules.

Comment thread core-spec/spec.md Outdated
Comment on lines +529 to +531
All three properties are optional, but a `measurement` object MUST contain at least one of them.

When `quantity_kind` is `currency`, `unit` is the authoritative currency of the value. Consumers SHOULD derive the currency symbol and its placement from `unit` and the viewer's locale rather than from a symbol written into `display_format` (see [Display Format](#display-format)).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we reference ISO 4217 for currency identity and Unicode CLDR for localized currency presentation? That would build on established definitions rather than require Ossie to define its own symbol mapping.

It would also help to specify currency-code case normalization and the meaning of an omitted unit. For example, $ alone does not distinguish USD, AUD and CAD. I’d suggest that an omitted unit leaves currency unspecified; neither the display symbol nor the viewer’s locale should supply an inferred currency identity.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done:

  • Identity follows ISO 4217 and presentation follows CLDR, both cited, so Ossie defines no symbol mapping of its own.
  • Case: currency codes compare case-insensitively, and uppercase is canonical; the examples are now USD. This applies only to currency codes, because UCUM codes are case-sensitive.
  • Omitted unit: I took your wording. It leaves the currency unspecified, and a consumer MUST NOT infer one from a symbol in display_format or from the viewer's locale.

Comment thread core-spec/spec.md
| `quantitative` | Numeric values representing quantities |
| `monetary` | Currency/financial values |
| `temporal` | Time or date values |
| `geographic` | Location-related values (country, lat/lng, region) |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we add worked examples showing how more specific geographic meaning is represented? Two cases from experience with Power BI are country versus state/subdivision (CA might mean Canada or California), and latitude versus longitude (both may be numeric angular coordinates, but they are not interchangeable).

The URI alternative already seems to offer room for that specificity. It would help to demonstrate a documented vocabulary that distinguishes these roles, and explain when an author should use a specific URI rather than the broad geographic token. A converter could retain its original category in custom_extensions where no agreed portable mapping exists.

This need not turn the core enum into a long category list; the aim would be to show how consumers preserve distinctions needed to interpret the values correctly.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a worked example under Semantic Type:

  • Country vs subdivision: https://schema.org/Country vs https://schema.org/State, with ISO 3166-1 / 3166-2 noted in the descriptions.
  • Latitude vs longitude: https://schema.org/latitude / longitude, with measurement in degrees.

There's also guidance on when to use a URI instead of the geographic token, and a rule that a converter with a finer source category and no agreed URI keeps the closest token and preserves the original under custom_extensions. The core enum is unchanged.

jklahr and others added 3 commits October 26, 2026 10:41
…tended metadata

The merge with main kept both the relocated OssieMetric and the extended one.
OssieDataset.metrics bound to the first, so a dataset-scoped metric using any
extended metadata property was rejected as an unknown key. Collapse to one
class and add a test that both metric placements share it.

Generated with [Snowflake CoCo](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code)

Co-authored-by: Snowflake CoCo <noreply@snowflake.com>
…eography

- default_aggregation gains `none`, distinct from leaving it unset
- display_format cites ECMA-376/MS-OI29500, defines a core subset, preserves unsupported patterns
- default_sort separates result ordering from the sort key and defines by_field scope
- currency identity per ISO 4217, presentation per CLDR; omitted unit is unspecified
- worked example for geographic roles using URI semantic types

Generated with [Snowflake CoCo](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code)

Co-authored-by: Snowflake CoCo <noreply@snowflake.com>
@jklahr

jklahr commented Oct 26, 2026

Copy link
Copy Markdown
Contributor Author

@djwaldo thank you, these answers are exactly what the overlays proposal needed. How each one changes it:

  1. A single audience. Adopting it. Org and group become audiences at the deployment layer, outside the format.
  2. Precedence: declared order, not an error. I'll drop "equally specific overlays conflict = validation error". Instead, the most specific match wins, and among equally specific matches, the author's declared order decides, so the author stays in control as you suggested. Validation will warn on equally specific overlaps so they're visible. I'd also like to note your debugging approach as a consumer option: when a value can't be resolved, return the base string so the gap is obvious rather than silently picked.
  3. Synonyms. An array on the overlay record, next to the label and description.
  4. Deltas. Out of the format. Splitting and merging files is an import-pipeline concern.
  5. Foundational. Agreed. That's why this PR settles the rendering rules now: locale-invariant formats and currency identity. Overlays will then only have to carry strings.

I'll open presentation overlays as its own issue and link it here.

Changes since the last review, for anyone catching up:

  • Fix: resolving the merge conflict with main (dataset-scoped metrics, feat(core-spec): support dataset-scoped metrics #591) had left two OssieMetric classes in the Python SDK, so a dataset-scoped metric using extended metadata was rejected. There's now a single class, plus a regression test that both metric placements share it.

  • @eisber's review, all five suggestions:

    • default_aggregation: none
    • a precise display_format reference with a core subset and a preservation rule
    • default_sort sort-key semantics
    • currency identity and omission rules (ISO 4217 / CLDR)
    • geographic worked examples

    Details are in the inline replies.

@eisber

eisber commented Oct 26, 2026

Copy link
Copy Markdown
Contributor

Thanks for addressing the first-round suggestions—the sort-key scope and resolution rules are much clearer now. One remaining gap: the spec now declares missing by_field targets, self-references and cycles invalid, but validation/validate.py's validate_document() does not check default_sort.by_field, so these cases can still receive Validation PASSED.

Could we include scope-aware sort-reference validation in this PR, with negative tests for unknown targets, self-references, cross-scope references and cycles, plus a valid month-name/month-number case? These are document-level checks; the data-dependent one-sort-key-value-per-owner-value constraint can remain explicitly unchecked as documented.

There is also a concrete fixture to update: _extended_document() in python/tests/test_models.py sets the revenue metric's by_field to revenue (line 761 at head 6056a21), which is a self-reference under the new rules. The schema-only round-trip test does not catch that semantic invalidity.

@christianeu-db christianeu-db left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for putting this together! left a few comments about cases where some fields seem related / could be combined.

it might be worth landing some of these changes piece-meal on the list. I could see more discussion on the ICU v. Excel-based formatting & don't think we should block some much clearer cases (e.g. some of the folder/tag-like metadata) on that

Comment thread core-spec/spec.md
| `expression` | object | Yes | Expression definition with dialect support |
| `dimension` | object | No | Dimension metadata (e.g., `is_time` flag) |
| `label` | string | No | Label for categorization |
| `display_label` | string | No | Human-readable display name for UI and AI interaction |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would display_name be more consistent here?

name covers how this field is referenced in the expression language. display_name is how this is referenced in the ui

label seems more for categorization

Comment thread core-spec/spec.md
| `description` | string | No | Human-readable description of what the metric measures |
| `datatype` | string (enum) | No | Logical data type for this metric. See [Data types](#data-types). |
| `ai_context` | string/object | No | Additional context for AI tools (e.g., synonyms) |
| `display_label` | string | No | Human-readable display name for UI and AI interaction |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread core-spec/spec.md
| `semantic_type` | string | No | High-level semantic classification (see [Semantic Type](#semantic-type)) |
| `measurement` | object | No | Unit and quantity metadata (see [Measurement](#measurement)) |
| `display_format` | string | No | Excel-compatible, locale-invariant format pattern (e.g., `#,##0.00`, `0.0%`). See [Display Format](#display-format). |
| `default_aggregation` | string | No | Default aggregation when used as a measure: `sum`, `avg`, `min`, `max`, `count`, `count_distinct`, or `none` (do not aggregate implicitly). Omit when there is no preference. |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: should this be an enum?

Comment thread core-spec/spec.md
| `default_sort` | object | No | Default sorting behavior (see [Default Sort](#default-sort)) |
| `default_time_granularity` | string | No | Default time bucket for temporal fields: `day`, `week`, `month`, `quarter`, `year` |
| `glossary_references` | array | No | References to concepts in external glossaries or ontologies (see [Glossary References](#glossary-references)) |
| `hidden` | boolean | No | Whether this field should be hidden from consumer UIs |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

might be a question for composition, but I could see there being two types of hidden (especially for datset-level fields/metrics).

  1. removing the field from all consumers (in sql terms SELECT * EXCEPT(...)) - this is useful for hiding helper metrics / metrics that are often useful for building more complex fields/metrics. for this hidden, no one should see these columns
  2. hidden from UI: just a UI hint - perhaps, could be a special label type

Comment thread core-spec/spec.md
| `measurement` | object | No | Unit and quantity metadata (see [Measurement](#measurement)) |
| `display_format` | string | No | Excel-compatible, locale-invariant format pattern (e.g., `#,##0.00`, `0.0%`). See [Display Format](#display-format). |
| `desired_direction` | string | No | KPI polarity: `higher_is_better`, `lower_is_better`, `neutral` |
| `default_sort` | object | No | Default sorting behavior (see [Default Sort](#default-sort)) |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not quite sure what this means for the metric case.

default_sort for a field makes a lot of sense (e.g. you drag out this field, use the sort logic to pick the sort order)

for metrics, it's a bit less clear

Comment thread core-spec/spec.md
Comment on lines +292 to +293
| `semantic_type` | string | No | High-level semantic classification (see [Semantic Type](#semantic-type)) |
| `measurement` | object | No | Unit and quantity metadata (see [Measurement](#measurement)) |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wondering if these should be folded into an object. there is some coupling here (e.g. monetary in semantic_type and then separate details in measurement) but these are separate fields

Comment thread core-spec/spec.md
Defines default sorting behavior for a field or metric. It carries two distinct things:

- **Default result ordering** — `direction` and `nulls`: how a result is ordered when it is sorted by this field or metric and the user has not said otherwise (for example, revenue descending).
- **Sort key** — `by_field`: the values of this field or metric are ordered by the values of another one (for example, `month_name` ordered by `month_number`, so that January precedes February rather than April).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could these be generalized to more general expressions. For example, you might want want the default sort on region to be SUM(sales).

},
"display_format": {
"type": "string",
"description": "Excel-compatible, locale-invariant display format pattern (e.g., #,##0.00, 0.0%). '.' and ',' are decimal and grouping separator placeholders rendered in the viewer's locale."

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Excel-compatible part might be worth a discussion on the thread to get more vendor buy-in.

There are differences between Excel and ICU, which might lean towards one vendor. For example:

ICU: yyyy-MM-dd (upper to distinguish from minutes)
Excel: yyyy-mm-dd

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants