Repository navigation
Conversation
|
|
djwaldo good question
|
Introduces optional, non-executional interpretability metadata to the core spec. New fields enable consumers (BI tools, AI agents, developers) to correctly interpret, render, and present semantic model data. New field-level attributes: display_label, semantic_type, measurement, display_format, default_aggregation, default_sort, default_time_granularity, semantic_mappings, hidden, group_label. New metric-level attributes: display_label, semantic_type, measurement, display_format, desired_direction, default_sort, semantic_mappings, hidden, group_label. All additions are optional and backward compatible. Existing models remain valid without modification. .... Generated with [Cortex Code](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code) Co-Authored-By: Cortex Code <noreply@snowflake.com>
…c_type Incorporate suggestions from semantido maintainer: - Restructure semantic_mappings: replace source+identifier with target (URI) + predicate (open vocabulary, SKOS baseline) + optional provenance. The open predicate vocabulary allows non-positive assertions such as DISTINCT_FROM to be expressed via extensions, addressing the regulatory homonym problem (e.g. EMIR vs MiFIR Counterparty disambiguation). - Change semantic_type from a closed enum to token-or-URI. Well-known tokens (categorical, monetary, temporal, etc.) continue to work as-is; a URI value allows governed external type systems (ISO 20022, FIBO, SKOS vocabularies) to carry domain-specific types without requiring spec changes. .... Generated with [Cortex Code](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code) Co-Authored-By: Cortex Code <noreply@snowflake.com>
Allow a field or metric to belong to more than one organizational group. Renamed from singular `group_label` (string) to `group_labels` (array of strings). .... Generated with [Cortex Code](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code) Co-Authored-By: Cortex Code <noreply@snowflake.com>
The token-or-URI form of semantic_type was encoded as oneOf over an enum branch and a branch using "format": "uri". JSON Schema treats format as an annotation rather than an assertion unless a validator opts in, so a plain token such as "monetary" matched both branches and oneOf rejected it. Every well-known token was therefore invalid, including the values used throughout the examples in spec.md. Replaces oneOf with anyOf and asserts the URI branch with a pattern requiring a scheme (^[A-Za-z][A-Za-z0-9+.-]*:). This makes the two branches disjoint without relying on format, and adds validation the original design would not have provided even had it worked: a value that is neither a known token nor a scheme-qualified URI, such as a misspelled "monetory", is now rejected instead of being silently accepted as an opaque type reference. spec.md and spec.yaml are updated to state that the URI form requires a scheme. Found by validating a document that exercises every extended metadata field. The existing checks did not catch it because neither the TPC-DS example nor the Complete Example in spec.md sets semantic_type. .... Generated with [Cortex Code](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code) Co-Authored-By: Cortex Code <noreply@snowflake.com>
f0507d9 to
070a81c
Compare
Renames the field on both fields and metrics, and the corresponding schema definition from SemanticMapping to GlossaryReference, per review input from Khushboo. "Glossary references" describes what the field carries more directly: pointers to concepts in an external glossary, ontology, or standard. "Semantic mappings" was easy to confuse with semantic_type, and read as though it described a transformation rather than a reference. No structural change. The target/predicate/provenance shape, the SKOS baseline predicates, and the open predicate vocabulary are unchanged. Also updates the ROADMAP entry for this proposal, which listed the old field name. .... Generated with [Cortex Code](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code) Co-Authored-By: Cortex Code <noreply@snowflake.com>
|
Thanks for picking up the grouping point, On the locale question, let me offer what ThoughtSpot actually does, since we shipped this and the shape may be useful input. Short version: we came to the same "locale files" conclusion you floated, but the key turned out not to be locale. How it works. Localized names live in a separate document, a So the same column can read "Revenue" for one audience, "Chiffre d'affaires" for a French locale, and "Net Bookings (AUD)" for a particular tenant, all at once, resolved at read time against the viewer's locale and their org and group membership. Three things fell out of that which might be worth folding into the design:
On So I would gently push back on Happy to write up the alias schema in more detail if a locale proposal gets going — including the parts we would do differently. |
|
Two concrete mismatches from mapping ThoughtSpot onto this PR's fields. Both are cheap to change now and awkward once the enum ships, so raising them while it is open. 1. ThoughtSpot's per-column default grain is Under the enum as drafted, a model using it either loses the setting or has to pick a fixed grain that is wrong at most zoom levels. Adding 2. Is "Excel-compatible" the canonical vocabulary for The field is specified as an "Excel-compatible display format string (e.g.
One option worth considering: allow the field to carry the format plus an optional vocabulary tag, defaulting to Excel. That keeps the simple case simple and gives a lossless path for tools that have their own. But mainly I want to flag that "Excel-compatible" is doing a lot of load-bearing work in one adjective, and the round-trip semantics are the part that will bite implementers. Neither of these blocks the PR from my side, the extended-metadata layer is a clear improvement and I would rather see it land than see it perfect. |
PR #287 (extended metadata for fields and metrics, jklahr, open) proposes eleven optional non-executional fields. Five are things the construct mapping currently stashes under vendor_name: THOUGHTSPOT precisely because the standard had nowhere to put them. Recorded as a watch item, not a plan. #287 is OPEN, so nothing here is spec and the converter must not emit these as first-class fields yet — a display_format on a Field fails schema validation today exactly as a THOUGHTSPOT dialect does. New section 'Pending upstream' maps all five: - data_panel_column_groups -> group_labels. Clean; our map values are always empty strings so it degrades to an array. The array shape is downstream of ThoughtSpot feedback on the PR (2026-08-20). - properties.is_hidden -> hidden. Clean, with our own caveat that inbound hidden: true stays an issue rather than a silent write. - default_date_bucket -> default_time_granularity. MISMATCH: AUTO has no target. - format_pattern -> display_format. MISMATCH: Excel-compatible vs ThoughtSpot's own vocabulary, so a translation rather than a passthrough. - currency_type -> measurement. Partial: iso_code maps, the column and is_browser forms describe a per-row unit and have no target. A5 re-graded: display_label answers it against our reading rather than for it, so re-grade rather than re-post, and the residual question belongs on #287 not on the dormant discussion #37. G2 narrows to its identifier half. Also corrects two venues found dead while checking: A12 -> issue #290 (Cube.dev, opened the day before our mapping; our scan only covered discussions, so an issue was structurally invisible) and A3 -> discussion #342 (6 comments in two days) rather than #5, which has one comment ever, ours. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
- measurement must contain at least one property (minProperties: 1) - by_field is a name, not an expression; document it on metric default_sort, which the shared DefaultSort schema already permits - display_format is a format pattern, not an expression or named preset - state explicitly that default_* properties are hints that do not change query results, and that display labels are not referenceable names Generated with [Snowflake CoCo](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code) Co-authored-by: Snowflake CoCo <noreply@snowflake.com>
Mirror the extended metadata properties in the Python SDK so documents that use them load instead of failing as unknown keys, and register the new Measurement, DefaultSort and GlossaryReference nodes in the schema/model parity check. Generated with [Snowflake CoCo](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code) Co-authored-by: Snowflake CoCo <noreply@snowflake.com>
- display_format is locale-invariant; '.' and ',' are separator placeholders - currency symbol derives from measurement.unit, not a literal in the pattern - localized rendering must not change a value's currency - base strings are in the authoring language; variants are out of scope Generated with [Snowflake CoCo](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code) Co-authored-by: Snowflake CoCo <noreply@snowflake.com>
|
@djwaldo thanks, this is exactly the input the locale question needed, and the ThoughtSpot write-up changed my view of where it belongs. I've pushed an update that settles what can be settled in this PR, and I'd like to take the rest (localized and audience-specific names) to a separate proposal. A sketch of it is below so you can tear into it before it's written up. What changed in this PRSpec wording only. No new properties, and the schema changes are description text. 1. 2. Currency comes from 3. Localization never changes the unit. New normative sentence: a consumer MUST NOT present a value in a currency other than its 4. Base strings are monolingual. Your other two points, Proposed separately: Ossie presentation overlaysYour three lessons are the design constraints: (1) locale alone isn't the key, (2) the description travels with the label, (3) it has to live outside the core document. Here's the shape I have in mind. A separate document that targets a model and overrides presentation properties when a selector matches: # sales.overlays.yaml
version: 0.2.0.dev0
overlay_for: sales # name of the base model
overlays:
- when: { locale: fr }
metrics:
total_revenue:
display_label: Chiffre d'affaires
description: Chiffre d'affaires total des commandes terminées
ai_context: { synonyms: [CA, ventes] }
datasets:
orders:
fields:
order_date:
display_label: Date de commande
- when: { audience: finance }
metrics:
total_revenue:
display_label: Net Bookings
ai_context: { synonyms: [bookings, net bookings] }
- when: { audience: finance, locale: fr }
metrics:
total_revenue:
display_label: Réservations nettesSelectors. Precedence.
Only presentation properties can be overridden.
Label and description together. A partial overlay is allowed, but validation should warn when an overlay localizes The base model declares its language. An optional Why a separate document.
Questions I'd value your view on
If this direction looks right to you, I'll open it as its own issue and discussion thread and link it here, so this PR can land on the base metadata without waiting on it. |
|
1 - Selector granularity. Did keeping org and group as separate keys earn its keep at ThoughtSpot, or would a single audience string have been enough? Damian >> I think an single audience is probably enough. The concept of Org and Group is more related to ThoughtSpot architecture. I.e. within a single cluster we can setup multi-tenancy with separate Orgs. Within an Org users can be members of Groups, where these Groups could represent a tenant. We built our aliasing to support clients that are setup in either manner. The only wrinkle to this is where a combination of Org->Group->Local = En-AU results in a different definition. This is unlikely an probably unnecessary for language aliasing. E.g. Sofa vs Couch. These are both valid in en-au but different groups use different terms. I do not think we need this level of definition.
Damian >> In ThoughtSpot a user can only have a single locale at any one time. Therefore we did not have a precedence issue. The issue is how to debug the issue. For an adjacent area of the product if a single area could not be resolved we returned the base string. In this way we could quickly identify that the user could have multiple definitions. Personally I lean to a fixed order of definition. This assumes that you can change this order. This gives a use the control.
Damian >> we parked synonyms for the initial release. This just added additional work for limited value. However I feel that synonyms are just an array of values that live in the alias file. I.e. these are valid synonyms for the current alias.
Damian >> So the quickest path to manage change is a complete replacement. However as the number and subsequent size of the alias files grow this becomes problematic. I.e. I think we have capped files size of objects at 25MB. This is more than enough however I could see large alias files being above this if everything was updated. Also then all the back ground meta data services need to update everything. So the concern is more about import-pipeline.
Damian >> i like the proposed updates in the PR. For me the key is thinking about this as a foundational layer so we get it correct now rather than have to retrofit it later. |
eisber
left a comment
There was a problem hiding this comment.
A few suggestions based on experience with Power BI, in case they help make the semantics clearer across tools.
| "DefaultAggregation": { | ||
| "type": "string", | ||
| "enum": ["sum", "avg", "min", "max", "count", "count_distinct"], | ||
| "description": "Default aggregation function to apply when this field is used as a measure" | ||
| }, |
There was a problem hiding this comment.
Based on experience with Power BI, would it be useful to include none, distinct from leaving this property unset? Its SummarizeBy property distinguishes a default choice from an explicit “do not summarize” preference—for example, on a numeric customer identifier.
Perhaps omission could mean “no preference specified,” while none means “do not implicitly aggregate.” That would preserve the author’s intent without preventing a consumer from explicitly requesting an aggregation.
There was a problem hiding this comment.
Agreed, and added. none is now in the enum, with the distinction you described:
- Omitted: no preference; the consumer applies its own default.
none: don't aggregate implicitly, e.g. a numericcustomer_id. A consumer SHOULD NOT add an aggregate when the field is placed in a query, but MAY still apply one the user explicitly asks for.
That keeps the author's intent without blocking an explicit request, which I think is the part of SummarizeBy worth carrying over.
| The `display_format` string follows Excel-compatible custom number format conventions. Consumers MAY support a subset, but interoperability is improved when adhering to common patterns. | ||
|
|
||
| The value is a format pattern. It is not an expression, and named presets such as `currency` or `percent` are not part of the vocabulary. | ||
|
|
||
| **The pattern is locale-invariant.** In a pattern, `.` denotes the decimal separator and `,` the grouping separator; they are placeholders, not literal characters. Consumers render them with the viewer's locale, so one pattern serves every locale: `#,##0.00` displays as `1,234.56` for `en-AU` and `1 234,56` for `sv-SE`. This is the convention of LDML/ICU number patterns, and how Excel number formats behave across regional settings. A separator meant literally is quoted or escaped, as in Excel. |
There was a problem hiding this comment.
Excel-style patterns seem a practical starting point. Could we make the reference more precise by citing the SpreadsheetML number-format definitions and Microsoft’s MS-OI29500 grammar, then identifying the subset Ossie intends to support?
That reference describes Office behavior and includes locale considerations, so Ossie would still need to state its locale-invariant convention explicitly. A few examples covering quoting, positive/negative sections and percentage scaling would help different implementations agree. It would also be useful to say how unsupported patterns are preserved, rather than silently approximated.
There was a problem hiding this comment.
Done. The section now cites the SpreadsheetML number-format grammar (ECMA-376 Part 1, numFmt) and MS-OI29500, and says that where Office's behavior is locale-dependent, Ossie's locale-invariant reading applies instead.
- Core subset:
- digit placeholders
0 # ? .and,separator placeholders, including trailing-comma scaling by 1,000%(×100)E+/E-;sections (positive;negative;zero)- quoted or escaped literals
- date/time parts
- digit placeholders
- Outside the core: bracketed codes (colors, conditions, locale/currency tags, elapsed time) and padding/fill.
- New examples: a zero section with a quoted literal, percent scaling, and thousands scaling with a suffix.
- Preservation: a consumer that can't render a pattern falls back to its own formatting and MUST NOT rewrite the stored pattern. A converter whose source format has no core equivalent omits
display_formatand keeps the native format incustom_extensions, rather than approximating it, so the round trip stays lossless.
| |-------|------|-------------| | ||
| | `direction` | string | Sort direction: `asc`, `desc` | | ||
| | `nulls` | string | Null positioning: `first`, `last` | | ||
| | `by_field` | string | Name of a different field or metric to sort by (e.g., sort month names by month number). A name, not an expression. When omitted, the sort is on the owning field or metric. | |
There was a problem hiding this comment.
One distinction that has been useful in Power BI is between a column’s sort key—such as MonthName ordered using MonthNumber—and the ordering of a particular result, such as revenue descending. Could we clarify whether this property represents both, or distinguish them?
For the sort-key case, it would help to define reference scope and the handling of missing targets or cycles. The month example also has a data constraint: each month-name value needs one corresponding month-number value. Making that explicit would give consumers a common interpretation rather than leaving them to infer one.
There was a problem hiding this comment.
Good distinction. The section now names the two explicitly: direction/nulls are the default result ordering, and by_field is the sort key. For the sort key:
- Scope: on a field,
by_fieldnames another field of the same dataset. On a metric, it names another metric in the same scope (model-scoped → model-scoped, dataset-scoped → same dataset). - Resolution: a
by_fieldthat names nothing, or names its owner, is invalid, and so is a cycle. - Data constraint: each value of the owner must correspond to exactly one sort-key value. That's documented as a property of the data rather than validated, and the ordering is undefined if it doesn't hold.
I kept it as one object rather than splitting it, since both are presentation defaults for the same field. Happy to split if you think the combination will confuse implementers. Enforcing resolution and cycles in validate.py is a follow-up; this PR states the rules.
| All three properties are optional, but a `measurement` object MUST contain at least one of them. | ||
|
|
||
| When `quantity_kind` is `currency`, `unit` is the authoritative currency of the value. Consumers SHOULD derive the currency symbol and its placement from `unit` and the viewer's locale rather than from a symbol written into `display_format` (see [Display Format](#display-format)). |
There was a problem hiding this comment.
Could we reference ISO 4217 for currency identity and Unicode CLDR for localized currency presentation? That would build on established definitions rather than require Ossie to define its own symbol mapping.
It would also help to specify currency-code case normalization and the meaning of an omitted unit. For example, $ alone does not distinguish USD, AUD and CAD. I’d suggest that an omitted unit leaves currency unspecified; neither the display symbol nor the viewer’s locale should supply an inferred currency identity.
There was a problem hiding this comment.
Done:
- Identity follows ISO 4217 and presentation follows CLDR, both cited, so Ossie defines no symbol mapping of its own.
- Case: currency codes compare case-insensitively, and uppercase is canonical; the examples are now
USD. This applies only to currency codes, because UCUM codes are case-sensitive. - Omitted unit: I took your wording. It leaves the currency unspecified, and a consumer MUST NOT infer one from a symbol in
display_formator from the viewer's locale.
| | `quantitative` | Numeric values representing quantities | | ||
| | `monetary` | Currency/financial values | | ||
| | `temporal` | Time or date values | | ||
| | `geographic` | Location-related values (country, lat/lng, region) | |
There was a problem hiding this comment.
Could we add worked examples showing how more specific geographic meaning is represented? Two cases from experience with Power BI are country versus state/subdivision (CA might mean Canada or California), and latitude versus longitude (both may be numeric angular coordinates, but they are not interchangeable).
The URI alternative already seems to offer room for that specificity. It would help to demonstrate a documented vocabulary that distinguishes these roles, and explain when an author should use a specific URI rather than the broad geographic token. A converter could retain its original category in custom_extensions where no agreed portable mapping exists.
This need not turn the core enum into a long category list; the aim would be to show how consumers preserve distinctions needed to interpret the values correctly.
There was a problem hiding this comment.
Added a worked example under Semantic Type:
- Country vs subdivision:
https://schema.org/Countryvshttps://schema.org/State, with ISO 3166-1 / 3166-2 noted in the descriptions. - Latitude vs longitude:
https://schema.org/latitude/longitude, withmeasurementin degrees.
There's also guidance on when to use a URI instead of the geographic token, and a rule that a converter with a finer source category and no agreed URI keeps the closest token and preserves the original under custom_extensions. The core enum is unchanged.
…tended metadata The merge with main kept both the relocated OssieMetric and the extended one. OssieDataset.metrics bound to the first, so a dataset-scoped metric using any extended metadata property was rejected as an unknown key. Collapse to one class and add a test that both metric placements share it. Generated with [Snowflake CoCo](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code) Co-authored-by: Snowflake CoCo <noreply@snowflake.com>
…eography - default_aggregation gains `none`, distinct from leaving it unset - display_format cites ECMA-376/MS-OI29500, defines a core subset, preserves unsupported patterns - default_sort separates result ordering from the sort key and defines by_field scope - currency identity per ISO 4217, presentation per CLDR; omitted unit is unspecified - worked example for geographic roles using URI semantic types Generated with [Snowflake CoCo](https://docs.snowflake.com/en/user-guide/cortex-code/cortex-code) Co-authored-by: Snowflake CoCo <noreply@snowflake.com>
|
@djwaldo thank you, these answers are exactly what the overlays proposal needed. How each one changes it:
I'll open presentation overlays as its own issue and link it here. Changes since the last review, for anyone catching up:
|
|
Thanks for addressing the first-round suggestions—the sort-key scope and resolution rules are much clearer now. One remaining gap: the spec now declares missing Could we include scope-aware sort-reference validation in this PR, with negative tests for unknown targets, self-references, cross-scope references and cycles, plus a valid month-name/month-number case? These are document-level checks; the data-dependent one-sort-key-value-per-owner-value constraint can remain explicitly unchecked as documented. There is also a concrete fixture to update: |
christianeu-db
left a comment
There was a problem hiding this comment.
thanks for putting this together! left a few comments about cases where some fields seem related / could be combined.
it might be worth landing some of these changes piece-meal on the list. I could see more discussion on the ICU v. Excel-based formatting & don't think we should block some much clearer cases (e.g. some of the folder/tag-like metadata) on that
| | `expression` | object | Yes | Expression definition with dialect support | | ||
| | `dimension` | object | No | Dimension metadata (e.g., `is_time` flag) | | ||
| | `label` | string | No | Label for categorization | | ||
| | `display_label` | string | No | Human-readable display name for UI and AI interaction | |
There was a problem hiding this comment.
Would display_name be more consistent here?
name covers how this field is referenced in the expression language. display_name is how this is referenced in the ui
label seems more for categorization
| | `description` | string | No | Human-readable description of what the metric measures | | ||
| | `datatype` | string (enum) | No | Logical data type for this metric. See [Data types](#data-types). | | ||
| | `ai_context` | string/object | No | Additional context for AI tools (e.g., synonyms) | | ||
| | `display_label` | string | No | Human-readable display name for UI and AI interaction | |
There was a problem hiding this comment.
| | `semantic_type` | string | No | High-level semantic classification (see [Semantic Type](#semantic-type)) | | ||
| | `measurement` | object | No | Unit and quantity metadata (see [Measurement](#measurement)) | | ||
| | `display_format` | string | No | Excel-compatible, locale-invariant format pattern (e.g., `#,##0.00`, `0.0%`). See [Display Format](#display-format). | | ||
| | `default_aggregation` | string | No | Default aggregation when used as a measure: `sum`, `avg`, `min`, `max`, `count`, `count_distinct`, or `none` (do not aggregate implicitly). Omit when there is no preference. | |
There was a problem hiding this comment.
nit: should this be an enum?
| | `default_sort` | object | No | Default sorting behavior (see [Default Sort](#default-sort)) | | ||
| | `default_time_granularity` | string | No | Default time bucket for temporal fields: `day`, `week`, `month`, `quarter`, `year` | | ||
| | `glossary_references` | array | No | References to concepts in external glossaries or ontologies (see [Glossary References](#glossary-references)) | | ||
| | `hidden` | boolean | No | Whether this field should be hidden from consumer UIs | |
There was a problem hiding this comment.
might be a question for composition, but I could see there being two types of hidden (especially for datset-level fields/metrics).
- removing the field from all consumers (in sql terms SELECT * EXCEPT(...)) - this is useful for hiding helper metrics / metrics that are often useful for building more complex fields/metrics. for this hidden, no one should see these columns
- hidden from UI: just a UI hint - perhaps, could be a special label type
| | `measurement` | object | No | Unit and quantity metadata (see [Measurement](#measurement)) | | ||
| | `display_format` | string | No | Excel-compatible, locale-invariant format pattern (e.g., `#,##0.00`, `0.0%`). See [Display Format](#display-format). | | ||
| | `desired_direction` | string | No | KPI polarity: `higher_is_better`, `lower_is_better`, `neutral` | | ||
| | `default_sort` | object | No | Default sorting behavior (see [Default Sort](#default-sort)) | |
There was a problem hiding this comment.
not quite sure what this means for the metric case.
default_sort for a field makes a lot of sense (e.g. you drag out this field, use the sort logic to pick the sort order)
for metrics, it's a bit less clear
| | `semantic_type` | string | No | High-level semantic classification (see [Semantic Type](#semantic-type)) | | ||
| | `measurement` | object | No | Unit and quantity metadata (see [Measurement](#measurement)) | |
There was a problem hiding this comment.
wondering if these should be folded into an object. there is some coupling here (e.g. monetary in semantic_type and then separate details in measurement) but these are separate fields
| Defines default sorting behavior for a field or metric. It carries two distinct things: | ||
|
|
||
| - **Default result ordering** — `direction` and `nulls`: how a result is ordered when it is sorted by this field or metric and the user has not said otherwise (for example, revenue descending). | ||
| - **Sort key** — `by_field`: the values of this field or metric are ordered by the values of another one (for example, `month_name` ordered by `month_number`, so that January precedes February rather than April). |
There was a problem hiding this comment.
could these be generalized to more general expressions. For example, you might want want the default sort on region to be SUM(sales).
| }, | ||
| "display_format": { | ||
| "type": "string", | ||
| "description": "Excel-compatible, locale-invariant display format pattern (e.g., #,##0.00, 0.0%). '.' and ',' are decimal and grouping separator placeholders rendered in the viewer's locale." |
There was a problem hiding this comment.
The Excel-compatible part might be worth a discussion on the thread to get more vendor buy-in.
There are differences between Excel and ICU, which might lean towards one vendor. For example:
ICU: yyyy-MM-dd (upper to distinguish from minutes)
Excel: yyyy-mm-dd
Summary
This PR proposes a new extended metadata layer for the Apache Ossie core spec — a standardized, optional set of non-executional, interpretability metadata fields for
fieldsandmetrics.A companion announcement and discussion will be sent to
dev@ossie.apache.org.Motivation
OSI/Ossie defines what data is. There is currently limited standardization for how data should be interpreted and presented. As a result, consumers (BI tools, AI agents, developers) must repeatedly infer or redefine units, display conventions, aggregation behavior, KPI polarity, and field visibility.
This proposal addresses those gaps through optional metadata that does not affect query execution.
Extended proposal document (design rationale, field-by-field detail, open questions): https://docs.google.com/document/d/1DLuteF0WwlVbpD5kmHhcqa6vItqNFL4MoAQrImy6dKo
Changes
Three files in
core-spec/are updated:spec.yaml— adds extended metadata fields tofieldsandmetricsosi-schema.json— adds new$defs:SemanticType,Measurement,DefaultSort,SemanticMapping,DesiredDirection,DefaultAggregation,DefaultTimeGranularity,UnitSystemspec.md— documents all new fields with tables, type descriptions, and examplesNew fields
display_labelsemantic_typemonetary, orhttps://www.iso20022.org/glossary/LEI)measurementquantity_kind,unitISO 4217,unit_system)display_format$#,##0.00)default_sortglossary_referencestarget,predicate,provenance)hiddengroup_labeldesired_directionhigher_is_better,lower_is_better,neutraldefault_aggregationdefault_time_granularityday,week,month,quarter,yearDesign principles
Community input incorporated
Dragos Crintea (maintainer of semantido, a SQLAlchemy-based semantic layer used in capital-markets/regulatory-reporting contexts) provided substantive feedback that has been incorporated into this PR:
glossary_referencesuses an open predicate vocabulary (SKOS baseline:exactMatch,closeMatch,broadMatch,narrowMatch,relatedMatch) rather than a closed enum, allowing non-positive assertions likeDISTINCT_FROMvia extensions. This addresses the regulatory homonym problem — e.g., "Counterparty" under EMIR and "Counterparty" under MiFIR are distinct concepts that should not be conflated.semantic_typeaccepts either a well-known token or a URI, keeping simple cases simple while supporting governed external type systems (ISO 20022, FIBO, SKOS vocabularies, LEI, ISIN, etc.).Dragos has also offered to contribute a reference implementation in the semantido-OSI converter and benchmark data showing accuracy improvements from this class of metadata. Those contributions will follow in subsequent PRs.
Naming
glossary_referenceswas originally proposed assemantic_mappings; renamed per review input from Khushboo. The name describes what the field carries more directly — pointers to concepts in an external glossary, ontology, or standard — and avoids confusion withsemantic_type. The structure is unchanged.Impact on existing implementations
None — all additions are optional. The JSON schema uses
additionalProperties: falseper existing convention; all new fields are added explicitly to theFieldandMetricdefinitions.Related discussions
AI disclosure
This contribution was developed with AI assistance (Snowflake Cortex Code). The author (Josh Klahr) has reviewed all changes and is personally responsible for the content per the ASF Generative Tooling Guidance.
.... Generated with Cortex Code
Co-Authored-By: Cortex Code noreply@snowflake.com