Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 22 additions & 3 deletions converters/orionbelt/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,19 +97,38 @@ obml_again = OssietoOBML(ossie).convert()
Other Ossie consumers read a metric's SQL, not the `ORIONBELT` extension, so the
OBML to Ossie export writes SQL that computes what OrionBelt computes:

- columns are referenced as `"<dataset>"."<field>"`, always double-quoted;
- columns are referenced as `<dataset>.<field>`, the OBML data object and column
names, following the expression language's identifier rules: a regular
identifier is normalized to upper case, a quoted one is taken as written. A
name is bare where it can be a regular identifier. A name that cannot (a
space, a leading digit, a reserved word such as `Order`) is double-quoted as
written; a regular name DuckDB or sqlglot will not read bare, such as
`Qualify`, is quoted in its normalized form, `"QUALIFY"`. Names that differ
only by case are one identifier to the language: they are quoted as written,
with a warning. The import resolves references by the same rules, and keeps a
metric whose reference matches no field, or more than one, verbatim;
- measure filters become `AGG(CASE WHEN <condition> THEN <arg> END)`, totals the
grand-total window `SUM(SUM(x)) OVER ()`, and `defaultValue` a `COALESCE`;
- synthesized counts and metric-on-metric `{[Name]}` references are inlined;
- cumulative and window metrics order and partition by each dimension at its
`timeGrain`, as the query groups it.

Each expression is written in `ANSI_SQL` and, where it has a form in Apache
Ossie's own expression language, also in `OSSIE_SQL_2026`. That form uses the
language's functions where they differ from ANSI (`CONTAINS`, `STARTSWITH` and
`ENDSWITH` instead of `LIKE ... ESCAPE`, and a `CAST` for a timestamp literal) and
is checked against `core-spec/expression_language.md`, both as written and as
parsed with sqlglot. A measure with no such form (`MODE`, a pattern that needs
`ESCAPE`, an authored expression outside the language) keeps `ANSI_SQL` only.

A measure or metric with no faithful single expression (period-over-period,
`grain`, `filterContext`, `anchor`, nested windows, reference cycles) is left out
of the Ossie metrics with a warning and kept whole in the model-level extension
(`obml_unexported`). Every exported measure and metric also carries its OBML
definition, and every field its OBML column name, so Ossie to OBML restores the
original model instead of re-parsing the SQL.
definition, and every field is named by its OBML column name with the column
`code` as its expression, so Ossie to OBML restores the original model instead of
re-parsing the SQL. Documents from earlier versions, which named fields by `code`,
still import.

`validate_ossie` checks JSON Schema conformance, unique names, and relationship
references. If the schema file or `jsonschema` package is unavailable, it emits
Expand Down
6 changes: 3 additions & 3 deletions converters/orionbelt/ossie_obml_mapping_analysis.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@
- **Ossie** uses snake_case codes everywhere (`name: "store_sales"`)
- **OBML** supports dual naming — a display name as the dictionary key and a `code` for the physical SQL reference

During Ossie → OBML conversion, field names are used directly as both the display name and code. During OBML → Ossie conversion, the `code` value becomes the Ossie field `name`, and the OBML column name rides in the field's `custom_extensions` (`obml_column_name`) so the reverse trip restores it. Metric expressions reference columns as `"<dataset>"."<field>"`: the OBML data object name and the column code, always double-quoted so reserved words parse and names match exactly. Cumulative and window metrics order and partition by each dimension exactly as the query groups it: truncated to its `timeGrain` (`DATE_TRUNC`) and cast back to a temporal `resultType`. A reference cycle among metrics is left out with a warning.
During Ossie → OBML conversion, field names are used directly as both the display name and code. During OBML → Ossie conversion, the OBML column name becomes the Ossie field `name` and the column `code` its expression, so a calculated column imported from Ossie (`amount_with_tax`, `amount * 1.1`) keeps its name; `primary_key`, `unique_keys` and relationship columns use the same names. Documents written by earlier versions, which named each field by its `code` and kept the column name in `obml_column_name`, still import, and the unique keys and unconverted metrics they carry are renamed with the fields. Metric expressions reference columns as `<dataset>.<field>`, the OBML data object and column names: following the expression language's identifier rules (a regular identifier is normalized to upper case, a quoted one is taken as written): bare where the name can be a regular identifier, double-quoted as written for a space, a leading digit or a reserved word, quoted in upper case for a regular name DuckDB or sqlglot will not read bare (`"QUALIFY"`), and quoted as written, with a warning, for names that differ only by case. The import resolves references by the same rules and keeps a metric whose reference matches no field, or more than one, verbatim. Cumulative and window metrics order and partition by each dimension exactly as the query groups it: truncated to its `timeGrain` (`DATE_TRUNC`) and cast back to a temporal `resultType`. A reference cycle among metrics is left out with a warning.

### 2.2 Relationship Placement

Expand Down Expand Up @@ -151,7 +151,7 @@ These OBML features have no direct Ossie equivalent. Where possible, metadata is

- **`primary_key`** — natively represented: Ossie's dataset-level `primary_key` array maps to per-column `primaryKey: true` on OBML columns (`DataObjectColumn.primaryKey`), and back to the dataset array on export.
- **`unique_keys`** — no native OBML equivalent; round-trips via an `Ossie`-vendor `customExtension` (`obml_unique_keys`).
- **Multi-dialect expressions** — on import the converter reads the first available SQL dialect in the order `ANSI_SQL`, `OSSIE_SQL_2026`, `SNOWFLAKE`, `DATABRICKS`; non-SQL dialects (`MDX`, `TABLEAU`, `MAQL`, `SIGMA`, `THOUGHTSPOT`, `DAX`) are not parsed. A metric with no SQL-parseable dialect, or an expression OBML cannot decompose, is preserved verbatim (`obml_unconverted_metrics`) with a `LOSSY:` warning rather than dropped. On export, OBML measures/metrics emit `ANSI_SQL`, except `aggregation: measure`, whose `MEASURE("<name>")` call is tagged `DATABRICKS`.
- **Multi-dialect expressions** — on import the converter reads the first available SQL dialect in the order `ANSI_SQL`, `OSSIE_SQL_2026`, `SNOWFLAKE`, `DATABRICKS`; non-SQL dialects (`MDX`, `TABLEAU`, `MAQL`, `SIGMA`, `THOUGHTSPOT`, `DAX`) are not parsed. A metric with no SQL-parseable dialect, or an expression OBML cannot decompose, is preserved verbatim (`obml_unconverted_metrics`) with a `LOSSY:` warning rather than dropped, as is an expression containing a query (a subquery or a `UNION`), whose own tables and aliases are not datasets. On export, OBML measures/metrics emit `ANSI_SQL`, except `aggregation: measure`, whose `MEASURE("<name>")` call is tagged `DATABRICKS`. Each `ANSI_SQL` expression is also written in `OSSIE_SQL_2026` where it has a form in that language (`CONTAINS`, `STARTSWITH` and `ENDSWITH` instead of `LIKE ... ESCAPE`, a `CAST` for a timestamp literal), checked against `core-spec/expression_language.md` as written (every function, `CAST` type and typed literal is one the spec lists, no foreign operator such as `::`) and as parsed (no construct the spec does not list); a measure using `MODE`, a pattern that needs `ESCAPE`, or an authored expression outside the language keeps `ANSI_SQL` only.
- **`ai_context`** — preserved losslessly via `customExtensions` (see Section 2.4).
- **`custom_extensions`** — mapped to OBML `customExtensions`.

Expand All @@ -169,7 +169,7 @@ These OBML features have no direct Ossie equivalent. Where possible, metadata is
### 3.2 OBML → Ossie

1. Combine `database.schema.code` into the Ossie `source` string
2. Convert columns to fields with `ANSI_SQL` dialect expressions
2. Convert columns to fields named by the OBML column name, with the column `code` as an `ANSI_SQL` expression
3. Extract inline joins into global relationships with generated names
4. Convert measures to Ossie metrics with SQL expressions that compute what OrionBelt computes (filters, totals, defaults spelled out)
5. Expand metric templates by substituting measure, synthesized-count and metric SQL into `{[Name]}` references
Expand Down
13 changes: 7 additions & 6 deletions converters/orionbelt/ossie_obml_ontology_mapping_analysis.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,10 +52,10 @@ ontology_mappings:
semantic_model: { ...full Ossie core-spec model... } # reused from OBMLtoOssie
concept_mappings:
- concept: <Entity>
object_mappings: [{ expression: "<table>.<key_col>" }]
object_mappings: [{ expression: "<Entity>.<key_field>" }]
link_mappings:
- relationship: <A>_to_<B>
object_mapping: { concept: <B>, expression: "<table_A>.<fk_col>" }
object_mapping: { concept: <B>, expression: "<A>.<fk_field>" }
```

## Mapping rules
Expand All @@ -66,12 +66,13 @@ ontology_mappings:
| `dataObject.description` / `.comment` | `concept.description` | first non-empty wins |
| `join` (A → B) | `Relationship` under A's component | declaring concept A is the implicit first role; B is an explicit `role` |
| `join.joinType` | `Relationship.multiplicity` | `many-to-one`→`ManyToOne`, `one-to-one`→`OneToOne` |
| `column.primaryKey` | entity `object_mappings[].expression` | `<table>.<pk_code>`; identifies the entity |
| `join.columnsFrom` (FK) | `link_mappings[].object_mapping.expression` | `<table_A>.<fk_code>`; binds the relationship to its far role |
| `column.primaryKey` | entity `object_mappings[].expression` | `<dataset>.<pk_field>`; identifies the entity |
| `join.columnsFrom` (FK) | `link_mappings[].object_mapping.expression` | `<dataset_A>.<fk_field>`; binds the relationship to its far role |
| whole core model | `ontology_mappings[].semantic_model` | complete output from `OBMLtoOssie.convert()`|

`<table>` is the final identifier of the dataset `source` (e.g. `db.schema.t` → `t`),
falling back to the dataset name when `source` has no dotted physical table.
Mapping expressions are SQL over the embedded model's dataset fields: `<dataset>` is
the dataset name (the OBML data object name) and `<field>` the field name (the OBML
column name), quoted as in the model's metrics, e.g. `Orders."Order ID"`.

## Gaps and warnings (emitted to `warnings`)

Expand Down
5 changes: 5 additions & 0 deletions converters/orionbelt/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,11 @@ dependencies = [
"pyyaml>=6.0",
"jsonschema>=4.18",
"referencing>=0.30",
# Runtime: the OSSIE_SQL_2026 check reads an expression's tokens and tree,
# quoting asks whether a name parses bare, and field renames in preserved
# metrics edit tokens. Capped below 31 because the check depends on token
# types and on how sqlglot spells normalized functions.
"sqlglot>=30,<31",
]

[project.optional-dependencies]
Expand Down
9 changes: 9 additions & 0 deletions converters/orionbelt/src/ossie_orionbelt/_common.py
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,15 @@
# THOUGHTSPOT, DAX) are never parsed as SQL.
_SQL_PARSEABLE_DIALECTS = ("ANSI_SQL", "OSSIE_SQL_2026", "SNOWFLAKE", "DATABRICKS")

# Ossie SQL-dialect tag -> sqlglot ``read`` dialect. ANSI and the portable
# OSSIE_SQL_2026 map to sqlglot's default (None).
_OSSIE_DIALECT_TO_SQLGLOT: dict[str, str | None] = {
"ANSI_SQL": None,
"OSSIE_SQL_2026": None,
"SNOWFLAKE": "snowflake",
"DATABRICKS": "databricks",
}

# Matches a ``dataset.column`` reference inside a SQL expression, where each
# side is a bare identifier or a quoted identifier (double quotes, backticks, or
# brackets). The leading lookbehind prevents matching the tail of a longer path
Expand Down
Loading
Loading