diff --git a/converters/orionbelt/README.md b/converters/orionbelt/README.md index a038ce85..1dacc28e 100644 --- a/converters/orionbelt/README.md +++ b/converters/orionbelt/README.md @@ -97,19 +97,38 @@ obml_again = OssietoOBML(ossie).convert() Other Ossie consumers read a metric's SQL, not the `ORIONBELT` extension, so the OBML to Ossie export writes SQL that computes what OrionBelt computes: -- columns are referenced as `"".""`, always double-quoted; +- columns are referenced as `.`, the OBML data object and column + names, following the expression language's identifier rules: a regular + identifier is normalized to upper case, a quoted one is taken as written. A + name is bare where it can be a regular identifier. A name that cannot (a + space, a leading digit, a reserved word such as `Order`) is double-quoted as + written; a regular name DuckDB or sqlglot will not read bare, such as + `Qualify`, is quoted in its normalized form, `"QUALIFY"`. Names that differ + only by case are one identifier to the language: they are quoted as written, + with a warning. The import resolves references by the same rules, and keeps a + metric whose reference matches no field, or more than one, verbatim; - measure filters become `AGG(CASE WHEN THEN END)`, totals the grand-total window `SUM(SUM(x)) OVER ()`, and `defaultValue` a `COALESCE`; - synthesized counts and metric-on-metric `{[Name]}` references are inlined; - cumulative and window metrics order and partition by each dimension at its `timeGrain`, as the query groups it. +Each expression is written in `ANSI_SQL` and, where it has a form in Apache +Ossie's own expression language, also in `OSSIE_SQL_2026`. That form uses the +language's functions where they differ from ANSI (`CONTAINS`, `STARTSWITH` and +`ENDSWITH` instead of `LIKE ... ESCAPE`, and a `CAST` for a timestamp literal) and +is checked against `core-spec/expression_language.md`, both as written and as +parsed with sqlglot. A measure with no such form (`MODE`, a pattern that needs +`ESCAPE`, an authored expression outside the language) keeps `ANSI_SQL` only. + A measure or metric with no faithful single expression (period-over-period, `grain`, `filterContext`, `anchor`, nested windows, reference cycles) is left out of the Ossie metrics with a warning and kept whole in the model-level extension (`obml_unexported`). Every exported measure and metric also carries its OBML -definition, and every field its OBML column name, so Ossie to OBML restores the -original model instead of re-parsing the SQL. +definition, and every field is named by its OBML column name with the column +`code` as its expression, so Ossie to OBML restores the original model instead of +re-parsing the SQL. Documents from earlier versions, which named fields by `code`, +still import. `validate_ossie` checks JSON Schema conformance, unique names, and relationship references. If the schema file or `jsonschema` package is unavailable, it emits diff --git a/converters/orionbelt/ossie_obml_mapping_analysis.md b/converters/orionbelt/ossie_obml_mapping_analysis.md index 7f598b64..388e6847 100644 --- a/converters/orionbelt/ossie_obml_mapping_analysis.md +++ b/converters/orionbelt/ossie_obml_mapping_analysis.md @@ -46,7 +46,7 @@ - **Ossie** uses snake_case codes everywhere (`name: "store_sales"`) - **OBML** supports dual naming — a display name as the dictionary key and a `code` for the physical SQL reference -During Ossie → OBML conversion, field names are used directly as both the display name and code. During OBML → Ossie conversion, the `code` value becomes the Ossie field `name`, and the OBML column name rides in the field's `custom_extensions` (`obml_column_name`) so the reverse trip restores it. Metric expressions reference columns as `"".""`: the OBML data object name and the column code, always double-quoted so reserved words parse and names match exactly. Cumulative and window metrics order and partition by each dimension exactly as the query groups it: truncated to its `timeGrain` (`DATE_TRUNC`) and cast back to a temporal `resultType`. A reference cycle among metrics is left out with a warning. +During Ossie → OBML conversion, field names are used directly as both the display name and code. During OBML → Ossie conversion, the OBML column name becomes the Ossie field `name` and the column `code` its expression, so a calculated column imported from Ossie (`amount_with_tax`, `amount * 1.1`) keeps its name; `primary_key`, `unique_keys` and relationship columns use the same names. Documents written by earlier versions, which named each field by its `code` and kept the column name in `obml_column_name`, still import, and the unique keys and unconverted metrics they carry are renamed with the fields. Metric expressions reference columns as `.`, the OBML data object and column names: following the expression language's identifier rules (a regular identifier is normalized to upper case, a quoted one is taken as written): bare where the name can be a regular identifier, double-quoted as written for a space, a leading digit or a reserved word, quoted in upper case for a regular name DuckDB or sqlglot will not read bare (`"QUALIFY"`), and quoted as written, with a warning, for names that differ only by case. The import resolves references by the same rules and keeps a metric whose reference matches no field, or more than one, verbatim. Cumulative and window metrics order and partition by each dimension exactly as the query groups it: truncated to its `timeGrain` (`DATE_TRUNC`) and cast back to a temporal `resultType`. A reference cycle among metrics is left out with a warning. ### 2.2 Relationship Placement @@ -151,7 +151,7 @@ These OBML features have no direct Ossie equivalent. Where possible, metadata is - **`primary_key`** — natively represented: Ossie's dataset-level `primary_key` array maps to per-column `primaryKey: true` on OBML columns (`DataObjectColumn.primaryKey`), and back to the dataset array on export. - **`unique_keys`** — no native OBML equivalent; round-trips via an `Ossie`-vendor `customExtension` (`obml_unique_keys`). -- **Multi-dialect expressions** — on import the converter reads the first available SQL dialect in the order `ANSI_SQL`, `OSSIE_SQL_2026`, `SNOWFLAKE`, `DATABRICKS`; non-SQL dialects (`MDX`, `TABLEAU`, `MAQL`, `SIGMA`, `THOUGHTSPOT`, `DAX`) are not parsed. A metric with no SQL-parseable dialect, or an expression OBML cannot decompose, is preserved verbatim (`obml_unconverted_metrics`) with a `LOSSY:` warning rather than dropped. On export, OBML measures/metrics emit `ANSI_SQL`, except `aggregation: measure`, whose `MEASURE("")` call is tagged `DATABRICKS`. +- **Multi-dialect expressions** — on import the converter reads the first available SQL dialect in the order `ANSI_SQL`, `OSSIE_SQL_2026`, `SNOWFLAKE`, `DATABRICKS`; non-SQL dialects (`MDX`, `TABLEAU`, `MAQL`, `SIGMA`, `THOUGHTSPOT`, `DAX`) are not parsed. A metric with no SQL-parseable dialect, or an expression OBML cannot decompose, is preserved verbatim (`obml_unconverted_metrics`) with a `LOSSY:` warning rather than dropped, as is an expression containing a query (a subquery or a `UNION`), whose own tables and aliases are not datasets. On export, OBML measures/metrics emit `ANSI_SQL`, except `aggregation: measure`, whose `MEASURE("")` call is tagged `DATABRICKS`. Each `ANSI_SQL` expression is also written in `OSSIE_SQL_2026` where it has a form in that language (`CONTAINS`, `STARTSWITH` and `ENDSWITH` instead of `LIKE ... ESCAPE`, a `CAST` for a timestamp literal), checked against `core-spec/expression_language.md` as written (every function, `CAST` type and typed literal is one the spec lists, no foreign operator such as `::`) and as parsed (no construct the spec does not list); a measure using `MODE`, a pattern that needs `ESCAPE`, or an authored expression outside the language keeps `ANSI_SQL` only. - **`ai_context`** — preserved losslessly via `customExtensions` (see Section 2.4). - **`custom_extensions`** — mapped to OBML `customExtensions`. @@ -169,7 +169,7 @@ These OBML features have no direct Ossie equivalent. Where possible, metadata is ### 3.2 OBML → Ossie 1. Combine `database.schema.code` into the Ossie `source` string -2. Convert columns to fields with `ANSI_SQL` dialect expressions +2. Convert columns to fields named by the OBML column name, with the column `code` as an `ANSI_SQL` expression 3. Extract inline joins into global relationships with generated names 4. Convert measures to Ossie metrics with SQL expressions that compute what OrionBelt computes (filters, totals, defaults spelled out) 5. Expand metric templates by substituting measure, synthesized-count and metric SQL into `{[Name]}` references diff --git a/converters/orionbelt/ossie_obml_ontology_mapping_analysis.md b/converters/orionbelt/ossie_obml_ontology_mapping_analysis.md index 5d39e182..a1dbe35e 100644 --- a/converters/orionbelt/ossie_obml_ontology_mapping_analysis.md +++ b/converters/orionbelt/ossie_obml_ontology_mapping_analysis.md @@ -52,10 +52,10 @@ ontology_mappings: semantic_model: { ...full Ossie core-spec model... } # reused from OBMLtoOssie concept_mappings: - concept: - object_mappings: [{ expression: "." }] + object_mappings: [{ expression: "." }] link_mappings: - relationship: _to_ - object_mapping: { concept: , expression: "." } + object_mapping: { concept: , expression: "." } ``` ## Mapping rules @@ -66,12 +66,13 @@ ontology_mappings: | `dataObject.description` / `.comment` | `concept.description` | first non-empty wins | | `join` (A → B) | `Relationship` under A's component | declaring concept A is the implicit first role; B is an explicit `role` | | `join.joinType` | `Relationship.multiplicity` | `many-to-one`→`ManyToOne`, `one-to-one`→`OneToOne` | -| `column.primaryKey` | entity `object_mappings[].expression` | `
.`; identifies the entity | -| `join.columnsFrom` (FK) | `link_mappings[].object_mapping.expression` | `.`; binds the relationship to its far role | +| `column.primaryKey` | entity `object_mappings[].expression` | `.`; identifies the entity | +| `join.columnsFrom` (FK) | `link_mappings[].object_mapping.expression` | `.`; binds the relationship to its far role | | whole core model | `ontology_mappings[].semantic_model` | complete output from `OBMLtoOssie.convert()`| -`
` is the final identifier of the dataset `source` (e.g. `db.schema.t` → `t`), -falling back to the dataset name when `source` has no dotted physical table. +Mapping expressions are SQL over the embedded model's dataset fields: `` is +the dataset name (the OBML data object name) and `` the field name (the OBML +column name), quoted as in the model's metrics, e.g. `Orders."Order ID"`. ## Gaps and warnings (emitted to `warnings`) diff --git a/converters/orionbelt/pyproject.toml b/converters/orionbelt/pyproject.toml index d26b7ed4..715ba15c 100644 --- a/converters/orionbelt/pyproject.toml +++ b/converters/orionbelt/pyproject.toml @@ -45,6 +45,11 @@ dependencies = [ "pyyaml>=6.0", "jsonschema>=4.18", "referencing>=0.30", + # Runtime: the OSSIE_SQL_2026 check reads an expression's tokens and tree, + # quoting asks whether a name parses bare, and field renames in preserved + # metrics edit tokens. Capped below 31 because the check depends on token + # types and on how sqlglot spells normalized functions. + "sqlglot>=30,<31", ] [project.optional-dependencies] diff --git a/converters/orionbelt/src/ossie_orionbelt/_common.py b/converters/orionbelt/src/ossie_orionbelt/_common.py index 9330874e..8f342b8f 100644 --- a/converters/orionbelt/src/ossie_orionbelt/_common.py +++ b/converters/orionbelt/src/ossie_orionbelt/_common.py @@ -42,6 +42,15 @@ # THOUGHTSPOT, DAX) are never parsed as SQL. _SQL_PARSEABLE_DIALECTS = ("ANSI_SQL", "OSSIE_SQL_2026", "SNOWFLAKE", "DATABRICKS") +# Ossie SQL-dialect tag -> sqlglot ``read`` dialect. ANSI and the portable +# OSSIE_SQL_2026 map to sqlglot's default (None). +_OSSIE_DIALECT_TO_SQLGLOT: dict[str, str | None] = { + "ANSI_SQL": None, + "OSSIE_SQL_2026": None, + "SNOWFLAKE": "snowflake", + "DATABRICKS": "databricks", +} + # Matches a ``dataset.column`` reference inside a SQL expression, where each # side is a bare identifier or a quoted identifier (double quotes, backticks, or # brackets). The leading lookbehind prevents matching the tail of a longer path diff --git a/converters/orionbelt/src/ossie_orionbelt/_field_names.py b/converters/orionbelt/src/ossie_orionbelt/_field_names.py new file mode 100644 index 00000000..0b2f1fda --- /dev/null +++ b/converters/orionbelt/src/ossie_orionbelt/_field_names.py @@ -0,0 +1,258 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +"""Field names in what the converter keeps verbatim. + +An Ossie field is named by its OBML column name. Earlier exports named it by +the column ``code``, so the unique keys and the unconverted metrics +an import keeps verbatim (Ossie-vendor extensions) can name fields by code. + +The import renames them with the exact mapping an older export left in +``obml_column_name`` and marks each payload with :data:`FIELD_NAMES_KEY`. The +export renames an unmarked payload, from OBML saved by an older version, from +the column codes; a code that is also another column's name cannot be told +apart and is kept with a warning. + +In a metric's SQL only the characters of each ``.`` field are +replaced: regenerating the SQL from a parsed tree would drop what sqlglot does +not keep, such as a weighted percentile's third argument. The parsed tree +tells whether a query in the SQL declares the qualifier as its own table or +alias. +""" + +from __future__ import annotations + +import re +from collections.abc import Callable, Collection, Iterable +from typing import Any + +import sqlglot +from sqlglot import expressions as exp +from sqlglot.tokens import Token, TokenType + +from ossie_orionbelt._common import _OSSIE_DIALECT_TO_SQLGLOT, _SQL_PARSEABLE_DIALECTS +from ossie_orionbelt._portable import case_collisions, quoted_name + +# Marks a payload whose field references are column names. +FIELD_NAMES_KEY = "obml_field_names" +FIELD_NAMES_COLUMN = "column" + +# data object -> old field name -> column name +Renames = dict[str, dict[str, str]] +# data object -> column names that equal another of its columns regardless of case +Collisions = dict[str, frozenset[str]] +Warn = Callable[[str], None] + + +def code_renames(data_objects: dict[str, Any]) -> tuple[Renames, dict[str, set[str]]]: + """Column ``code`` -> column name per data object, and the ambiguous codes. + + A code is ambiguous when it is also the name of another column, or the + code of more than one column: an older reference to it cannot be resolved. + """ + renames: Renames = {} + ambiguous: dict[str, set[str]] = {} + for do_name, do_obj in data_objects.items(): + columns = do_obj.get("columns") or {} + seen: dict[str, str] = {} + for col_name, col in columns.items(): + code = col.get("code") if isinstance(col, dict) else None + if not isinstance(code, str) or code == col_name: + continue + if code in columns or code in seen: + ambiguous.setdefault(do_name, set()).add(code) + seen[code] = col_name + mapping = {c: n for c, n in seen.items() if c not in ambiguous.get(do_name, set())} + if mapping: + renames[do_name] = mapping + return renames, ambiguous + + +def rename_keys( + groups: list[Any], + renames: dict[str, str], + ambiguous: Collection[str], + warn: Warn, + where: str, +) -> list[Any]: + """Unique-key column groups with each old field name renamed.""" + for name in {c for g in groups for c in g if c in ambiguous}: + warn( + f"{where}: unique key column '{name}' is both a column name and another " + f"column's code; kept as the column name." + ) + return [[renames.get(c, c) for c in group] for group in groups] + + +def field_collisions(data_objects: dict[str, Any]) -> Collisions: + """Per data object, the column names a bare reference could not tell apart.""" + return { + name: case_collisions(obj.get("columns") or {}) + for name, obj in data_objects.items() + if isinstance(obj, dict) + } + + +def rename_metric( + metric: dict[str, Any], + renames: Renames, + ambiguous: dict[str, set[str]], + collisions: Collisions, + warn: Warn, +) -> None: + """Rename the field references in each SQL expression of *metric*, in place. + + A preserved metric is one the converter could not handle, so anything not + shaped like a metric expression is left as it is. + """ + expression = metric.get("expression") + if not isinstance(expression, dict): + return + for entry in expression.get("dialects") or []: + if not isinstance(entry, dict): + continue + dialect = entry.get("dialect") + sql = entry.get("expression") + if dialect not in _SQL_PARSEABLE_DIALECTS or not isinstance(sql, str): + continue + where = f"Preserved metric '{metric.get('name')}' ({dialect})" + entry["expression"] = rename_sql(sql, dialect, renames, ambiguous, collisions, warn, where) + + +def rename_sql( + sql: str, + dialect: str, + renames: Renames, + ambiguous: dict[str, set[str]], + collisions: Collisions, + warn: Warn, + where: str, +) -> str: + """*sql* with each ``.`` reference renamed. + + The references are found on the tokens, which keep what the parsed tree + drops; the tree tells whether a query in the SQL declares the qualifier + itself. A quoted old name matches exactly, as the older exports that wrote + it quoted every name as written; an unquoted one matches regardless of + case. A new name is written as :func:`quoted_name` says, *collisions* + being the names of its data object that differ only by case. SQL that + does not parse is kept. + """ + datasets = set(renames) | set(ambiguous) + if not datasets: + return sql + read = _OSSIE_DIALECT_TO_SQLGLOT.get(dialect) + parsed = parse_expression(sql, read) + if parsed is None: + warn(f"{where} does not parse; its field references were not renamed.") + return sql + text, tree = parsed + columns = { + c.this.meta["start"]: c + for c in tree.find_all(exp.Column) + if isinstance(c.this, exp.Identifier) and "start" in c.this.meta + } + has_queries = tree.find(exp.Select) is not None + tokens = sqlglot.tokenize(text, read=read) + edits: list[tuple[int, int, str, str]] = [] + for i in range(len(tokens) - 2): + table, dot, field = tokens[i : i + 3] + if dot.token_type is not TokenType.DOT: + continue + # Only a two-part reference: ``a.b.c`` is not ``.``. + if i > 0 and tokens[i - 1].token_type is TokenType.DOT: + continue + if i + 3 < len(tokens) and tokens[i + 3].token_type is TokenType.DOT: + continue + do_name = _match(table, datasets, warn, where) + if do_name is None: + continue + if _match(field, ambiguous.get(do_name, set()), warn, where) is not None: + warn( + f"{where}: '{field.text}' is both a column name and another column's " + f"code; the reference was not renamed." + ) + continue + old = _match(field, renames.get(do_name, {}), warn, where) + if old is None: + continue + column = columns.get(field.start) + if column is None and has_queries: + warn(f"{where}: the scope of '{table.text}.{field.text}' is unknown; not renamed.") + continue + if column is not None and _declared_in_scope(column, table.text): + continue + edits.append((field.start, field.end, renames[do_name][old], do_name)) + for start, end, name, do_name in sorted(edits, reverse=True): + quoted = quoted_name(name, collisions.get(do_name, frozenset())) + if quoted is None: + ident = name + elif sql[start] == "[": + ident = f"[{quoted}]" + else: + ident = exp.to_identifier(quoted, quoted=True).sql(dialect=read) + sql = sql[:start] + ident + sql[end + 1 :] + return sql + + +def parse_expression(sql: str, read: str | None) -> tuple[str, exp.Expr] | None: + """The text parsed and its tree, retrying with ``[name]`` read as ``"name"`` + like the import. + + Both spellings have the same length (a name holding a double quote stays + bracketed), so positions in the parsed text address the original. + """ + try: + return sql, sqlglot.parse_one(sql, read=read) + except sqlglot.errors.SqlglotError: + pass + normalized = re.sub(r"\[([^\]\"]+)\]", r'"\1"', sql) + if normalized == sql: + return None + try: + return normalized, sqlglot.parse_one(normalized, read=read) + except sqlglot.errors.SqlglotError: + return None + + +def _match(token: Token, names: Iterable[str], warn: Warn, where: str) -> str | None: + """The name *token* refers to: exactly when quoted, else ignoring case.""" + if token.token_type is TokenType.IDENTIFIER: + return token.text if token.text in names else None + hits = [n for n in names if n.lower() == token.text.lower()] + if len(hits) > 1: + warn(f"{where}: unquoted '{token.text}' matches {sorted(hits)}; not renamed.") + return hits[0] if len(hits) == 1 else None + + +def _declared_in_scope(node: exp.Expr, name: str) -> bool: + """Whether a query around *node* declares a table, alias or CTE *name*.""" + lowered = name.lower() + parent = node.parent + while parent is not None: + if isinstance(parent, exp.Select): + sources: list[exp.Expr] = [] + if parent.args.get("from_") is not None: + sources.append(parent.args["from_"].this) + sources.extend(join.this for join in parent.args.get("joins") or []) + with_ = parent.args.get("with_") + if with_ is not None: + sources.extend(with_.expressions) + if any(s.alias_or_name.lower() == lowered for s in sources): + return True + parent = parent.parent + return False diff --git a/converters/orionbelt/src/ossie_orionbelt/_ossie_sql.py b/converters/orionbelt/src/ossie_orionbelt/_ossie_sql.py new file mode 100644 index 00000000..0f87f7cc --- /dev/null +++ b/converters/orionbelt/src/ossie_orionbelt/_ossie_sql.py @@ -0,0 +1,346 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +"""Whether an expression stays inside Apache Ossie's ``OSSIE_SQL_2026`` language. + +The export tags an expression ``OSSIE_SQL_2026`` only when it is written in the +language ``core-spec/expression_language.md`` defines, so the tag is a claim +another tool can rely on. The check has two layers: + +* **as written**, on the tokens: every function called is one the spec defines + at the REQUIRED or RECOMMENDED level, every ``CAST`` target is a listed type, + a typed literal is ``DATE``, ``TIME`` or ``TIMESTAMP_NTZ``, and no operator + the spec lacks (``::``, ``->``, ``[ ]``, ...) appears. This layer exists + because sqlglot normalizes while parsing: it reads ``SAFE_CAST`` as + ``TRY_CAST`` and ``INT64`` as ``BIGINT``, yet the export writes the original. +* **as parsed**, on the sqlglot tree: every node is a construct the spec lists + or one its functions parse into, so a subquery, an ``INTERVAL`` or a bare + ``CURRENT_USER`` is rejected; ``DISTINCT`` appears only in ``SUM`` and + ``COUNT``; date parts are listed ones. +""" + +from __future__ import annotations + +import re +from functools import cache + +import sqlglot +from sqlglot import expressions as exp +from sqlglot.tokens import Token, TokenType + +# Every function the spec defines at the REQUIRED or RECOMMENDED level, with a +# sample call. The names are what an expression may call, as written; parsing +# the samples gives the sqlglot nodes those calls produce. EXPERIMENTAL forms +# (TO_CHAR, TO_DATE / TO_TIMESTAMP with a format) are not in the language. +_FUNCTIONS: dict[str, str] = { + # Aggregation + "SUM": "SUM(x)", + "COUNT": "COUNT(x) + COUNT(*) + COUNT(DISTINCT x)", + "AVG": "AVG(x)", + "MIN": "MIN(x)", + "MAX": "MAX(x)", + "STDDEV": "STDDEV(x)", + "STDDEV_POP": "STDDEV_POP(x)", + "STDDEV_SAMP": "STDDEV_SAMP(x)", + "VARIANCE": "VARIANCE(x)", + "VAR_POP": "VAR_POP(x)", + "VAR_SAMP": "VAR_SAMP(x)", + "MEDIAN": "MEDIAN(x)", + "PERCENTILE_CONT": "PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY x)", + "PERCENTILE_DISC": "PERCENTILE_DISC(0.5) WITHIN GROUP (ORDER BY x DESC)", + "APPROX_COUNT_DISTINCT": "APPROX_COUNT_DISTINCT(x)", + "APPROX_PERCENTILE": "APPROX_PERCENTILE(x, 0.5)", + # Date and time + "CURRENT_DATE": "CURRENT_DATE() + CURRENT_DATE", + "CURRENT_TIMESTAMP": "CURRENT_TIMESTAMP() + CURRENT_TIMESTAMP", + "CURRENT_TIME": "CURRENT_TIME() + CURRENT_TIME", + "YEAR": "YEAR(d)", + "QUARTER": "QUARTER(d)", + "MONTH": "MONTH(d)", + "DAY": "DAY(d)", + "DAYOFYEAR": "DAYOFYEAR(d)", + "HOUR": "HOUR(d)", + "MINUTE": "MINUTE(d)", + "SECOND": "SECOND(d)", + "EXTRACT": "EXTRACT(YEAR FROM d)", + "DATE_PART": "DATE_PART('year', d)", + "DATE_TRUNC": "DATE_TRUNC('month', d)", + "DATEADD": "DATEADD(day, 7, d)", + "DATEDIFF": "DATEDIFF(day, a, b)", + "TO_DATE": "TO_DATE(s)", + "TO_TIMESTAMP": "TO_TIMESTAMP(s)", + # Strings + "CONCAT": "CONCAT(a, b) || c", + "LENGTH": "LENGTH(s)", + "LOWER": "LOWER(s)", + "UPPER": "UPPER(s)", + "TRIM": "TRIM(s)", + "LTRIM": "LTRIM(s)", + "RTRIM": "RTRIM(s)", + "LEFT": "LEFT(s, 2)", + "RIGHT": "RIGHT(s, 2)", + "SUBSTRING": "SUBSTRING(s, 1, 2)", + "REPLACE": "REPLACE(s, 'a', 'b')", + "SPLIT_PART": "SPLIT_PART(s, ',', 1)", + "POSITION": "POSITION('a' IN s)", + "CHARINDEX": "CHARINDEX('a', s)", + "CONTAINS": "CONTAINS(s, 'a')", + "STARTSWITH": "STARTSWITH(s, 'a')", + "ENDSWITH": "ENDSWITH(s, 'a')", + "REGEXP_LIKE": "REGEXP_LIKE(s, 'a')", + "REGEXP_EXTRACT": "REGEXP_EXTRACT(s, 'a')", + "REGEXP_REPLACE": "REGEXP_REPLACE(s, 'a', 'b')", + "REGEXP_COUNT": "REGEXP_COUNT(s, 'a')", + # Math + "ABS": "ABS(x)", + "ROUND": "ROUND(x, 2)", + "FLOOR": "FLOOR(x)", + "CEIL": "CEIL(x)", + "CEILING": "CEILING(x)", + "TRUNC": "TRUNC(x, 2)", + "TRUNCATE": "TRUNCATE(x, 2)", + "MOD": "MOD(x, y)", + "SIGN": "SIGN(x)", + "POWER": "POWER(x, y)", + "SQRT": "SQRT(x)", + "EXP": "EXP(x)", + "LN": "LN(x)", + "LOG": "LOG(2, x)", + "LOG10": "LOG10(x)", + "SIN": "SIN(x)", + "COS": "COS(x)", + "TAN": "TAN(x)", + "ASIN": "ASIN(x)", + "ACOS": "ACOS(x)", + "ATAN": "ATAN(x)", + "ATAN2": "ATAN2(y, x)", + "RADIANS": "RADIANS(x)", + "DEGREES": "DEGREES(x)", + "PI": "PI()", + "GREATEST": "GREATEST(a, b)", + "LEAST": "LEAST(a, b)", + # Conditional + "IF": "IF(c, a, b)", + "IFF": "IFF(c, a, b)", + "NULLIF": "NULLIF(a, b)", + "COALESCE": "COALESCE(a, b)", + "IFNULL": "IFNULL(a, b)", + "NVL": "NVL(a, b)", + "NVL2": "NVL2(a, b, c)", + "ZEROIFNULL": "ZEROIFNULL(a)", + "NULLIFZERO": "NULLIFZERO(a)", + # Window + "ROW_NUMBER": "ROW_NUMBER() OVER ()", + "RANK": "RANK() OVER (ORDER BY x)", + "DENSE_RANK": "DENSE_RANK() OVER (ORDER BY x)", + "NTILE": "NTILE(4) OVER (ORDER BY x)", + "PERCENT_RANK": "PERCENT_RANK() OVER (ORDER BY x)", + "CUME_DIST": "CUME_DIST() OVER (ORDER BY x)", + "LAG": "LAG(x, 1, 0) OVER (ORDER BY d)", + "LEAD": "LEAD(x, 1, 0) OVER (ORDER BY d)", + "FIRST_VALUE": "FIRST_VALUE(x) OVER (ORDER BY d)", + "LAST_VALUE": "LAST_VALUE(x) OVER (ORDER BY d)", + "NTH_VALUE": "NTH_VALUE(x, 2) OVER (ORDER BY d)", + # Type conversion + "CAST": "CAST(x AS INT)", + "TRY_CAST": "TRY_CAST(x AS INT)", +} + +# Syntax the spec lists that is not a function call: references, literals, +# operators, CASE, IN, BETWEEN, IS [NOT] DISTINCT FROM, the window clauses. +_CONSTRUCTS = ( + "SUM(CASE WHEN a = 1 AND NOT b <> 2 OR c IS NULL THEN -d * 2 / 3 % 4 - 5 + 6 END)" + " + COUNT(CASE WHEN a > 1 AND b >= 2 AND c < 3 AND d <= 4 THEN 1 ELSE NULL END)" + " + (a BETWEEN 1 AND 2) + (a IN (1, 2)) + (a LIKE 'x') + (a ILIKE 'x')" + " + (a IS NOT DISTINCT FROM b) + (a IS DISTINCT FROM b) + (a = TRUE)" + " + SUM(x) OVER (PARTITION BY a ORDER BY b DESC NULLS LAST" + " ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)" + " + CAST(x AS DECIMAL(10, 2)) + DATE '2024-01-01'" +) + +# Keywords that may stand before "(" without being a function call, or before +# a string without making it a typed literal. +_NOT_CALLS = {"OVER", "IN", "GROUP", "AND", "OR", "NOT", "WHEN", "THEN", "ELSE", "CASE", "BY"} +_NOT_CALLS |= {"DISTINCT", "BETWEEN", "LIKE", "ILIKE", "IS", "FROM", "AS"} + +# CAST target types and typed literals the spec lists, as written. +_CAST_TYPES = {"VARCHAR", "STRING", "INTEGER", "INT", "BIGINT", "DECIMAL", "NUMERIC", "FLOAT"} +_CAST_TYPES |= {"DOUBLE", "BOOLEAN", "DATE", "TIMESTAMP", "TIMESTAMP_NTZ", "TIME"} +_TYPED_LITERALS = {"DATE", "TIME", "TIMESTAMP_NTZ"} + +# Operators the spec does not have. +_FOREIGN_OPERATORS = { + TokenType.DCOLON: "::", + TokenType.ARROW: "->", + TokenType.DARROW: "->>", + TokenType.HASH_ARROW: "#>", + TokenType.DHASH_ARROW: "#>>", + TokenType.L_BRACKET: "[", + TokenType.COLON: ":", + TokenType.NULLSAFE_EQ: "<=>", + TokenType.PLACEHOLDER: "?", + TokenType.PARAMETER: "@", + TokenType.SEMICOLON: ";", +} + +_DATE_TRUNC_PARTS = {"year", "quarter", "month", "week", "day", "hour", "minute", "second"} +_EXTRACT_PARTS = _DATE_TRUNC_PARTS | {"dayofweek", "dayofyear", "millisecond"} + +_NAME = re.compile(r"[A-Za-z_][A-Za-z0-9_]*") + + +# Nodes sqlglot writes under a name the spec does not have, and the spec's name. +_SPEC_NAMES: dict[type[exp.Expr], str] = { + exp.VariancePop: "VAR_POP", + exp.ApproxDistinct: "APPROX_COUNT_DISTINCT", + exp.DayOfYear: "DAYOFYEAR", + exp.StartsWith: "STARTSWITH", + exp.EndsWith: "ENDSWITH", +} +_SPEC_TYPES = {exp.DataType.Type.TEXT: "STRING", exp.DataType.Type.TIMESTAMPNTZ: "TIMESTAMP_NTZ"} + + +def _parse(sql: str) -> exp.Expr: + """*sql* as one expression, parsed where a SELECT list item stands. + + A leading ``TRUNCATE(`` would otherwise read as a TRUNCATE statement. Any + clause besides the one expression is refused, so ``SUM(x) FROM t`` fails. + """ + select = sqlglot.parse_one(f"SELECT {sql}") + clauses = {key for key, value in select.args.items() if value} + if not isinstance(select, exp.Select) or clauses != {"expressions"}: + raise sqlglot.errors.ParseError("not a single expression") + if len(select.expressions) != 1: + raise sqlglot.errors.ParseError("not a single expression") + return select.expressions[0] + + +def ossie_sql(sql: str) -> str: + """*sql* with the spec's spelling where sqlglot's portable one differs. + + An imported formula is kept in sqlglot's spelling, which OrionBelt executes + (``STARTS_WITH``, ``VARIANCE_POP``, ``STR_POSITION``, ``CAST(x AS TEXT)``); + none of those is in the language, so the export respells them + (``STARTSWITH``, ``VAR_POP``, ``CHARINDEX``, ``STRING``). A type keeps its + parameters, so ``TIMESTAMP_NTZ(3)`` keeps its precision. *sql* that does + not parse is returned as it is. + """ + try: + node = _parse(sql) + except sqlglot.errors.SqlglotError: + return sql + + def spell(n: exp.Expr) -> exp.Expr: + name = _SPEC_NAMES.get(type(n)) + if name is not None: + args = [a for a in (n.this, n.args.get("expression")) if a is not None] + return exp.Anonymous(this=name, expressions=args) + if isinstance(n, exp.StrPosition) and not n.args.get("position"): + # CHARINDEX(substr, str) is the spec's alias for POSITION. + return exp.Anonymous(this="CHARINDEX", expressions=[n.args["substr"], n.this]) + if isinstance(n, exp.DataType) and n.this in _SPEC_TYPES: + return exp.DataType( + this=exp.DataType.Type.USERDEFINED, + kind=_SPEC_TYPES[n.this], + expressions=n.expressions, + nested=n.args.get("nested"), + ) + return n + + return node.transform(spell).sql() + + +@cache +def _allowed_nodes() -> frozenset[type[exp.Expr]]: + """The sqlglot node types the spec's functions and constructs parse into.""" + samples = [*_FUNCTIONS.values(), _CONSTRUCTS] + return frozenset(type(node) for s in samples for node in _parse(s).walk()) + + +def subset_violations(sql: str) -> list[str]: + """What in *sql* falls outside ``OSSIE_SQL_2026``; empty when nothing does.""" + try: + tokens = sqlglot.tokenize(sql) + tree = _parse(sql) + except sqlglot.errors.SqlglotError: + return ["the expression does not parse"] + return _written_violations(tokens) + _parsed_violations(tree) + + +def _written_violations(tokens: list[Token]) -> list[str]: + found: list[str] = [] + for i, token in enumerate(tokens): + nxt = tokens[i + 1] if i + 1 < len(tokens) else None + prev = tokens[i - 1] if i else None + name = token.text.upper() + # Only a bare word is a keyword: a string 'AS' or a quoted "Escape" is not. + bare = token.token_type not in (TokenType.STRING, TokenType.IDENTIFIER) + if token.token_type in _FOREIGN_OPERATORS: + operator = _FOREIGN_OPERATORS[token.token_type] + found.append(f"operator {operator} is not in OSSIE_SQL_2026") + elif token.token_type is TokenType.ESCAPE: + found.append("LIKE ... ESCAPE is not in OSSIE_SQL_2026") + elif prev is not None and prev.token_type is TokenType.ALIAS: + if not bare or name not in _CAST_TYPES: + found.append(f"type {token.text} is not in OSSIE_SQL_2026") + elif nxt is not None and nxt.token_type is TokenType.L_PAREN: + quoted_call = token.token_type is TokenType.IDENTIFIER + bare_call = bare and _NAME.fullmatch(token.text) and name not in _NOT_CALLS + if quoted_call or (bare_call and name not in _FUNCTIONS): + found.append(f"function {token.text} is not in OSSIE_SQL_2026") + elif name in ("TO_DATE", "TO_TIMESTAMP") and _argument_count(tokens, i + 1) != 1: + found.append(f"{name} with a format is EXPERIMENTAL, not in OSSIE_SQL_2026") + elif ( + bare + and nxt is not None + and nxt.token_type is TokenType.STRING + and _NAME.fullmatch(token.text) + and name not in _TYPED_LITERALS | _NOT_CALLS + ): + found.append(f"typed literal {token.text} '...' is not in OSSIE_SQL_2026") + return found + + +def _argument_count(tokens: list[Token], open_paren: int) -> int: + """The number of arguments in the call whose "(" is at *open_paren*.""" + depth, commas, empty = 1, 0, True + for token in tokens[open_paren + 1 :]: + if token.token_type is TokenType.R_PAREN and depth == 1: + break + empty = False + if token.token_type is TokenType.L_PAREN: + depth += 1 + elif token.token_type is TokenType.R_PAREN: + depth -= 1 + elif token.token_type is TokenType.COMMA and depth == 1: + commas += 1 + return 0 if empty else commas + 1 + + +def _parsed_violations(tree: exp.Expr) -> list[str]: + found: list[str] = [] + allowed = _allowed_nodes() + for node in tree.walk(): + if type(node) not in allowed: + found.append(f"{type(node).__name__} is not in OSSIE_SQL_2026") + elif isinstance(node, exp.Distinct) and not isinstance(node.parent, exp.Sum | exp.Count): + found.append("DISTINCT is defined only for SUM and COUNT") + elif isinstance(node, exp.DateTrunc) and node.text("unit").lower() not in _DATE_TRUNC_PARTS: + found.append(f"DATE_TRUNC part '{node.text('unit')}' is not in OSSIE_SQL_2026") + elif isinstance(node, exp.Extract) and node.text("this").lower() not in _EXTRACT_PARTS: + found.append(f"EXTRACT part '{node.text('this')}' is not in OSSIE_SQL_2026") + return found diff --git a/converters/orionbelt/src/ossie_orionbelt/_portable.py b/converters/orionbelt/src/ossie_orionbelt/_portable.py index 2a54c641..b5a205f0 100644 --- a/converters/orionbelt/src/ossie_orionbelt/_portable.py +++ b/converters/orionbelt/src/ossie_orionbelt/_portable.py @@ -22,8 +22,8 @@ (measure ``filters``, ``total``, synthesized counts, metric-on-metric references) that the expression must spell out: -* column references are ``.``: the OBML data object name and - the column's physical code, which are the Ossie dataset and field names; +* column references are ``.``: the OBML data object and column + names, which are the Ossie dataset and field names; * measure ``filters`` become ``AGG(CASE WHEN THEN END)``, the spec's portable filtered aggregation, mirroring the compiler's own rendering; * ``total: true`` becomes the grand-total window the compiler emits, e.g. @@ -38,8 +38,16 @@ from __future__ import annotations import re +from collections import Counter +from collections.abc import Iterable +from functools import cache from typing import Any +import sqlglot +from sqlglot import expressions as exp + +from ossie_orionbelt._sql_reserved import DUCKDB_REJECTED_WORDS, SQL_RESERVED_WORDS + _MEASURE_REF = re.compile(r"\{\[([^\]]+)\]\}") _COLUMN_REF = re.compile(r"\{\[([^\]]+)\]\.\[([^\]]+)\]\}") @@ -75,6 +83,16 @@ "last_value", } +# Filter operators whose OSSIE_SQL_2026 form differs from the ANSI one. +_PATTERN_OPERATORS = { + "contains", + "notcontains", + "starts_with", + "ends_with", + "like", + "notlike", +} + _COMPARISONS = { "equals": "=", "notequals": "<>", @@ -92,13 +110,98 @@ class NotPortableError(Exception): def sql_ident(name: str) -> str: """Render *name* as a double-quoted identifier. - Always quoted: a plain-looking name can still be a reserved word (a data - object called ``Order``), and a quoted identifier matches the Ossie dataset - or field name exactly, case included. + A quoted identifier matches exactly, case included. A reference to an + Ossie dataset or field uses :func:`ossie_ident`, which quotes only when + it has to. """ return '"' + name.replace('"', '""') + '"' +_REGULAR_IDENTIFIER = re.compile(r"[A-Za-z_][A-Za-z0-9_]*") + + +@cache +def needs_quoting(name: str) -> bool: + """Whether *name* cannot be a regular (unquoted) identifier in an export. + + Not identifier-shaped, a SQL:2023 reserved word, a word DuckDB refuses + bare, or one sqlglot does not read as ``.``: what runs and + validates these exports must parse the reference too. + """ + if not _REGULAR_IDENTIFIER.fullmatch(name): + return True + if name.lower() in SQL_RESERVED_WORDS | DUCKDB_REJECTED_WORDS: + return True + try: + node = sqlglot.parse_one(f"{name}.{name}") + except sqlglot.errors.SqlglotError: + return True + return not (isinstance(node, exp.Column) and node.table == name and node.name == name) + + +def is_regular(name: str) -> bool: + """Whether *name* can be an ANSI regular identifier: identifier-shaped and + not a SQL:2023 reserved word. + + The expression language normalizes a regular identifier to upper case; any + other name is matched as written. + """ + return bool(_REGULAR_IDENTIFIER.fullmatch(name)) and name.lower() not in SQL_RESERVED_WORDS + + +def case_collisions(names: Iterable[str]) -> frozenset[str]: + """The names that equal another one regardless of case. + + A bare identifier matches regardless of case, so ``Amount`` and ``amount`` + would be one reference. + """ + names = list(dict.fromkeys(names)) + folded = Counter(name.upper() for name in names) + return frozenset(name for name in names if folded[name.upper()] > 1) + + +def quoted_name(name: str, collisions: frozenset[str] = frozenset()) -> str | None: + """The text a reference to *name* quotes, or ``None`` when it is written bare. + + The expression language follows ANSI SQL (``core-spec/expression_language.md``, + "Namespacing and Identifier Resolution"): a regular identifier is normalized + to upper case, a quoted one is taken as written. A regular name is written + bare. One that DuckDB or sqlglot will not read bare, such as ``Qualify``, is + quoted in its normalized form, ``"QUALIFY"``, which matches it. A name that + cannot be regular (a space, a leading digit, a reserved word such as + ``Order``) is matched as written, so it is quoted as written. + + Names in *collisions* differ only by case, so the language cannot tell them + apart; each is quoted as written, which a reader that matches a quoted name + exactly, as this converter does, resolves. + """ + if name in collisions or not is_regular(name): + return name + if needs_quoting(name): + return name.upper() + return None + + +def ossie_ident(name: str, collisions: frozenset[str] = frozenset()) -> str: + """*name* as an identifier in an Apache Ossie expression (see :func:`quoted_name`).""" + text = quoted_name(name, collisions) + return name if text is None else sql_ident(text) + + +def matching_names(ref: str, quoted: bool, names: Iterable[str]) -> list[str]: + """The declared *names* the reference *ref* (unquoted text) matches. + + A quoted reference matches the name it spells exactly, else a regular name + whose normalized form it spells (``"QUALIFY"`` matches ``Qualify``). A bare + one matches regardless of case. More than one match is ambiguous. + """ + names = list(names) + if quoted: + exact = [n for n in names if n == ref] + return exact or [n for n in names if is_regular(n) and n.upper() == ref] + return [n for n in names if n.upper() == ref.upper()] + + def _string_literal(value: str) -> str: return "'" + value.replace("'", "''") + "'" @@ -113,8 +216,12 @@ def _scalar_literal(value: Any) -> str: return _string_literal(str(value)) -def _filter_literal(fv: dict[str, Any]) -> str: - """Render a typed OBML ``FilterValue`` as the literal the compiler compares against.""" +def _filter_literal(fv: dict[str, Any], ossie: bool = False) -> str: + """Render a typed OBML ``FilterValue`` as the literal the compiler compares against. + + ``OSSIE_SQL_2026`` has no plain ``TIMESTAMP '...'`` literal, so with *ossie* + a timestamp is a ``CAST``. + """ if fv.get("isNull"): return "NULL" data_type = fv.get("dataType") @@ -127,7 +234,8 @@ def _filter_literal(fv: dict[str, Any]) -> str: if data_type == "date" and fv.get("valueDate") is not None: return f"DATE {_string_literal(str(fv['valueDate']))}" if data_type == "timestamp" and fv.get("valueDate") is not None: - return f"TIMESTAMP {_string_literal(str(fv['valueDate']))}" + text = _string_literal(str(fv["valueDate"])) + return f"CAST({text} AS TIMESTAMP)" if ossie else f"TIMESTAMP {text}" return _scalar_literal(fv.get("valueString")) @@ -171,15 +279,30 @@ def synthesized_counts(obml: dict[str, Any]) -> dict[str, str]: class PortableRenderer: - """Render OBML measures and metrics as portable Ossie SQL expressions.""" + """Render OBML measures and metrics as portable Ossie SQL expressions. - def __init__(self, obml: dict[str, Any]) -> None: + By default the expressions are ANSI SQL. With *ossie* they use the forms of + Apache Ossie's ``OSSIE_SQL_2026`` language where the two differ: ``CONTAINS``, + ``STARTSWITH`` and ``ENDSWITH`` instead of ``LIKE ... ESCAPE``, and a + ``CAST`` for a timestamp literal. A filter that needs ``ESCAPE`` has + no such form and raises :class:`NotPortableError`. + """ + + def __init__(self, obml: dict[str, Any], *, ossie: bool = False) -> None: + self.ossie = ossie self.data_objects: dict[str, Any] = obml.get("dataObjects") or {} self.dimensions: dict[str, Any] = obml.get("dimensions") or {} self.measures: dict[str, Any] = obml.get("measures") or {} self.metrics: dict[str, Any] = obml.get("metrics") or {} self.counts = synthesized_counts(obml) self._resolving: set[str] = set() + # Names a bare reference could not tell apart: dataset names across the + # model, field names within each data object. + self._dataset_collisions = case_collisions(self.data_objects) + self._field_collisions = { + name: case_collisions(obj.get("columns") or {}) + for name, obj in self.data_objects.items() + } # ── references ──────────────────────────────────────────────────── @@ -188,9 +311,9 @@ def column(self, data_object: str, column: str) -> str: obj = self.data_objects.get(data_object) if obj is None: raise NotPortableError(f"references unknown data object '{data_object}'") - col = (obj.get("columns") or {}).get(column, {}) - code = col.get("code", column.lower().replace(" ", "_")) - return f"{sql_ident(data_object)}.{sql_ident(code)}" + dataset = ossie_ident(data_object, self._dataset_collisions) + field = ossie_ident(column, self._field_collisions.get(data_object, frozenset())) + return f"{dataset}.{field}" def _column_ref(self, ref: dict[str, Any]) -> str: return self.column(ref.get("dataObject", ""), ref.get("column", "")) @@ -295,17 +418,20 @@ def _filter_leaf(self, f: dict[str, Any]) -> str: col = self._column_ref(f.get("column") or {}) op = str(f.get("operator", "")).lower() values = f.get("values") or [] - first = _filter_literal(values[0]) if values else "NULL" + first = _filter_literal(values[0], self.ossie) if values else "NULL" if op in _COMPARISONS: return f"{col} {_COMPARISONS[op]} {first}" if op in ("inlist", "notinlist"): keyword = "NOT IN" if op == "notinlist" else "IN" - return f"{col} {keyword} ({', '.join(_filter_literal(v) for v in values)})" + literals = ", ".join(_filter_literal(v, self.ossie) for v in values) + return f"{col} {keyword} ({literals})" if op == "set": return f"{col} IS NOT NULL" if op == "notset": return f"{col} IS NULL" text = _filter_text(values[0]) if values else "" + if self.ossie and op in _PATTERN_OPERATORS: + return self._ossie_pattern(col, op, text) if op in ("contains", "notcontains"): return _like(col, ("%", "%"), text, negated=op == "notcontains") if op == "starts_with": @@ -319,9 +445,32 @@ def _filter_leaf(self, f: dict[str, Any]) -> str: if len(values) < 2: return f"{col} {'<>' if op == 'notbetween' else '='} {first}" keyword = "NOT BETWEEN" if op == "notbetween" else "BETWEEN" - return f"{col} {keyword} {first} AND {_filter_literal(values[1])}" + return f"{col} {keyword} {first} AND {_filter_literal(values[1], self.ossie)}" raise NotPortableError(f"measure filter operator '{op}' has no portable form") + @staticmethod + def _ossie_pattern(col: str, op: str, text: str) -> str: + """A pattern filter in its ``OSSIE_SQL_2026`` form. + + The substring operators match *text* literally, so they need no escaping; + a ``LIKE`` pattern that relies on OBML's backslash escape has no form, + because the language has no ``ESCAPE`` clause. + """ + literal = _string_literal(text) + if op in ("contains", "notcontains"): + negation = "NOT " if op == "notcontains" else "" + return f"{negation}CONTAINS({col}, {literal})" + if op == "starts_with": + return f"STARTSWITH({col}, {literal})" + if op == "ends_with": + return f"ENDSWITH({col}, {literal})" + if "\\" in text: + raise NotPortableError( + f"pattern {literal} needs an ESCAPE clause, which OSSIE_SQL_2026 does not have" + ) + keyword = op.removeprefix("not").upper() + return f"{col} {'NOT ' if op.startswith('not') else ''}{keyword} {literal}" + # ── metrics ─────────────────────────────────────────────────────── def metric(self, name: str) -> str: diff --git a/converters/orionbelt/src/ossie_orionbelt/_sql_reserved.py b/converters/orionbelt/src/ossie_orionbelt/_sql_reserved.py new file mode 100644 index 00000000..d2dde565 --- /dev/null +++ b/converters/orionbelt/src/ossie_orionbelt/_sql_reserved.py @@ -0,0 +1,553 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +"""Words an unquoted identifier in an exported expression cannot be. + +``SQL_RESERVED_WORDS`` is the SQL:2023 column of the PostgreSQL key-word +appendix (https://www.postgresql.org/docs/17/sql-keywords-appendix.html); +``END-EXEC`` is left out, being quoted anyway. ``DUCKDB_REJECTED_WORDS`` adds +what DuckDB, which runs these exports, refuses bare. sqlglot, which validates +them, is asked directly (see ``_portable.needs_quoting``). +""" + +from __future__ import annotations + +SQL_RESERVED_WORDS = frozenset( + { + "abs", + "absent", + "acos", + "all", + "allocate", + "alter", + "and", + "any", + "any_value", + "are", + "array", + "array_agg", + "array_max_cardinality", + "as", + "asensitive", + "asin", + "asymmetric", + "at", + "atan", + "atomic", + "authorization", + "avg", + "begin", + "begin_frame", + "begin_partition", + "between", + "bigint", + "binary", + "blob", + "boolean", + "both", + "btrim", + "by", + "call", + "called", + "cardinality", + "cascaded", + "case", + "cast", + "ceil", + "ceiling", + "char", + "char_length", + "character", + "character_length", + "check", + "classifier", + "clob", + "close", + "coalesce", + "collate", + "collect", + "column", + "commit", + "condition", + "connect", + "constraint", + "contains", + "convert", + "copy", + "corr", + "corresponding", + "cos", + "cosh", + "count", + "covar_pop", + "covar_samp", + "create", + "cross", + "cube", + "cume_dist", + "current", + "current_catalog", + "current_date", + "current_default_transform_group", + "current_path", + "current_role", + "current_row", + "current_schema", + "current_time", + "current_timestamp", + "current_transform_group_for_type", + "current_user", + "cursor", + "cycle", + "datalink", + "date", + "day", + "deallocate", + "dec", + "decfloat", + "decimal", + "declare", + "default", + "define", + "delete", + "dense_rank", + "deref", + "describe", + "deterministic", + "disconnect", + "distinct", + "dlnewcopy", + "dlpreviouscopy", + "dlurlcomplete", + "dlurlcompleteonly", + "dlurlcompletewrite", + "dlurlpath", + "dlurlpathonly", + "dlurlpathwrite", + "dlurlscheme", + "dlurlserver", + "dlvalue", + "double", + "drop", + "dynamic", + "each", + "element", + "else", + "empty", + "end", + "end_frame", + "end_partition", + "equals", + "escape", + "every", + "except", + "exec", + "execute", + "exists", + "exp", + "external", + "extract", + "false", + "fetch", + "filter", + "first_value", + "float", + "floor", + "for", + "foreign", + "frame_row", + "free", + "from", + "full", + "function", + "fusion", + "get", + "global", + "grant", + "greatest", + "group", + "grouping", + "groups", + "having", + "hold", + "hour", + "identity", + "import", + "in", + "indicator", + "initial", + "inner", + "inout", + "insensitive", + "insert", + "int", + "integer", + "intersect", + "intersection", + "interval", + "into", + "is", + "join", + "json", + "json_array", + "json_arrayagg", + "json_exists", + "json_object", + "json_objectagg", + "json_query", + "json_scalar", + "json_serialize", + "json_table", + "json_table_primitive", + "json_value", + "lag", + "language", + "large", + "last_value", + "lateral", + "lead", + "leading", + "least", + "left", + "like", + "like_regex", + "listagg", + "ln", + "local", + "localtime", + "localtimestamp", + "log", + "log10", + "lower", + "lpad", + "ltrim", + "match", + "match_number", + "match_recognize", + "matches", + "max", + "member", + "merge", + "method", + "min", + "minute", + "mod", + "modifies", + "module", + "month", + "multiset", + "national", + "natural", + "nchar", + "nclob", + "new", + "no", + "none", + "normalize", + "not", + "nth_value", + "ntile", + "null", + "nullif", + "numeric", + "occurrences_regex", + "octet_length", + "of", + "offset", + "old", + "omit", + "on", + "one", + "only", + "open", + "or", + "order", + "out", + "outer", + "over", + "overlaps", + "overlay", + "parameter", + "partition", + "pattern", + "per", + "percent", + "percent_rank", + "percentile_cont", + "percentile_disc", + "period", + "portion", + "position", + "position_regex", + "power", + "precedes", + "precision", + "prepare", + "primary", + "procedure", + "ptf", + "range", + "rank", + "reads", + "real", + "recursive", + "ref", + "references", + "referencing", + "regr_avgx", + "regr_avgy", + "regr_count", + "regr_intercept", + "regr_r2", + "regr_slope", + "regr_sxx", + "regr_sxy", + "regr_syy", + "release", + "result", + "return", + "returns", + "revoke", + "right", + "rollback", + "rollup", + "row", + "row_number", + "rows", + "rpad", + "rtrim", + "running", + "savepoint", + "scope", + "scroll", + "search", + "second", + "seek", + "select", + "sensitive", + "session_user", + "set", + "show", + "similar", + "sin", + "sinh", + "skip", + "smallint", + "some", + "specific", + "specifictype", + "sql", + "sqlexception", + "sqlstate", + "sqlwarning", + "sqrt", + "start", + "static", + "stddev_pop", + "stddev_samp", + "submultiset", + "subset", + "substring", + "substring_regex", + "succeeds", + "sum", + "symmetric", + "system", + "system_time", + "system_user", + "table", + "tablesample", + "tan", + "tanh", + "then", + "time", + "timestamp", + "timezone_hour", + "timezone_minute", + "to", + "trailing", + "translate", + "translate_regex", + "translation", + "treat", + "trigger", + "trim", + "trim_array", + "true", + "truncate", + "uescape", + "union", + "unique", + "unknown", + "unnest", + "update", + "upper", + "user", + "using", + "value", + "value_of", + "values", + "var_pop", + "var_samp", + "varbinary", + "varchar", + "varying", + "versioning", + "when", + "whenever", + "where", + "width_bucket", + "window", + "with", + "within", + "without", + "xml", + "xmlagg", + "xmlattributes", + "xmlbinary", + "xmlcast", + "xmlcomment", + "xmlconcat", + "xmldocument", + "xmlelement", + "xmlexists", + "xmlforest", + "xmliterate", + "xmlnamespaces", + "xmlparse", + "xmlpi", + "xmlquery", + "xmlserialize", + "xmltable", + "xmltext", + "xmlvalidate", + "year", + } +) + +# DuckDB 1.5.6 keywords that fail as a bare ``.`` reference, +# measured with ``SELECT SUM(k.k) FROM (SELECT 1 AS "k") AS "k"`` for every keyword in +# ``duckdb_keywords()``: the whole ``reserved`` category and part of ``type_function``. +DUCKDB_REJECTED_WORDS = frozenset( + { + "all", + "analyse", + "analyze", + "and", + "anti", + "any", + "array", + "as", + "asc", + "asof", + "asymmetric", + "at", + "authorization", + "binary", + "both", + "by", + "case", + "cast", + "check", + "collate", + "collation", + "column", + "concurrently", + "constraint", + "create", + "cross", + "default", + "deferrable", + "desc", + "describe", + "distinct", + "do", + "else", + "end", + "except", + "false", + "fetch", + "for", + "foreign", + "freeze", + "from", + "full", + "glob", + "group", + "having", + "ilike", + "in", + "initially", + "inner", + "intersect", + "into", + "is", + "isnull", + "join", + "lambda", + "lateral", + "leading", + "left", + "like", + "limit", + "natural", + "not", + "notnull", + "null", + "offset", + "on", + "only", + "or", + "order", + "outer", + "overlaps", + "pivot", + "pivot_longer", + "pivot_wider", + "placing", + "positional", + "primary", + "qualify", + "references", + "returning", + "right", + "select", + "semi", + "show", + "similar", + "some", + "summarize", + "symmetric", + "table", + "tablesample", + "then", + "to", + "trailing", + "true", + "union", + "unique", + "unpack", + "unpivot", + "using", + "variadic", + "verbose", + "when", + "where", + "window", + "with", + } +) diff --git a/converters/orionbelt/src/ossie_orionbelt/obml_to_ossie.py b/converters/orionbelt/src/ossie_orionbelt/obml_to_ossie.py index 5e476a20..f234d7c1 100644 --- a/converters/orionbelt/src/ossie_orionbelt/obml_to_ossie.py +++ b/converters/orionbelt/src/ossie_orionbelt/obml_to_ossie.py @@ -24,6 +24,8 @@ from __future__ import annotations import json +from collections.abc import Callable +from functools import partial from typing import Any from ossie_orionbelt._common import ( @@ -35,7 +37,16 @@ OBML_TO_OSSIE_TYPE, obml_datatype_to_ossie, ) -from ossie_orionbelt._portable import NotPortableError, PortableRenderer +from ossie_orionbelt._field_names import ( + FIELD_NAMES_COLUMN, + FIELD_NAMES_KEY, + code_renames, + field_collisions, + rename_keys, + rename_metric, +) +from ossie_orionbelt._ossie_sql import ossie_sql, subset_violations +from ossie_orionbelt._portable import NotPortableError, PortableRenderer, case_collisions class OBMLtoOssie: @@ -62,6 +73,11 @@ def convert(self) -> dict: # instance does not duplicate warnings or left-out entities. self.warnings = [] self.unexported = {} + # For what OBML saved by an earlier version keeps verbatim, which may + # name fields by code (see _field_names). + self._legacy_renames, self._legacy_ambiguous = code_renames( + self.obml.get("dataObjects", {}) or {} + ) ossie: dict[str, Any] = {"version": _OSSIE_VERSION} @@ -99,6 +115,7 @@ def convert(self) -> dict: sem_model["ai_context"] = {"instructions": self.ai_instructions} sem_model["datasets"] = datasets + self._warn_case_collisions() if all_relationships: sem_model["relationships"] = all_relationships @@ -148,10 +165,12 @@ def convert(self) -> dict: self.obml.get("customExtensions"), sem_model["custom_extensions"] ) + # One model per document, its properties at the root (apache/ossie#383). + # The root is additionalProperties:false, so there are no root-level + # ``dialects`` / ``vendors`` arrays: dialects live per expression in + # ``expression.dialects[]`` and vendors per entity in + # ``custom_extensions[].vendor_name``. ossie.update(sem_model) - - # Dialects and vendors are represented on the expressions and extensions - # that use them; root-level advertisement arrays are not supported. return ossie def _emit_foreign_extensions(self, obml_exts: list[dict] | None, ossie_exts: list[dict]) -> None: @@ -193,13 +212,9 @@ def _convert_data_object( for col_name, col in (do_obj.get("columns", {}) or {}).items() if col.get("primaryKey") ] - # Use the physical ``code`` for each column when present — Ossie - # field names mirror the physical column code (see _convert_column). + # Ossie field names are the OBML column names (see _convert_column). if pk_columns: - columns_map = do_obj.get("columns", {}) or {} - dataset["primary_key"] = [ - columns_map[c].get("code", c.lower().replace(" ", "_")) for c in pk_columns - ] + dataset["primary_key"] = pk_columns # ── Unique keys (v0.2 first-class, lossless roundtrip via OBSL) ── # OBML doesn't model unique keys natively today; round-trip via the @@ -216,6 +231,14 @@ def _convert_data_object( uk = data.get("obml_unique_keys") if isinstance(uk, list) and all(isinstance(g, list) for g in uk): unique_keys_extra = [list(g) for g in uk] + if data.get(FIELD_NAMES_KEY) != FIELD_NAMES_COLUMN: + unique_keys_extra = rename_keys( + unique_keys_extra, + self._legacy_renames.get(do_name, {}), + self._legacy_ambiguous.get(do_name, set()), + self.warnings.append, + f"Dataset '{do_name}'", + ) break if unique_keys_extra: dataset["unique_keys"] = unique_keys_extra @@ -258,7 +281,7 @@ def _convert_data_object( relationships = [] joins = do_obj.get("joins", []) for i, join in enumerate(joins): - rel = self._convert_join_to_relationship(ossie_name, do_name, do_obj, join, i) + rel = self._convert_join_to_relationship(ossie_name, do_name, join, i) if rel: relationships.append(rel) @@ -299,11 +322,16 @@ def _convert_data_object( def _convert_column( self, col_name: str, col_obj: dict, do_name: str, obml_dimensions: dict ) -> dict: - """Convert an OBML column to an Ossie field.""" + """Convert an OBML column to an Ossie field. + + The field is named by the OBML column name and computed by the column + ``code``. The code can be a SQL expression (a calculated column from an + Ossie import), which must never become the field name. + """ code = col_obj.get("code", col_name.lower().replace(" ", "_")) field: dict[str, Any] = { - "name": code, + "name": col_name, "expression": { "dialects": [ { @@ -340,8 +368,6 @@ def _convert_column( field["description"] = col_obj["description"] elif col_obj.get("comment"): field["description"] = col_obj["comment"] - else: - field["description"] = col_name # Use display name as description # ── Field label (Ossie v0.2 first-class) ── # Surfaced from OBSL-vendor customExtensions ``obml_field_label`` — @@ -376,11 +402,6 @@ def _convert_column( merged = list(existing) + [s for s in obml_col_synonyms if s not in existing] ai_ctx["synonyms"] = merged - # Build ai_context with synonyms from display name - display_synonym = col_name.lower() - code_clean = code.lower() - if display_synonym != code_clean: - synonyms.insert(0, col_name) if synonyms: ai_ctx.setdefault("synonyms", []).extend( s for s in synonyms if s not in ai_ctx.get("synonyms", []) @@ -399,11 +420,6 @@ def _convert_column( "data_type": ossie_type, "obml_abstract_type": abstract_type, } - # The Ossie field name is the physical code; keep the OBML column name - # so the reverse trip restores it (measure filters and model filters - # refer to columns by that name). - if col_name != code: - ext_data["obml_column_name"] = col_name # Preserve OBML-only column properties if col_obj.get("sqlType"): ext_data["obml_sql_type"] = col_obj["sqlType"] @@ -454,21 +470,16 @@ def _convert_column( return field def _convert_join_to_relationship( - self, ossie_from_name: str, _obml_from_name: str, from_do: dict, join: dict, index: int + self, ossie_from_name: str, _obml_from_name: str, join: dict, index: int ) -> dict | None: """Convert an OBML join to an Ossie relationship.""" join_to_display = join.get("joinTo", "") # Use the OBML display name as the Ossie target name (consistent with # _convert_data_object which uses display name as Ossie dataset name) to_name = join_to_display - target_do = self.obml.get("dataObjects", {}).get(join_to_display, {}) - - # Map column display names to codes - from_columns_display = join.get("columnsFrom", []) - to_columns_display = join.get("columnsTo", []) - - from_cols = self._resolve_column_codes(from_do, from_columns_display) - to_cols = self._resolve_column_codes(target_do, to_columns_display) + # Ossie field names are the OBML column names (see _convert_column). + from_cols = list(join.get("columnsFrom", [])) + to_cols = list(join.get("columnsTo", [])) # Generate relationship name path_name = join.get("pathName", "") @@ -499,21 +510,13 @@ def _convert_join_to_relationship( return rel - def _resolve_column_codes(self, do_obj: dict, col_display_names: list) -> list: - """Resolve OBML column display names to their code values.""" - columns = do_obj.get("columns", {}) - codes = [] - for display in col_display_names: - col = columns.get(display, {}) - codes.append(col.get("code", display.lower().replace(" ", "_"))) - return codes - def _restore_unconverted_metrics(self) -> list[dict]: """Recover Ossie metrics preserved verbatim during Ossie -> OBML import. The import path stashes metrics OBML can't represent under an Ossie-vendor model-level customExtension (``obml_unconverted_metrics``). Re-emit them - unchanged so a full Ossie -> OBML -> Ossie roundtrip keeps them. + so a full Ossie -> OBML -> Ossie roundtrip keeps them, renaming the field + references of a payload saved by an earlier version (see _field_names). """ restored: list[dict] = [] for ext in self.obml.get("customExtensions", []) or []: @@ -524,10 +527,42 @@ def _restore_unconverted_metrics(self) -> list[dict]: except (json.JSONDecodeError, TypeError): continue preserved = data.get("obml_unconverted_metrics") - if isinstance(preserved, list): - restored.extend(m for m in preserved if isinstance(m, dict)) + if not isinstance(preserved, list): + continue + metrics = [m for m in preserved if isinstance(m, dict)] + if data.get(FIELD_NAMES_KEY) != FIELD_NAMES_COLUMN: + collisions = field_collisions(self.obml.get("dataObjects", {}) or {}) + for metric in metrics: + rename_metric( + metric, + self._legacy_renames, + self._legacy_ambiguous, + collisions, + self.warnings.append, + ) + restored.extend(metrics) return restored + def _warn_case_collisions(self) -> None: + """Warn of names that differ only by case. + + The expression language normalizes a regular identifier to upper case, + so it cannot tell such names apart. + """ + data_objects = self.obml.get("dataObjects", {}) or {} + groups = [("Datasets", case_collisions(data_objects))] + [ + (f"Dataset '{name}': fields", case_collisions(obj.get("columns") or {})) + for name, obj in data_objects.items() + if isinstance(obj, dict) + ] + for where, names in groups: + if names: + self.warnings.append( + f"{where} {sorted(names)} differ only by case, which Apache Ossie " + f"identifiers ignore; references to them are quoted as written, which " + f"only a reader that matches quoted names exactly tells apart." + ) + def _merge_restored_metrics(self, ossie_metrics: list[dict]) -> None: """Append preserved (unconverted) Ossie metrics to the converted ones. @@ -557,9 +592,12 @@ def _convert_measures_and_metrics(self, obml_measures: dict, obml_metrics: dict) :class:`PortableRenderer` and computes what OrionBelt computes. A measure or metric with no faithful expression is left out of the document and kept whole in ``self.unexported`` for the model-level extension; the - reverse conversion restores it from there. + reverse conversion restores it from there. Each expression is + ``ANSI_SQL``, and also ``OSSIE_SQL_2026`` where it has a form inside + that language (see :meth:`_add_ossie_dialect`). """ renderer = PortableRenderer(self.obml) + ossie_renderer = PortableRenderer(self.obml, ossie=True) ossie_metrics = [] for name, measure in obml_measures.items(): @@ -572,6 +610,7 @@ def _convert_measures_and_metrics(self, obml_measures: dict, obml_metrics: dict) self._leave_out("measures", name, measure, str(exc)) continue ossie_metric = self._convert_measure(name, measure, sql) + self._add_ossie_dialect(ossie_metric, partial(ossie_renderer.measure, name)) self._finish_ossie_metric("measure", measure, ossie_metric) ossie_metrics.append(ossie_metric) @@ -582,11 +621,34 @@ def _convert_measures_and_metrics(self, obml_measures: dict, obml_metrics: dict) self._leave_out("metrics", name, metric, str(exc)) continue ossie_metric = self._convert_metric(name, metric, sql) + self._add_ossie_dialect(ossie_metric, partial(ossie_renderer.metric, name)) self._finish_ossie_metric("metric", metric, ossie_metric) ossie_metrics.append(ossie_metric) return ossie_metrics + @staticmethod + def _add_ossie_dialect(ossie_metric: dict, render: Callable[[], str]) -> None: + """Add the expression in ``OSSIE_SQL_2026`` when it has a form in that language. + + Rendered in the language's own forms, then checked node by node against + the subset the spec defines, so the tag is a claim another tool can rely + on. A measure using ``MODE``, ``LISTAGG``, an escaped pattern, or an + authored expression outside the subset keeps its ``ANSI_SQL`` form only. + """ + try: + sql = render() + except NotPortableError: + return + if subset_violations(sql): + # An imported formula keeps sqlglot's spelling (STARTS_WITH); the + # language's own may differ (STARTSWITH). + sql = ossie_sql(sql) + if not subset_violations(sql): + ossie_metric["expression"]["dialects"].append( + {"dialect": "OSSIE_SQL_2026", "expression": sql} + ) + def _leave_out(self, kind: str, name: str, definition: dict, reason: str) -> None: """Keep a non-portable measure or metric for the model-level extension.""" self.unexported.setdefault(kind, {})[name] = definition diff --git a/converters/orionbelt/src/ossie_orionbelt/ontology.py b/converters/orionbelt/src/ossie_orionbelt/ontology.py index e5ee6897..44e09cca 100644 --- a/converters/orionbelt/src/ossie_orionbelt/ontology.py +++ b/converters/orionbelt/src/ossie_orionbelt/ontology.py @@ -26,6 +26,7 @@ from typing import Any from ossie_orionbelt._common import _OSSIE_VERSION +from ossie_orionbelt._portable import PortableRenderer from ossie_orionbelt.obml_to_ossie import OBMLtoOssie @@ -62,23 +63,7 @@ def __init__( self.ai_instructions = ai_instructions self.warnings: list[str] = [] - @staticmethod - def _table_ref(do_obj: dict, fallback: str) -> str: - """Best-effort physical table identifier for SQL mapping expressions.""" - database = do_obj.get("database", "") - schema = do_obj.get("schema", "") - code = do_obj.get("code", "") - source = f"{database}.{schema}.{code}" if database else code - if source: - return source.split(".")[-1] - return fallback - - @staticmethod - def _col_code(do_obj: dict, display: str) -> str: - col = (do_obj.get("columns", {}) or {}).get(display, {}) - return col.get("code", display.lower().replace(" ", "_")) - - def _entity_key_expr(self, do_name: str, do_obj: dict, table_ref: str) -> str | None: + def _entity_key_expr(self, do_name: str, do_obj: dict, refs: PortableRenderer) -> str | None: """Identifying SQL expression for an entity (primary key, else first column).""" cols = do_obj.get("columns", {}) or {} pk = [c for c, col in cols.items() if col.get("primaryKey")] @@ -89,13 +74,13 @@ def _entity_key_expr(self, do_name: str, do_obj: dict, table_ref: str) -> str | self.warnings.append( f"Entity '{do_name}' has no primary key; identified by first column '{first}'" ) - return f"{table_ref}.{self._col_code(do_obj, first)}" + return refs.column(do_name, first) if len(pk) > 1: self.warnings.append( f"Entity '{do_name}' has a composite primary key; ontology object " f"mapping uses only the first key column '{pk[0]}'" ) - return f"{table_ref}.{self._col_code(do_obj, pk[0])}" + return refs.column(do_name, pk[0]) def convert(self) -> dict: # 1. Core semantic model (embedded — required by Ossie OntologyMap). @@ -109,7 +94,9 @@ def convert(self) -> dict: self.warnings.extend(core_conv.warnings) data_objects = self.obml.get("dataObjects", {}) or {} - table_refs = {n: self._table_ref(o, n) for n, o in data_objects.items()} + # Mappings are SQL over the embedded model's fields: ``.``, + # quoted as the model's metrics are. + refs = PortableRenderer(self.obml, ossie=True) # 2. Ontology components (one EntityType concept per dataObject) plus # the outgoing relationships keyed by that concept. Collect link @@ -183,7 +170,7 @@ def convert(self) -> dict: f"Join {do_name} -> {to_name} has a composite key; link " f"mapping uses only the first column '{from_cols[0]}'" ) - fk_expr = f"{table_refs[do_name]}.{self._col_code(do_obj, from_cols[0])}" + fk_expr = refs.column(do_name, from_cols[0]) rel_links.setdefault(do_name, []).append((rel_name, to_name, fk_expr)) if relationships: @@ -195,7 +182,7 @@ def convert(self) -> dict: concept_mappings: list[dict[str, Any]] = [] for do_name, do_obj in data_objects.items(): cm: dict[str, Any] = {"concept": do_name} - key_expr = self._entity_key_expr(do_name, do_obj, table_refs[do_name]) + key_expr = self._entity_key_expr(do_name, do_obj, refs) if key_expr: cm["object_mappings"] = [{"expression": key_expr}] links: list[dict[str, Any]] = [] diff --git a/converters/orionbelt/src/ossie_orionbelt/ossie_to_obml.py b/converters/orionbelt/src/ossie_orionbelt/ossie_to_obml.py index bed4b8c1..c9ecf61d 100644 --- a/converters/orionbelt/src/ossie_orionbelt/ossie_to_obml.py +++ b/converters/orionbelt/src/ossie_orionbelt/ossie_to_obml.py @@ -27,10 +27,13 @@ import re from typing import Any +from sqlglot import expressions as exp + from ossie_orionbelt._common import ( _COLUMN_REF_RE, _INTERNAL_VENDORS, _OBML_VENDOR_READ, + _OSSIE_DIALECT_TO_SQLGLOT, _OSSIE_VERSION, _SQL_PARSEABLE_DIALECTS, _VENDOR_OSSIE, @@ -42,6 +45,15 @@ obml_decimal_default, ossie_metric_datatype_to_obml, ) +from ossie_orionbelt._field_names import ( + FIELD_NAMES_COLUMN, + FIELD_NAMES_KEY, + field_collisions, + parse_expression, + rename_keys, + rename_metric, +) +from ossie_orionbelt._portable import matching_names # A dataset/column identifier in a resolved metric expression: either a bare SQL # word or a bracket-quoted token. ``_resolve_column_refs`` bracket-quotes any @@ -50,6 +62,17 @@ _RESOLVED_IDENT = r"(?:\w+|\[[^\]]+\])" +def _contains_query(sql: str, dialect: str) -> bool: + """Whether *sql* holds a query, such as a subquery or a UNION. + + A metric formula is over measures. A query inside it has no OBML form, and + its own tables and aliases are not datasets, so its aggregates must not be + lifted into measures. + """ + parsed = parse_expression(sql, _OSSIE_DIALECT_TO_SQLGLOT.get(dialect)) + return parsed is not None and parsed[1].find(exp.Query) is not None + + class OssietoOBML: """Convert an Ossie semantic model YAML to OBML format.""" @@ -195,7 +218,12 @@ def convert(self) -> dict: obml.setdefault("customExtensions", []).append( { "vendor": _VENDOR_OSSIE, - "data": json.dumps({"obml_unconverted_metrics": self._unconverted_metrics}), + "data": json.dumps( + { + "obml_unconverted_metrics": self._unconverted_metrics, + FIELD_NAMES_KEY: FIELD_NAMES_COLUMN, + } + ), } ) @@ -242,10 +270,11 @@ def _restore_unexported(self, unexported: object, obml: dict[str, Any]) -> None: def _restore_column_names(self, obml: dict[str, Any], datasets: list) -> None: """Rename columns from their Ossie field name back to the OBML column name. - The export names each field by its physical code and keeps the OBML - column name in ``obml_column_name``. Measure filters and model filters - refer to columns by that name, so every column key and every reference - the import built from field names is renamed to match. + Exports before this version named each field by its physical code and + kept the OBML column name in ``obml_column_name``. Measure filters and + model filters refer to columns by that name, so every column key and + every reference the import built from field names is renamed to match, + and so are the unique keys and unconverted metrics it keeps verbatim. """ renames: dict[str, dict[str, str]] = {} for ds in datasets: @@ -281,6 +310,39 @@ def rename(data_object: str, column: str) -> str: lambda m: "{[" + m.group(1) + "].[" + rename(m.group(1), m.group(2)) + "]}", measure["expression"], ) + # What the import keeps verbatim names fields as the document did: the + # unique keys, and the metrics OBML cannot represent. + for do_name, do_obj in obml.get("dataObjects", {}).items(): + for ext in do_obj.get("customExtensions", []) or []: + data = self._own_stash(ext, "obml_unique_keys") + if data is not None: + data["obml_unique_keys"] = rename_keys( + data["obml_unique_keys"], + renames.get(do_name, {}), + (), + self.warnings.append, + f"Dataset '{do_name}'", + ) + ext["data"] = json.dumps(data) + for ext in obml.get("customExtensions", []) or []: + data = self._own_stash(ext, "obml_unconverted_metrics") + if data is not None: + collisions = field_collisions(obml.get("dataObjects", {})) + for metric in data["obml_unconverted_metrics"]: + if isinstance(metric, dict): + rename_metric(metric, renames, {}, collisions, self.warnings.append) + ext["data"] = json.dumps(data) + + @staticmethod + def _own_stash(ext: dict[str, Any], key: str) -> dict[str, Any] | None: + """The payload of an Ossie-vendor extension this import wrote under *key*.""" + if ext.get("vendor") != _VENDOR_OSSIE: + return None + try: + data = json.loads(ext.get("data", "{}")) + except (json.JSONDecodeError, TypeError): + return None + return data if isinstance(data, dict) and isinstance(data.get(key), list) else None @staticmethod def _stashed_obml_settings(model: dict) -> object: @@ -347,20 +409,20 @@ def _convert_dataset(self, ds: dict, rel_by_from: dict) -> tuple[str, dict]: columns[col_name] = col_obj # ── Primary key flag propagation (Ossie v0.2 first-class) ── - # ``primary_key`` lists physical column codes; mark every matching - # column with ``primaryKey: true``. Unknown PK columns surface as - # a warning (the spec couples PK to relationship cardinality, so - # silently dropping is unsafe). - pk_codes = ds.get("primary_key") or [] - if pk_codes: - code_to_col = {col.get("code"): (cname, col) for cname, col in columns.items()} + # ``primary_key`` lists field names, which are the column names; mark + # every matching column with ``primaryKey: true``. A physical code + # still matches, as earlier exports listed codes. Unknown PK columns + # surface as a warning (the spec couples PK to relationship + # cardinality, so silently dropping is unsafe). + pk_names = ds.get("primary_key") or [] + if pk_names: + code_to_col = {col.get("code"): col for col in columns.values()} unknown_pks: list[str] = [] - for pk_code in pk_codes: - hit = code_to_col.get(pk_code) - if hit is None: - unknown_pks.append(pk_code) + for pk_name in pk_names: + col = columns.get(pk_name) or code_to_col.get(pk_name) + if col is None: + unknown_pks.append(pk_name) continue - _, col = hit col["primaryKey"] = True if unknown_pks: self.warnings.append( @@ -431,7 +493,12 @@ def _convert_dataset(self, ds: dict, rel_by_from: dict) -> tuple[str, dict]: do.setdefault("customExtensions", []).append( { "vendor": _VENDOR_OSSIE, - "data": json.dumps({"obml_unique_keys": [list(g) for g in unique_keys]}), + "data": json.dumps( + { + "obml_unique_keys": [list(g) for g in unique_keys], + FIELD_NAMES_KEY: FIELD_NAMES_COLUMN, + } + ), } ) @@ -461,24 +528,13 @@ def _convert_field(self, field: dict) -> tuple[str, dict]: elif code == name and dialects: code = dialects[0].get("expression", name) - # Determine abstract type. Precedence: the spec `datatype` (capitalised - # `DataType` enum) > legacy lowercase `data_type` > name heuristic. An - # OBML-origin field additionally restores its exact `abstractType` from - # the stashed extension below, keeping OBML -> Ossie -> OBML lossless, - # unless the `datatype` was edited since. A non-string value (a - # hand-authored document) counts as absent rather than crashing. + # An OBML-origin field additionally restores its exact `abstractType` + # from the stashed extension below, keeping OBML -> Ossie -> OBML + # lossless, unless the `datatype` was edited since. ossie_datatype = field.get("datatype") if not isinstance(ossie_datatype, str): ossie_datatype = "" - legacy_type = field.get("data_type") - if not isinstance(legacy_type, str): - legacy_type = "" - if ossie_datatype in OSSIE_DATATYPE_TO_OBML_ABSTRACT: - abstract_type = OSSIE_DATATYPE_TO_OBML_ABSTRACT[ossie_datatype] - elif legacy_type and legacy_type in OSSIE_TO_OBML_TYPE: - abstract_type = OSSIE_TO_OBML_TYPE[legacy_type] - else: - abstract_type = self._infer_obml_type(field) + abstract_type = self._field_abstract_type(field) col: dict[str, Any] = { "code": code, @@ -557,6 +613,21 @@ def _convert_field(self, field: dict) -> tuple[str, dict]: return name, col + def _field_abstract_type(self, field: dict) -> str: + """OBML abstractType of an Ossie field. + + Precedence: the spec `datatype` (capitalised `DataType` enum) > legacy + lowercase `data_type` > name heuristic. A non-string value (a + hand-authored document) counts as absent rather than crashing. + """ + ossie_datatype = field.get("datatype") + if isinstance(ossie_datatype, str) and ossie_datatype in OSSIE_DATATYPE_TO_OBML_ABSTRACT: + return OSSIE_DATATYPE_TO_OBML_ABSTRACT[ossie_datatype] + legacy_type = field.get("data_type") + if isinstance(legacy_type, str) and legacy_type in OSSIE_TO_OBML_TYPE: + return OSSIE_TO_OBML_TYPE[legacy_type] + return self._infer_obml_type(field) + def _infer_obml_type(self, field: dict) -> str: """Infer OBML abstractType from Ossie field metadata.""" @@ -685,7 +756,16 @@ def _extract_dimensions(self, datasets: list) -> dict: # Skip join key columns — they are FK/PK, not analytical dims if (ds_name, field_name) in self._join_key_columns: continue - abstract_type = self._infer_obml_type(field) + abstract_type = self._field_abstract_type(field) + # OBML marks a time dimension by its temporal result type, so a + # time dimension over a non-temporal field (an Integer year) + # stays a date dimension. + if ( + isinstance(dim, dict) + and dim.get("is_time") + and abstract_type not in ("date", "time", "timestamp", "timestamp_tz") + ): + abstract_type = "date" dim_def: dict[str, Any] = { "dataObject": ds_name, "column": field_name, @@ -751,37 +831,32 @@ def _convert_metrics(self, ossie_metrics: list, ds_map: dict) -> tuple[dict, dic measures: dict[str, Any] = {} metrics: dict[str, Any] = {} - # Case-insensitive dataset/field index for resolving SQL identifiers - # back to their canonical Ossie names (Snowflake/Databricks expressions - # commonly upper-case or quote them). Identifiers resolve by BOTH the - # Ossie name and the physical code (source-table code / bare-identifier - # field expression): our own OBML -> Ossie emitter writes metric SQL - # against the physical code (e.g. SUM(fact_orders.amount)), so resolving - # names only would drop such metrics on the return trip. Names take - # precedence over codes on any collision. - ds_lc: dict[str, str] = {} - for ds_name in ds_map: - ds_lc.setdefault(ds_name.lower(), ds_name) + # Dataset/field index for resolving SQL identifiers back to their + # canonical Ossie names. Identifiers resolve by the Ossie name and, + # failing that, by the physical code (source-table code / bare-identifier + # field expression), regardless of case: Snowflake/Databricks + # expressions commonly upper-case or quote them, and older exports + # wrote metric SQL against the physical code (e.g. SUM(fact_orders.amount)). + dataset_codes: dict[str, list[str]] = {} + fields: dict[str, list[str]] = {} + field_codes: dict[str, dict[str, list[str]]] = {} for ds_name, ds in ds_map.items(): # Unquote so a quoted source table (Snowflake/Databricks style, # e.g. WH.PUBLIC."fact_orders") is indexed by its bare code. table_code = self._unquote_identifier(self._parse_source(ds.get("source", ds_name))[2]) if table_code: - ds_lc.setdefault(table_code.lower(), ds_name) - - fields_lc: dict[str, dict[str, str]] = {} - for ds_name, ds in ds_map.items(): - fmap: dict[str, str] = {} - ossie_fields = ds.get("fields", []) or [] - for f in ossie_fields: - if isinstance(f, dict) and f.get("name"): - fmap.setdefault(f["name"].lower(), f["name"]) - for f in ossie_fields: - if isinstance(f, dict) and f.get("name"): + dataset_codes.setdefault(table_code.lower(), []).append(ds_name) + names: list[str] = [] + codes: dict[str, list[str]] = {} + for f in ds.get("fields", []) or []: + if isinstance(f, dict) and isinstance(f.get("name"), str) and f["name"]: + names.append(f["name"]) code = self._field_expr_identifier(f) if code: - fmap.setdefault(code.lower(), f["name"]) - fields_lc[ds_name] = fmap + codes.setdefault(code.lower(), []).append(f["name"]) + fields[ds_name] = names + field_codes[ds_name] = codes + datasets = (list(ds_map), dataset_codes) for m in ossie_metrics: name = m["name"] @@ -865,17 +940,20 @@ def _convert_metrics(self, ossie_metrics: list, ds_map: dict) -> tuple[dict, dic + ") expression", ) continue + if _contains_query(expr_text, _expr_dialect): + self._preserve_unconverted_metric(m, "expression contains a query") + continue # Resolve `dataset.column` references to canonical Ossie names - # (case-insensitive, quote-stripped) before parsing. Copying raw - # Snowflake/Databricks identifiers verbatim would yield OBML refs to - # non-existent dataObjects/columns; when a reference cannot be - # mapped the metric is preserved rather than emitted dangling. - resolved_expr = self._resolve_column_refs(expr_text, ds_lc, fields_lc) + # before parsing. Copying raw Snowflake/Databricks identifiers + # verbatim would yield OBML refs to non-existent dataObjects/columns; + # when a reference cannot be mapped to exactly one field the metric + # is preserved rather than emitted dangling or on the wrong column. + resolved_expr, problem = self._resolve_column_refs( + expr_text, datasets, fields, field_codes + ) if resolved_expr is None: - self._preserve_unconverted_metric( - m, "expression references columns not found in the model" - ) + self._preserve_unconverted_metric(m, problem) continue expr_text = resolved_expr @@ -1201,38 +1279,58 @@ def _field_expr_identifier(field: dict) -> str | None: return candidate return None + @classmethod + def _matching(cls, raw: str, names: list[str], codes: dict[str, list[str]]) -> list[str]: + """The names the identifier *raw*, as written in SQL, refers to. + + A name matches as the expression language resolves identifiers (see + :func:`matching_names`): a double-quoted or bracketed identifier + exactly, else by its normalized form; a bare or backticked one + (Databricks compares those regardless of case) regardless of case. + One that names nothing may name a physical code, regardless of case. + """ + ref = cls._unquote_identifier(raw) + if raw.startswith('"'): + ref = ref.replace('""', '"') + return matching_names(ref, raw[0] in '"[', names) or codes.get(ref.lower(), []) + def _resolve_column_refs( - self, expr: str, ds_lc: dict[str, str], fields_lc: dict[str, dict[str, str]] - ) -> str | None: + self, + expr: str, + datasets: tuple[list[str], dict[str, list[str]]], + fields: dict[str, list[str]], + field_codes: dict[str, dict[str, list[str]]], + ) -> tuple[str | None, str]: """Rewrite ``dataset.column`` references to canonical Ossie names. - Identifiers are matched case-insensitively and with SQL quoting - stripped, so Snowflake/Databricks forms such as ``SUM(ORDERS.AMOUNT)`` - or ``SUM("Orders"."amount")`` resolve to the real ``Orders.amount``. - Returns the rewritten expression, or ``None`` if any reference cannot be - mapped (the caller then preserves the metric verbatim rather than emit a - dangling reference that would fail query resolution). + Each part must match exactly one dataset or field (see :meth:`_matching`), + so forms such as ``SUM(ORDERS.AMOUNT)`` resolve. Returns the rewritten + expression, or ``None`` and the reason when a reference matches nothing + or more than one (the caller then preserves the metric verbatim rather + than emit a dangling reference or one to the wrong column). """ - unresolved = False + problems: list[str] = [] def repl(match: re.Match[str]) -> str: - nonlocal unresolved - ds_real = ds_lc.get(self._unquote_identifier(match.group("ds")).lower()) - if ds_real is None: - unresolved = True - return match.group(0) - col_real = fields_lc.get(ds_real, {}).get( - self._unquote_identifier(match.group("col")).lower() + ds_hits = self._matching(match.group("ds"), *datasets) + col_hits = ( + self._matching(match.group("col"), fields[ds_hits[0]], field_codes[ds_hits[0]]) + if len(ds_hits) == 1 + else [] ) - if col_real is None: - unresolved = True + if len(ds_hits) > 1 or len(col_hits) > 1: + hits = ds_hits if len(ds_hits) > 1 else col_hits + problems.append(f"reference {match.group(0)} is ambiguous: {sorted(hits)}") + return match.group(0) + if not col_hits: + problems.append("expression references columns not found in the model") return match.group(0) # Bracket-quote names that are not bare words so the downstream # dataset.column parsers keep multi-word display names intact. - return f"{self._q_ident(ds_real)}.{self._q_ident(col_real)}" + return f"{self._q_ident(ds_hits[0])}.{self._q_ident(col_hits[0])}" rewritten = _COLUMN_REF_RE.sub(repl, expr) - return None if unresolved else rewritten + return (None, problems[0]) if problems else (rewritten, "") def _preserve_unconverted_metric(self, ossie_metric: dict, reason: str) -> None: """Preserve an Ossie metric that has no OBML representation. diff --git a/converters/orionbelt/tests/test_ossie_converter_datatype.py b/converters/orionbelt/tests/test_ossie_converter_datatype.py index cfe2ac3a..3992054d 100644 --- a/converters/orionbelt/tests/test_ossie_converter_datatype.py +++ b/converters/orionbelt/tests/test_ossie_converter_datatype.py @@ -135,7 +135,7 @@ def test_emits_capitalised_datatype(self) -> None: ("boolean", "Boolean"), ]: ossie = conv.OBMLtoOssie(self._obml(abstract_type), model_name="s").convert() - assert _ossie_fields(ossie)["val"]["datatype"] == expected + assert _ossie_fields(ossie)["Val"]["datatype"] == expected class TestRoundtripLossless: @@ -314,6 +314,41 @@ def test_field_edit_wins(self) -> None: } } ossie = conv.OBMLtoOssie(obml, model_name="s").convert() - _ossie_fields(ossie)["val"]["datatype"] = "DateTime" + _ossie_fields(ossie)["Val"]["datatype"] = "DateTime" (col,) = _obml_columns(conv.OssietoOBML(ossie).convert()).values() assert col["abstractType"] == "timestamp" + + +def test_dimension_type_follows_the_field_datatype() -> None: + # A dimension without a resultType takes its field's datatype on import, + # as the column does, not a guess from its name ("Unit Price" names no type). + obml = { + "version": 1.0, + "dataObjects": { + "Items": { + "code": "items", + "columns": { + "Unit Price": {"code": "unit_price", "abstractType": "float"}, + "Item Count": {"code": "item_count", "abstractType": "int"}, + }, + } + }, + "dimensions": { + "Unit Price": {"dataObject": "Items", "column": "Unit Price"}, + "Item Count": {"dataObject": "Items", "column": "Item Count"}, + }, + } + back = conv.OssietoOBML(conv.OBMLtoOssie(obml).convert()).convert() + assert back["dimensions"]["Unit Price"]["resultType"] == "float" + assert back["dimensions"]["Item Count"]["resultType"] == "int" + + +def test_time_dimension_over_an_integer_field_stays_a_time_dimension() -> None: + ossie = _ossie_model([_ossie_field("d_year", datatype="Integer", dimension={"is_time": True})]) + obml = conv.OssietoOBML(ossie).convert() + (dim,) = obml["dimensions"].values() + assert dim["resultType"] == "date" + back = conv.OBMLtoOssie(obml).convert() + (field,) = back["datasets"][0]["fields"] + assert field["dimension"] == {"is_time": True} + assert field["datatype"] == "Integer" diff --git a/converters/orionbelt/tests/test_ossie_converter_ontology.py b/converters/orionbelt/tests/test_ossie_converter_ontology.py index 2ed29523..f3ce995a 100644 --- a/converters/orionbelt/tests/test_ossie_converter_ontology.py +++ b/converters/orionbelt/tests/test_ossie_converter_ontology.py @@ -27,6 +27,9 @@ from typing import Any +import sqlglot +from sqlglot import expressions as exp + import ossie_orionbelt.converter as conv _OBML: dict[str, Any] = { @@ -117,15 +120,40 @@ def test_concept_mappings_bind_keys_and_fks(self) -> None: doc = conv.OBMLtoOssieOntology(_OBML, model_name="sales").convert() cms = {cm["concept"]: cm for cm in doc["ontology_mappings"][0]["concept_mappings"]} - # Entity identity from primary key. - assert cms["Customers"]["object_mappings"] == [{"expression": "CUSTOMERS.CUSTOMER_ID"}] + # Entity identity from primary key, over the embedded model's fields. + assert cms["Customers"]["object_mappings"] == [{"expression": 'Customers."Customer ID"'}] - # Relationship link mapping binds to the FK column in the from-table. + # Relationship link mapping binds to the FK field in the from-dataset. links = {lm["relationship"]: lm for lm in cms["Orders"]["link_mappings"]} assert links["Orders_to_Customers"]["object_mapping"] == { "concept": "Customers", - "expression": "ORDERS.CUSTOMER_ID", + "expression": 'Orders."Customer Ref"', + } + + def test_mappings_name_fields_of_the_embedded_model(self) -> None: + # Mappings are SQL over the embedded semantic model's dataset fields, + # so each one must parse to a field that model declares. + doc = conv.OBMLtoOssieOntology(_OBML, model_name="sales").convert() + omap = doc["ontology_mappings"][0] + declared = { + (ds["name"], f["name"]) + for ds in omap["semantic_model"]["datasets"] + for f in ds["fields"] } + expressions = [ + m["expression"] + for cm in omap["concept_mappings"] + for m in cm.get("object_mappings", []) + ] + [ + lm["object_mapping"]["expression"] + for cm in omap["concept_mappings"] + for lm in cm.get("link_mappings", []) + ] + assert expressions + for expression in expressions: + column = sqlglot.parse_one(expression) + assert isinstance(column, exp.Column) + assert (column.table, column.name) in declared, expression def test_passes_ontology_semantic_validation(self) -> None: doc = conv.OBMLtoOssieOntology(_OBML, model_name="sales").convert() @@ -197,5 +225,5 @@ def test_missing_primary_key_warns_and_uses_first_column(self) -> None: c = conv.OBMLtoOssieOntology(obml) doc = c.convert() cm = doc["ontology_mappings"][0]["concept_mappings"][0] - assert cm["object_mappings"] == [{"expression": "A.LABEL"}] + assert cm["object_mappings"] == [{"expression": "A.Label"}] assert any("no primary key" in w for w in c.warnings) diff --git a/converters/orionbelt/tests/test_ossie_converter_properties.py b/converters/orionbelt/tests/test_ossie_converter_properties.py index 0e8759c8..bf6c60b9 100644 --- a/converters/orionbelt/tests/test_ossie_converter_properties.py +++ b/converters/orionbelt/tests/test_ossie_converter_properties.py @@ -269,21 +269,21 @@ def test_data_object_comment_roundtrip(self): class TestDimensionProperties: """Dimension properties roundtrip. - Note: Ossie uses column *code* as the field name, so after roundtrip the - dimension key becomes the physical code (ORDER_DATE). + Ossie names each field by its OBML column name, so the dimension keeps + its name through the roundtrip. """ def test_dimension_result_type_roundtrip(self): result = _roundtrip(_OBML_FULL) - assert result["dimensions"]["ORDER_DATE"].get("resultType") == "date" + assert result["dimensions"]["Order Date"].get("resultType") == "date" def test_dimension_description_roundtrip(self): result = _roundtrip(_OBML_FULL) - assert result["dimensions"]["ORDER_DATE"].get("description") == "Order date dimension" + assert result["dimensions"]["Order Date"].get("description") == "Order date dimension" def test_dimension_owner_roundtrip(self): result = _roundtrip(_OBML_FULL) - assert result["dimensions"]["ORDER_DATE"].get("owner") == "analytics" + assert result["dimensions"]["Order Date"].get("owner") == "analytics" class TestCumulativeMetricDataType: diff --git a/converters/orionbelt/tests/test_ossie_converter_roundtrip_robustness.py b/converters/orionbelt/tests/test_ossie_converter_roundtrip_robustness.py index 34f368c2..a151a0af 100644 --- a/converters/orionbelt/tests/test_ossie_converter_roundtrip_robustness.py +++ b/converters/orionbelt/tests/test_ossie_converter_roundtrip_robustness.py @@ -62,11 +62,11 @@ class TestMetricRoundTripCodeVsName: } def test_emitter_references_dataset_name(self) -> None: - # Ossie resolves ``.``: the dataset is named after the - # OBML data object, not its physical table code. + # Ossie resolves ``.``: the dataset and field are named + # after the OBML data object and column, not their physical codes. ossie = conv.OBMLtoOssie(self._OBML).convert() sql = ossie["metrics"][0]["expression"]["dialects"][0]["expression"] - assert sql == 'SUM("Orders"."amount")' + assert sql == "SUM(Orders.Amount)" def test_measure_survives_round_trip(self) -> None: ossie = conv.OBMLtoOssie(self._OBML).convert() diff --git a/converters/orionbelt/tests/test_ossie_converter_vendors.py b/converters/orionbelt/tests/test_ossie_converter_vendors.py index 536b9fbe..4da1fdd4 100644 --- a/converters/orionbelt/tests/test_ossie_converter_vendors.py +++ b/converters/orionbelt/tests/test_ossie_converter_vendors.py @@ -201,7 +201,7 @@ def test_foreign_dimension_emitted_to_field(self) -> None: } ossie = conv.OBMLtoOssie(obml).convert() field = next( - f for f in ossie["datasets"][0]["fields"] if f["name"] == "status" + f for f in ossie["datasets"][0]["fields"] if f["name"] == "Status" ) assert any(e["vendor_name"] == "TABLEAU" for e in field["custom_extensions"]) @@ -237,6 +237,8 @@ def test_legacy_common_and_obsl_still_read(self) -> None: } ossie = conv.OBMLtoOssie(obml).convert() ds = ossie["datasets"][0] - assert ds.get("unique_keys") == [["order_id"]] - order_id = next(f for f in ds["fields"] if f["name"] == "order_id") + # Saved before fields were named by column: the key names the code, + # and is exported by the field name. + assert ds.get("unique_keys") == [["Order ID"]] + order_id = next(f for f in ds["fields"] if f["name"] == "Order ID") assert order_id.get("label") == "filter" diff --git a/converters/orionbelt/tests/test_ossie_field_names.py b/converters/orionbelt/tests/test_ossie_field_names.py new file mode 100644 index 00000000..3a91de02 --- /dev/null +++ b/converters/orionbelt/tests/test_ossie_field_names.py @@ -0,0 +1,423 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +"""An Ossie field is named by the OBML column name, never by the column code. + +The code is the field's expression, and an Ossie import writes a calculated +field's SQL into it (apache/ossie#601). Naming the field by its code turned +``amount_with_tax`` into a field called ``amount * 1.1`` on the way back. +""" + +from __future__ import annotations + +import json +from typing import Any + +import ossie_orionbelt.converter as conv + + +def _expr(sql: str) -> dict[str, Any]: + return {"dialects": [{"dialect": "ANSI_SQL", "expression": sql}]} + + +def _ossie() -> dict[str, Any]: + return { + "version": "0.2.0", + "name": "sales", + "datasets": [ + { + "name": "orders", + "source": "db.orders", + "primary_key": ["id"], + "fields": [ + {"name": "id", "expression": _expr("order_id")}, + {"name": "amount", "expression": _expr("amount")}, + {"name": "amount_with_tax", "expression": _expr("amount * 1.1")}, + ], + } + ], + "metrics": [ + { + "name": "taxed_revenue", + "expression": _expr("SUM(orders.amount_with_tax)"), + } + ], + } + + +def _dataset(ossie: dict[str, Any]) -> dict[str, Any]: + return ossie["datasets"][0] + + +def _fields(ossie: dict[str, Any]) -> dict[str, dict[str, Any]]: + return {f["name"]: f for f in _dataset(ossie)["fields"]} + + +class TestOssieFirstRoundTrip: + def test_calculated_field_keeps_its_name(self) -> None: + back = conv.OBMLtoOssie(conv.OssietoOBML(_ossie()).convert()).convert() + fields = _fields(back) + assert set(fields) == {"id", "amount", "amount_with_tax"} + calc = fields["amount_with_tax"] + assert calc["expression"]["dialects"][0]["expression"] == "amount * 1.1" + assert "description" not in calc + assert "ai_context" not in calc + + def test_renamed_field_keeps_its_name_and_primary_key(self) -> None: + back = conv.OBMLtoOssie(conv.OssietoOBML(_ossie()).convert()).convert() + assert _fields(back)["id"]["expression"]["dialects"][0]["expression"] == "order_id" + assert _dataset(back)["primary_key"] == ["id"] + + def test_primary_key_matches_the_field_name(self) -> None: + obml = conv.OssietoOBML(_ossie()).convert() + assert obml["dataObjects"]["orders"]["columns"]["id"].get("primaryKey") is True + + def test_metric_references_the_calculated_field(self) -> None: + back = conv.OBMLtoOssie(conv.OssietoOBML(_ossie()).convert()).convert() + metric = next(m for m in back["metrics"] if m["name"] == "taxed_revenue") + assert metric["expression"]["dialects"][0]["expression"] == ("SUM(orders.amount_with_tax)") + + def test_no_column_name_stash(self) -> None: + back = conv.OBMLtoOssie(conv.OssietoOBML(_ossie()).convert()).convert() + for field in _dataset(back)["fields"]: + for ext in field.get("custom_extensions", []): + assert "obml_column_name" not in json.loads(ext["data"]) + + +class TestLegacyCodeNamedExport: + """Earlier exports named fields by code and stashed the column name.""" + + def test_column_name_and_primary_key_restored(self) -> None: + legacy = { + "version": "0.2.0", + "name": "sales", + "datasets": [ + { + "name": "Orders", + "source": "WH.PUBLIC.orders", + "primary_key": ["order_id"], + "fields": [ + { + "name": "order_id", + "expression": _expr("order_id"), + "custom_extensions": [ + { + "vendor_name": "ORIONBELT", + "data": json.dumps({"obml_column_name": "Order ID"}), + } + ], + } + ], + } + ], + } + columns = conv.OssietoOBML(legacy).convert()["dataObjects"]["Orders"]["columns"] + assert set(columns) == {"Order ID"} + assert columns["Order ID"]["code"] == "order_id" + assert columns["Order ID"].get("primaryKey") is True + + +def _legacy_field(code: str, column: str) -> dict[str, Any]: + """A field as earlier exports wrote it: named by code, column stashed.""" + return { + "name": code, + "expression": _expr(code), + "custom_extensions": [ + {"vendor_name": "ORIONBELT", "data": json.dumps({"obml_column_name": column})} + ], + } + + +def _legacy_export() -> dict[str, Any]: + return { + "version": "0.2.0", + "name": "sales", + "datasets": [ + { + "name": "Orders", + "source": "WH.PUBLIC.orders", + "primary_key": ["order_id"], + "fields": [ + _legacy_field("order_id", "Order ID"), + _legacy_field("customer_id", "Customer"), + _legacy_field("amount", "Net Amount"), + ], + }, + { + "name": "Customers", + "source": "WH.PUBLIC.customers", + "unique_keys": [["customer_id"]], + "fields": [_legacy_field("customer_id", "Customer ID")], + }, + ], + "relationships": [ + { + "name": "orders_to_customers", + "from": "Orders", + "to": "Customers", + "from_columns": ["customer_id"], + "to_columns": ["customer_id"], + } + ], + "metrics": [ + { + "name": "grand_total", + "expression": _expr('SUM("Orders"."amount") OVER ()'), + } + ], + } + + +class TestLegacyStashesFollowTheRename: + """What the import keeps verbatim from an older export is renamed with it.""" + + @staticmethod + def _metric(sql: str, dialect: str = "ANSI_SQL") -> tuple[str, list[str]]: + legacy = _legacy_export() + legacy["metrics"] = [ + {"name": "m", "expression": {"dialects": [{"dialect": dialect, "expression": sql}]}} + ] + importer = conv.OssietoOBML(legacy) + ossie = conv.OBMLtoOssie(importer.convert()).convert() + metric = next(m for m in ossie["metrics"] if m["name"] == "m") + return metric["expression"]["dialects"][0]["expression"], importer.warnings + + def test_unique_keys_name_the_fields(self) -> None: + obml = conv.OssietoOBML(_legacy_export()).convert() + assert obml["dataObjects"]["Orders"]["joins"][0]["joinType"] == "many-to-one" + ossie = conv.OBMLtoOssie(obml).convert() + customers = ossie["datasets"][1] + assert customers["unique_keys"] == [["Customer ID"]] + back = conv.OssietoOBML(ossie).convert() + assert back["dataObjects"]["Orders"]["joins"][0]["joinType"] == "many-to-one" + + def test_preserved_metric_references_the_fields(self) -> None: + sql, _ = self._metric('"Orders"."amount" * 2') + assert sql == '"Orders"."Net Amount" * 2' + + def test_unquoted_reference_matches_regardless_of_case(self) -> None: + sql, _ = self._metric("Orders.AMOUNT * 2") + assert sql == 'Orders."Net Amount" * 2' + + def test_only_the_field_tokens_change(self) -> None: + # sqlglot drops a weighted percentile's third argument when it + # regenerates SQL; the rename must not. + sql, _ = self._metric("percentile(Orders.amount, 0.5, Orders.customer_id)", "DATABRICKS") + assert sql == "percentile(Orders.`Net Amount`, 0.5, Orders.Customer)" + + def test_swapped_names_follow_the_stashed_mapping(self) -> None: + # Column "Net Amount" has code amount, column "amount" has code + # adjustment: field amount is "Net Amount", not the column amount. + legacy = _legacy_export() + orders = legacy["datasets"][0] + orders["fields"] = [ + _legacy_field("order_id", "Order ID"), + _legacy_field("customer_id", "Customer"), + _legacy_field("amount", "Net Amount"), + _legacy_field("adjustment", "amount"), + ] + orders["unique_keys"] = [["amount"], ["adjustment"]] + legacy["metrics"] = [ + { + "name": "m", + "expression": _expr("Orders.amount - Orders.adjustment"), + } + ] + ossie = conv.OBMLtoOssie(conv.OssietoOBML(legacy).convert()).convert() + dataset = ossie["datasets"][0] + assert dataset["unique_keys"] == [["Net Amount"], ["amount"]] + metric = ossie["metrics"][0] + assert metric["expression"]["dialects"][0]["expression"] == ( + 'Orders."Net Amount" - Orders.amount' + ) + + def test_untokenizable_metric_is_kept_and_warned(self) -> None: + sql, warnings = self._metric('"Orders"."amount" || \'unterminated') + assert sql == '"Orders"."amount" || \'unterminated' + assert any("'m'" in w and "not renamed" in w for w in warnings) + + def test_alias_declared_in_the_sql_is_not_the_dataset(self) -> None: + # The query keeps the metric out of OBML, so it is preserved verbatim. + sql, _ = self._metric( + 'SUM("Orders"."amount") OVER () + ' + '(SELECT MAX("Orders"."amount") FROM adjustments AS "Orders")' + ) + assert sql == ( + 'SUM("Orders"."Net Amount") OVER () + ' + '(SELECT MAX("Orders"."amount") FROM adjustments AS "Orders")' + ) + + def test_bracket_quoted_reference(self) -> None: + sql, _ = self._metric("[Orders].[amount] * 2") + assert sql == "[Orders].[Net Amount] * 2" + + +def _saved_by_an_older_version() -> dict[str, Any]: + """OBML an older import saved: its verbatim payloads name fields by code.""" + return { + "version": 1.0, + "dataObjects": { + "Orders": { + "code": "orders", + "database": "WH", + "schema": "PUBLIC", + "columns": { + "Order ID": {"code": "order_id", "abstractType": "string"}, + "Customer": {"code": "customer_id", "abstractType": "string"}, + "Net Amount": {"code": "amount", "abstractType": "float"}, + }, + "joins": [ + { + "joinType": "many-to-one", + "joinTo": "Customers", + "columnsFrom": ["Customer"], + "columnsTo": ["Customer ID"], + } + ], + }, + "Customers": { + "code": "customers", + "database": "WH", + "schema": "PUBLIC", + "columns": {"Customer ID": {"code": "customer_id", "abstractType": "string"}}, + "customExtensions": [ + {"vendor": "Ossie", "data": json.dumps({"obml_unique_keys": [["customer_id"]]})} + ], + }, + }, + "customExtensions": [ + { + "vendor": "Ossie", + "data": json.dumps( + { + "obml_unconverted_metrics": [ + {"name": "m", "expression": _expr('SUM("Orders"."amount") OVER ()')} + ] + } + ), + } + ], + } + + +class TestSavedByOlderVersion: + """OBML saved by an earlier version is renamed on export, from the column codes.""" + + def test_unique_keys_and_metric_name_the_fields(self) -> None: + ossie = conv.OBMLtoOssie(_saved_by_an_older_version()).convert() + model = ossie + assert model["datasets"][1]["unique_keys"] == [["Customer ID"]] + metric = next(m for m in model["metrics"] if m["name"] == "m") + assert metric["expression"]["dialects"][0]["expression"] == ( + 'SUM("Orders"."Net Amount") OVER ()' + ) + back = conv.OssietoOBML(ossie).convert() + assert back["dataObjects"]["Orders"]["joins"][0]["joinType"] == "many-to-one" + + def test_code_that_is_another_column_name_is_kept_and_warned(self) -> None: + obml = _saved_by_an_older_version() + customers = obml["dataObjects"]["Customers"] + customers["columns"]["customer_id"] = {"code": "legacy_id", "abstractType": "string"} + exporter = conv.OBMLtoOssie(obml) + ossie = exporter.convert() + assert ossie["datasets"][1]["unique_keys"] == [["customer_id"]] + assert any("'customer_id'" in w and "another column's code" in w for w in exporter.warnings) + + def test_marked_payload_is_emitted_as_is(self) -> None: + obml = conv.OssietoOBML(conv.OBMLtoOssie(_saved_by_an_older_version()).convert()).convert() + exporter = conv.OBMLtoOssie(obml) + ossie = exporter.convert() + assert ossie["datasets"][1]["unique_keys"] == [["Customer ID"]] + assert not any("unique key" in w for w in exporter.warnings) + + +class TestMalformedPreservedMetrics: + """A preserved metric is one the converter could not handle; renaming skips + what is not shaped like a metric expression instead of crashing.""" + + _STRING_EXPRESSION = {"name": "m", "expression": "SUM(Orders.amount)"} + + def test_export_of_obml_saved_by_an_older_version(self) -> None: + obml = _saved_by_an_older_version() + payload = {"obml_unconverted_metrics": [dict(self._STRING_EXPRESSION), "junk"]} + obml["customExtensions"] = [{"vendor": "Ossie", "data": json.dumps(payload)}] + ossie = conv.OBMLtoOssie(obml).convert() + assert self._STRING_EXPRESSION in ossie["metrics"] + + def test_import_of_an_older_export(self) -> None: + legacy = _legacy_export() + legacy["metrics"] = [dict(self._STRING_EXPRESSION)] + obml = conv.OssietoOBML(legacy).convert() + ossie = conv.OBMLtoOssie(obml).convert() + assert self._STRING_EXPRESSION in ossie["metrics"] + + +class TestRenamedFieldsDifferingOnlyByCase: + """A preserved metric renamed to Amount and amount keeps two references.""" + + @staticmethod + def _legacy() -> dict[str, Any]: + return { + "version": "0.2.0", + "name": "sales", + "datasets": [ + { + "name": "Orders", + "source": "WH.PUBLIC.orders", + "fields": [_legacy_field("a", "Amount"), _legacy_field("b", "amount")], + } + ], + "metrics": [ + {"name": "upper", "expression": _expr("Orders.a * 2")}, + {"name": "lower", "expression": _expr("Orders.b * 2")}, + ], + } + + @staticmethod + def _sql(ossie: dict[str, Any]) -> dict[str, str]: + return {m["name"]: m["expression"]["dialects"][0]["expression"] for m in ossie["metrics"]} + + def test_import_of_an_older_export(self) -> None: + ossie = conv.OBMLtoOssie(conv.OssietoOBML(self._legacy()).convert()).convert() + assert self._sql(ossie) == { + "upper": 'Orders."Amount" * 2', + "lower": 'Orders."amount" * 2', + } + + def test_export_of_obml_saved_by_an_older_version(self) -> None: + obml = { + "version": 1.0, + "dataObjects": { + "Orders": { + "code": "orders", + "columns": { + "Amount": {"code": "a", "abstractType": "float"}, + "amount": {"code": "b", "abstractType": "float"}, + }, + } + }, + "customExtensions": [ + { + "vendor": "Ossie", + "data": json.dumps({"obml_unconverted_metrics": self._legacy()["metrics"]}), + } + ], + } + assert self._sql(conv.OBMLtoOssie(obml).convert()) == { + "upper": 'Orders."Amount" * 2', + "lower": 'Orders."amount" * 2', + } diff --git a/converters/orionbelt/tests/test_ossie_metric_no_silent_loss.py b/converters/orionbelt/tests/test_ossie_metric_no_silent_loss.py index abc0eadf..dc8290be 100644 --- a/converters/orionbelt/tests/test_ossie_metric_no_silent_loss.py +++ b/converters/orionbelt/tests/test_ossie_metric_no_silent_loss.py @@ -59,6 +59,10 @@ def _ossie_model(metrics: list[dict[str, Any]]) -> dict[str, Any]: } +def _expr(sql: str) -> dict[str, Any]: + return {"dialects": [{"dialect": "ANSI_SQL", "expression": sql}]} + + def _metric(name: str, dialect: str, expression: str, **extra: Any) -> dict[str, Any]: m: dict[str, Any] = { "name": name, @@ -248,6 +252,30 @@ def test_undecomposable_sql_is_preserved_not_dropped(self) -> None: assert any(m["name"] == "Ratio" for m in _unconverted_stash(obml)) assert any(w.startswith("LOSSY:") and "Ratio" in w for w in converter.warnings) + @pytest.mark.parametrize( + "expression", + [ + # The subquery's own alias "Orders" is not the dataset. + 'SUM("Orders".amount) + (SELECT MAX("Orders".amount) FROM adjustments AS "Orders")', + "SUM(Orders.amount) / (SELECT COUNT(*) FROM Orders)", + "SUM(Orders.amount) - (SELECT 1 UNION ALL SELECT 2)", + # Bracket-quoted, which sqlglot reads only once rewritten as "name". + "SUM([Orders].[amount]) + (SELECT MAX([Orders].[amount]) FROM adjustments AS [Orders])", + ], + ) + def test_sql_with_a_query_is_preserved_not_decomposed(self, expression: str) -> None: + # A metric formula is over measures; a query inside it has no OBML + # form, so its aggregates must not be lifted into measures. + ossie = _ossie_model([_metric("Adjusted", "ANSI_SQL", expression)]) + converter = conv.OssietoOBML(ossie) + obml = converter.convert() + + assert "Adjusted" not in obml.get("metrics", {}) + assert not obml.get("measures") + stash = _unconverted_stash(obml) + assert [m["expression"]["dialects"][0]["expression"] for m in stash] == [expression] + assert any(w.startswith("LOSSY:") and "Adjusted" in w for w in converter.warnings) + def test_roundtrip_restores_preserved_metric(self) -> None: original = _metric("Mdx Thing", "MDX", "[Measures].[Amount]", description="from cube") ossie = _ossie_model([original]) @@ -417,3 +445,44 @@ def test_explicit_datatype_roundtrips() -> None: "total_amount", {"code": "total_amount", "abstractType": "string"}, "Orders", {} ) assert field.get("datatype") == "String" + + +class TestReferencesResolveLikeTheSpec: + """A metric reference resolves to exactly one field, as the expression language + resolves identifiers, or the metric is preserved.""" + + @staticmethod + def _convert(fields: list[str], sql: str) -> tuple[dict[str, Any], list[str]]: + ossie = _ossie_model([_metric("m", "OSSIE_SQL_2026", sql)]) + ossie["datasets"][0]["fields"] = [ + {"name": name, "expression": _expr(f"col_{i}")} for i, name in enumerate(fields) + ] + converter = conv.OssietoOBML(ossie) + return converter.convert(), converter.warnings + + @pytest.mark.parametrize("column", ["Amount", "amount"]) + def test_quoted_reference_matches_exactly(self, column: str) -> None: + obml, _ = self._convert(["Amount", "amount"], f'SUM(Orders."{column}")') + assert obml["measures"]["m"]["columns"] == [{"dataObject": "Orders", "column": column}] + + @pytest.mark.parametrize("sql", ["SUM(Orders.amount)", 'SUM(Orders."AMOUNT")']) + def test_reference_matching_two_fields_is_preserved(self, sql: str) -> None: + obml, warnings = self._convert(["Amount", "amount"], sql) + assert "m" not in obml.get("measures", {}) + assert [m["name"] for m in _unconverted_stash(obml)] == ["m"] + assert any(w.startswith("LOSSY:") and "ambiguous" in w for w in warnings) + + def test_quoted_normalized_form_matches_a_regular_name(self) -> None: + # A regular identifier is normalized to upper case: "AMOUNT" is amount. + obml, _ = self._convert(["amount"], 'SUM(Orders."AMOUNT")') + assert obml["measures"]["m"]["columns"] == [{"dataObject": "Orders", "column": "amount"}] + + def test_quoted_reference_in_another_case_is_preserved(self) -> None: + # "Amount" is taken as written; the regular name amount is AMOUNT. + obml, _ = self._convert(["amount"], 'SUM(Orders."Amount")') + assert "m" not in obml.get("measures", {}) + assert [m["name"] for m in _unconverted_stash(obml)] == ["m"] + + def test_bare_reference_matches_regardless_of_case(self) -> None: + obml, _ = self._convert(["amount"], "SUM(ORDERS.AMOUNT)") + assert obml["measures"]["m"]["columns"] == [{"dataObject": "Orders", "column": "amount"}] diff --git a/converters/orionbelt/tests/test_ossie_portable_expressions.py b/converters/orionbelt/tests/test_ossie_portable_expressions.py index 24e89666..7815c827 100644 --- a/converters/orionbelt/tests/test_ossie_portable_expressions.py +++ b/converters/orionbelt/tests/test_ossie_portable_expressions.py @@ -33,9 +33,10 @@ from typing import Any import pytest +import sqlglot import ossie_orionbelt.converter as conv -from ossie_orionbelt._portable import sql_ident +from ossie_orionbelt._portable import ossie_ident, sql_ident _MODEL: dict[str, Any] = { "version": 1.0, @@ -189,7 +190,6 @@ def _sql(ossie: dict[str, Any]) -> dict[str, str]: class TestReferencesResolve: def test_every_column_reference_is_a_dataset_field(self) -> None: - sqlglot = pytest.importorskip("sqlglot") ossie, _ = _export() fields = {ds["name"]: {f["name"] for f in ds["fields"]} for ds in ossie["datasets"]} for name, sql in _sql(ossie).items(): @@ -207,6 +207,52 @@ def test_sql_ident_always_quotes(self) -> None: assert sql_ident("Order Lines") == '"Order Lines"' assert sql_ident('a"b') == '"a""b"' + @pytest.mark.parametrize( + ("name", "rendered"), + [ + ("amount", "amount"), + ("Amount", "Amount"), + ("_net_2", "_net_2"), + ("Order Lines", '"Order Lines"'), + ("2nd", '"2nd"'), + ("Order", '"Order"'), + ("date", '"date"'), + ("value", '"value"'), + ('a"b', '"a""b"'), + ("Größe", '"Größe"'), + ], + ) + def test_ossie_ident_quotes_only_what_cannot_be_regular(self, name: str, rendered: str) -> None: + # The Ossie expression language follows ANSI SQL: a quoted "amount" + # would match a field declared amount only by exact case, so a regular + # name is bare; a space, a leading digit or a reserved word is quoted. + assert ossie_ident(name) == rendered + + def test_reserved_and_mixed_case_names_round_trip(self) -> None: + obml = copy.deepcopy(_MODEL) + lines = obml["dataObjects"]["Order Lines"]["columns"] + lines["value"] = lines.pop("Amount") + obml["measures"] = { + "Sales": { + "columns": [{"dataObject": "Order Lines", "column": "value"}], + "resultType": "float", + "aggregation": "sum", + }, + "Statuses": { + "columns": [{"dataObject": "Order Lines", "column": "Status"}], + "resultType": "int", + "aggregation": "count", + "distinct": True, + }, + } + obml["dimensions"], obml["metrics"] = {}, {} + ossie, _ = _export(obml) + assert _sql(ossie)["Sales"] == 'SUM("Order Lines"."value")' + assert _sql(ossie)["Statuses"] == 'COUNT(DISTINCT "Order Lines".Status)' + back = conv.OssietoOBML(ossie).convert() + assert back["measures"]["Sales"]["columns"][0]["column"] == "value" + assert back["measures"]["Statuses"]["columns"][0]["column"] == "Status" + def test_reserved_word_data_object_parses(self) -> None: obml = copy.deepcopy(_MODEL) obml["dataObjects"]["Order"] = obml["dataObjects"].pop("Order Lines") @@ -220,18 +266,19 @@ def test_reserved_word_data_object_parses(self) -> None: obml["dimensions"] = {} obml["metrics"] = {} ossie, _ = _export(obml) - assert _sql(ossie)["Sales"] == 'SUM("Order"."AMT")' - sqlglot = pytest.importorskip("sqlglot") + assert _sql(ossie)["Sales"] == 'SUM("Order".Amount)' sqlglot.parse_one(_sql(ossie)["Sales"]) -# Column references as the export writes them +# Column references as the export writes them: Ossie dataset and field names, +# which are the OBML data object and column names, not the physical codes. +# A name that can be a regular identifier is bare; "Order Lines" cannot. LINES = '"Order Lines"' -AMT = f'{LINES}."AMT"' -STATUS = f'{LINES}."STATUS"' -COUNTRY = '"Customers"."COUNTRY"' +AMT = f"{LINES}.Amount" +STATUS = f"{LINES}.Status" +COUNTRY = "Customers.Country" # "Line Date" declares resultType: date, so the truncation is cast back to DATE. -MONTH = f"CAST(DATE_TRUNC('month', {LINES}.\"LINE_DT\") AS DATE)" +MONTH = f"CAST(DATE_TRUNC('month', {LINES}.\"Line Date\") AS DATE)" class TestSemanticsSpelledOut: @@ -256,11 +303,11 @@ def test_avg_total_is_exact_and_default_applies(self) -> None: def test_count_distinct(self) -> None: ossie, _ = _export() - assert _sql(ossie)["Customers Reached"] == f'COUNT(DISTINCT {LINES}."CUST_ID")' + assert _sql(ossie)["Customers Reached"] == f"COUNT(DISTINCT {LINES}.Customer)" def test_synthesized_count_is_inlined(self) -> None: ossie, _ = _export() - assert _sql(ossie)["Sales per Line"] == f'SUM({AMT}) / COUNT({LINES}."LINE_ID")' + assert _sql(ossie)["Sales per Line"] == f'SUM({AMT}) / COUNT({LINES}."Line ID")' def test_metric_on_metric_is_inlined(self) -> None: ossie, _ = _export() @@ -292,8 +339,9 @@ def test_databricks_delegation_is_tagged_databricks(self) -> None: class TestExportedSqlRuns: """Every exported metric runs in the query OrionBelt would group it by. - The datasets are aliased to their Ossie names, and the query groups by the - time dimension at its grain and the partition dimension, as OrionBelt does. + Each dataset is read as an Ossie consumer reads it: its fields, each computed + by its expression, under the dataset's name. The query groups by the time + dimension at its grain and the partition dimension, as OrionBelt does. """ @staticmethod @@ -312,15 +360,27 @@ def _connection() -> Any: con.execute("INSERT INTO cust VALUES ('a', 'US'), ('b', 'DE')") return con + @staticmethod + def _dataset(ossie: dict[str, Any], name: str, table: str) -> str: + """The dataset as a subquery projecting each field from its expression.""" + ds = next(d for d in ossie["datasets"] if d["name"] == name) + fields = ", ".join( + f"{f['expression']['dialects'][0]['expression']} AS {sql_ident(f['name'])}" + for f in ds["fields"] + ) + return f"(SELECT {fields} FROM {table}) AS {sql_ident(name)}" + def test_every_exported_metric_executes(self) -> None: con = self._connection() ossie, _ = _export() + lines = self._dataset(ossie, "Order Lines", "fct") + customers = self._dataset(ossie, "Customers", "cust") for name, sql in _sql(ossie).items(): ( con.execute( f"SELECT {MONTH}, {COUNTRY}, {sql} " - f'FROM fct AS {LINES} LEFT JOIN cust AS "Customers" ' - f'ON {LINES}."CUST_ID" = "Customers"."ID" ' + f"FROM {lines} LEFT JOIN {customers} " + f'ON {LINES}."Customer" = "Customers"."Customer ID" ' f"GROUP BY {MONTH}, {COUNTRY}" ).fetchall(), name, @@ -404,8 +464,8 @@ def test_measures_metrics_and_column_names_survive(self) -> None: assert list(back["dataObjects"][do_name]["columns"]) == list(do_obj["columns"]) assert back["dataObjects"]["Order Lines"]["joins"][0]["columnsFrom"] == ["Customer"] assert back["dataObjects"]["Order Lines"]["joins"][0]["columnsTo"] == ["Customer ID"] - # The import names a dimension after its field; the column is the OBML one. - assert back["dimensions"]["COUNTRY"]["column"] == "Country" + # The import names a dimension after its field, the OBML column name. + assert back["dimensions"]["Country"]["column"] == "Country" def test_left_out_entities_ride_in_the_model_extension(self) -> None: ossie, _ = _export() @@ -413,3 +473,104 @@ def test_left_out_entities_ride_in_the_model_extension(self) -> None: kept = json.loads(ext["data"])["obml_unexported"] assert set(kept["metrics"]) == {"Sales MoM", "Fixed Grain Share"} assert set(kept["measures"]) == {"Sales At Fixed Grain"} + + +def _one_object_model(object_name: str, columns: dict[str, str]) -> dict[str, Any]: + """One data object with a SUM measure per column, column name -> code.""" + return { + "version": 1.0, + "dataObjects": { + object_name: { + "code": "fct", + "columns": { + name: {"code": code, "abstractType": "float"} for name, code in columns.items() + }, + } + }, + "measures": { + f"Total {name}": { + "columns": [{"dataObject": object_name, "column": name}], + "resultType": "float", + "aggregation": "sum", + } + for name in columns + }, + } + + +def _run_on_duckdb(ossie: dict[str, Any]) -> dict[str, Any]: + """Each exported metric over its dataset read as its fields, as a consumer does.""" + duckdb = pytest.importorskip("duckdb") + con = duckdb.connect() + con.execute("CREATE TABLE fct (A DOUBLE, B DOUBLE)") + con.execute("INSERT INTO fct VALUES (1, 10), (2, 20)") + ds = ossie["datasets"][0] + fields = ", ".join( + f"{f['expression']['dialects'][0]['expression']} AS {sql_ident(f['name'])}" + for f in ds["fields"] + ) + source = f"(SELECT {fields} FROM fct) AS {sql_ident(ds['name'])}" + return { + name: con.execute(f"SELECT {sql} FROM {source}").fetchone()[0] + for name, sql in _sql(ossie).items() + } + + +class TestParserKeywordsStayQuoted: + # A regular name DuckDB or sqlglot will not read bare is quoted in its + # normalized (upper case) form, which matches it; ``Left`` is reserved in + # SQL:2023, so it cannot be regular and is quoted as written. + @pytest.mark.parametrize( + ("name", "quoted"), + [("Qualify", "QUALIFY"), ("Limit", "LIMIT"), ("Pivot", "PIVOT"), ("Left", "Left")], + ) + def test_keyword_data_object_is_quoted_and_runs(self, name: str, quoted: str) -> None: + ossie, _ = _export(_one_object_model(name, {"Amount": "A"})) + sql = _sql(ossie)["Total Amount"] + assert sql == f'SUM("{quoted}".Amount)' + sqlglot.parse_one(sql) + assert _run_on_duckdb(ossie) == {"Total Amount": 3} + + @pytest.mark.parametrize("name", ["Qualify", "Left"]) + def test_keyword_field_round_trip(self, name: str) -> None: + obml = _one_object_model("Orders", {name: "A"}) + obml["metrics"] = {"Twice": {"expression": f"{{[Total {name}]}} * 2"}} + del obml["measures"][f"Total {name}"]["resultType"] + ossie, _ = _export(obml) + for metric in ossie["metrics"]: + metric["custom_extensions"] = [] + back = conv.OssietoOBML(ossie).convert() + columns = [c for m in back["measures"].values() for c in m.get("columns", [])] + assert columns == [{"dataObject": "Orders", "column": name}] + + +class TestNamesDifferingOnlyByCase: + def test_both_are_quoted_and_stay_distinct(self) -> None: + ossie, warnings = _export( + _one_object_model("Orders", {"Amount": "A", "amount": "B", "Qty": "B"}) + ) + assert _sql(ossie)["Total Amount"] == 'SUM(Orders."Amount")' + assert _sql(ossie)["Total amount"] == 'SUM(Orders."amount")' + assert _sql(ossie)["Total Qty"] == "SUM(Orders.Qty)" + assert any("['Amount', 'amount'] differ only by case" in w for w in warnings) + # Not run on DuckDB: it matches even quoted identifiers regardless of + # case, so no DuckDB relation can carry both fields. + + def test_round_trip_keeps_each_column(self) -> None: + ossie, _ = _export(_one_object_model("Orders", {"Amount": "A", "amount": "B"})) + back = conv.OssietoOBML(ossie).convert() + assert back["measures"]["Total Amount"]["columns"][0]["column"] == "Amount" + assert back["measures"]["Total amount"]["columns"][0]["column"] == "amount" + + def test_data_objects_differing_by_case_are_quoted(self) -> None: + obml = _one_object_model("Orders", {"Amount": "A"}) + other = _one_object_model("ORDERS", {"Amount": "B"}) + obml["dataObjects"].update(other["dataObjects"]) + obml["measures"]["Upper Total"] = { + "columns": [{"dataObject": "ORDERS", "column": "Amount"}], + "resultType": "float", + "aggregation": "sum", + } + ossie, _ = _export(obml) + assert _sql(ossie)["Total Amount"] == 'SUM("Orders".Amount)' + assert _sql(ossie)["Upper Total"] == 'SUM("ORDERS".Amount)' diff --git a/converters/orionbelt/tests/test_ossie_sql_2026_dialect.py b/converters/orionbelt/tests/test_ossie_sql_2026_dialect.py new file mode 100644 index 00000000..4f711948 --- /dev/null +++ b/converters/orionbelt/tests/test_ossie_sql_2026_dialect.py @@ -0,0 +1,336 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +"""Exported metrics also carry ``OSSIE_SQL_2026`` where it has a form. + +Each Ossie metric keeps its ``ANSI_SQL`` expression and gains one in Apache +Ossie's own expression language, rendered in that language's forms and checked +node by node against the subset ``core-spec/expression_language.md`` defines. +A metric with no such form keeps ``ANSI_SQL`` only. +""" + +from __future__ import annotations + +import copy +from typing import Any + +import pytest + +import ossie_orionbelt.converter as conv +from ossie_orionbelt._ossie_sql import _FUNCTIONS, ossie_sql, subset_violations + + +def _filter(operator: str, value: Any, data_type: str = "string") -> dict[str, Any]: + key = {"string": "valueString", "timestamp": "valueDate"}[data_type] + return { + "column": {"dataObject": "Orders", "column": "Status"}, + "operator": operator, + "values": [{"dataType": data_type, key: value}], + } + + +def _measure(aggregation: str = "sum", **extra: Any) -> dict[str, Any]: + return { + "columns": [{"dataObject": "Orders", "column": "Amount"}], + "resultType": "float", + "aggregation": aggregation, + **extra, + } + + +_OBML: dict[str, Any] = { + "version": 1.0, + "dataObjects": { + "Orders": { + "code": "orders", + "columns": { + "Amount": {"code": "amount", "abstractType": "float"}, + "Status": {"code": "status", "abstractType": "string"}, + }, + } + }, + "measures": { + "Revenue": _measure(), + "Discounted": _measure(filters=[_filter("contains", "50%")]), + "Prefixed": _measure(filters=[_filter("starts_with", "web_")]), + "Not Void": _measure(filters=[_filter("notcontains", "void")]), + "Pattern": _measure(filters=[_filter("like", "VOID%")]), + "Escaped": _measure(filters=[_filter("like", "VOID\\_%")]), + "Since": _measure(filters=[_filter("gte", "2024-01-01 00:00:00", "timestamp")]), + "Typical": _measure("mode"), + "Hashed": { + "expression": "MD5({[Orders].[Status]})", + "resultType": "int", + "aggregation": "count", + }, + }, + "metrics": {"Share": {"expression": "{[Discounted]} / {[Revenue]} * 100"}}, +} + + +def _dialects() -> dict[str, dict[str, str]]: + ossie = conv.OBMLtoOssie(copy.deepcopy(_OBML), "sales").convert() + return { + m["name"]: {d["dialect"]: d["expression"] for d in m["expression"]["dialects"]} + for m in ossie["metrics"] + } + + +class TestBothDialects: + def test_plain_measure_carries_both(self) -> None: + revenue = _dialects()["Revenue"] + assert list(revenue) == ["ANSI_SQL", "OSSIE_SQL_2026"] + assert revenue["OSSIE_SQL_2026"] == revenue["ANSI_SQL"] == "SUM(Orders.Amount)" + + def test_substring_filters_use_the_language_functions(self) -> None: + d = _dialects() + assert "ESCAPE" in d["Discounted"]["ANSI_SQL"] + assert d["Discounted"]["OSSIE_SQL_2026"] == ( + "SUM(CASE WHEN CONTAINS(Orders.Status, '50%') THEN Orders.Amount END)" + ) + assert "STARTSWITH(Orders.Status, 'web_')" in d["Prefixed"]["OSSIE_SQL_2026"] + assert "NOT CONTAINS(Orders.Status, 'void')" in d["Not Void"]["OSSIE_SQL_2026"] + + def test_like_is_kept(self) -> None: + d = _dialects()["Pattern"] + assert d["ANSI_SQL"] == d["OSSIE_SQL_2026"] + assert "Orders.Status LIKE 'VOID%'" in d["OSSIE_SQL_2026"] + + def test_timestamp_literal_is_a_cast(self) -> None: + d = _dialects()["Since"] + assert "TIMESTAMP '2024-01-01 00:00:00'" in d["ANSI_SQL"] + assert "CAST('2024-01-01 00:00:00' AS TIMESTAMP)" in d["OSSIE_SQL_2026"] + + def test_metric_over_measures_carries_both(self) -> None: + share = _dialects()["Share"] + assert share["OSSIE_SQL_2026"] == ( + "SUM(CASE WHEN CONTAINS(Orders.Status, '50%') THEN Orders.Amount END) " + "/ SUM(Orders.Amount) * 100" + ) + + def test_every_ossie_expression_is_inside_the_subset(self) -> None: + for name, dialects in _dialects().items(): + if "OSSIE_SQL_2026" in dialects: + assert subset_violations(dialects["OSSIE_SQL_2026"]) == [], name + + +class TestAnsiOnly: + @pytest.mark.parametrize( + "name", + [ + "Escaped", # needs ESCAPE, which the language does not have + "Typical", # MODE is not in the language + "Hashed", # an authored expression outside the subset + ], + ) + def test_no_ossie_form_keeps_ansi_only(self, name: str) -> None: + assert list(_dialects()[name]) == ["ANSI_SQL"] + + def test_untokenizable_expression_keeps_ansi_only(self) -> None: + # An unterminated string does not tokenize; the export must not fail. + obml = copy.deepcopy(_OBML) + obml["measures"] = { + "Broken": { + "expression": "{[Orders].[Status]} || 'x", + "resultType": "int", + "aggregation": "count", + } + } + obml["metrics"] = {} + ossie = conv.OBMLtoOssie(obml, "sales").convert() + assert [d["dialect"] for d in ossie["metrics"][0]["expression"]["dialects"]] == ["ANSI_SQL"] + + +def _hello_world(metric_sql: str) -> dict[str, Any]: + """The hello-world model of apache/ossie#576, tagged OSSIE_SQL_2026.""" + + def expr(sql: str) -> dict[str, Any]: + return {"dialects": [{"dialect": "OSSIE_SQL_2026", "expression": sql}]} + + return { + "version": "0.2.0.dev0", + "name": "hello_world", + "datasets": [ + { + "name": "sales", + "source": "hello.public.sales", + "fields": [ + {"name": "amount", "datatype": "Decimal", "expression": expr("amount")}, + {"name": "status", "datatype": "String", "expression": expr("status")}, + ], + } + ], + "metrics": [ + {"name": "total_amount", "datatype": "Decimal", "expression": expr(metric_sql)} + ], + } + + +class TestRoundTrip: + def test_import_reads_the_ansi_form(self) -> None: + ossie = conv.OBMLtoOssie(copy.deepcopy(_OBML), "sales").convert() + back = conv.OssietoOBML(ossie).convert() + assert ( + back["measures"]["Discounted"]["filters"] == _OBML["measures"]["Discounted"]["filters"] + ) + + def test_ossie_first_scalar_function_keeps_its_tag(self) -> None: + # ROUND is REQUIRED by the spec; dropping the tag would retag it again. + doc = _hello_world("ROUND(SUM(sales.amount), 2)") + back = conv.OBMLtoOssie(conv.OssietoOBML(doc).convert()).convert() + dialects = { + d["dialect"]: d["expression"] + for d in next(m for m in back["metrics"] if m["name"] == "total_amount")["expression"][ + "dialects" + ] + } + assert subset_violations(dialects["OSSIE_SQL_2026"]) == [] + assert "ROUND(" in dialects["OSSIE_SQL_2026"] + + @pytest.mark.parametrize( + ("written", "exported"), + [ + ("SUM(CHARINDEX('a', sales.amount))", "SUM(CHARINDEX('a', sales.amount))"), + ("SUM(POSITION('a' IN sales.amount))", "SUM(POSITION('a' IN sales.amount))"), + ("VAR_POP(sales.amount)", "VAR_POP(sales.amount)"), + ("SUM(CAST(sales.amount AS STRING))", "SUM(CAST(sales.amount AS STRING))"), + ("SUM(CASE WHEN STARTSWITH(sales.status, 'w') THEN sales.amount END)", None), + ("SUM(CASE WHEN ENDSWITH(sales.status, 'w') THEN sales.amount END)", None), + ], + ) + def test_imported_functions_keep_the_tag(self, written: str, exported: str | None) -> None: + back = conv.OBMLtoOssie(conv.OssietoOBML(_hello_world(written)).convert()).convert() + metric = next(m for m in back["metrics"] if m["name"] == "total_amount") + dialects = {d["dialect"]: d["expression"] for d in metric["expression"]["dialects"]} + assert dialects["OSSIE_SQL_2026"] == (exported or written) + + @pytest.mark.parametrize( + ("sql", "respelled"), + [ + ("CAST(t.x AS TIMESTAMPNTZ(3))", "CAST(t.x AS TIMESTAMP_NTZ(3))"), + ("CAST(t.x AS TEXT(2))", "CAST(t.x AS STRING(2))"), + ("CAST(t.x AS TEXT)", "CAST(t.x AS STRING)"), + ], + ) + def test_respelled_types_keep_their_parameters(self, sql: str, respelled: str) -> None: + assert ossie_sql(sql) == respelled + + def test_ossie_first_metric_keeps_its_tag(self) -> None: + doc = _hello_world("SUM(sales.amount)") + back = conv.OBMLtoOssie(conv.OssietoOBML(doc).convert()).convert() + dialects = { + d["dialect"]: d["expression"] for d in back["metrics"][0]["expression"]["dialects"] + } + assert dialects["OSSIE_SQL_2026"] == "SUM(sales.amount)" + + +class TestSubsetChecker: + @pytest.mark.parametrize( + "sql", + [ + "SUM(t.x)", + "COUNT(*)", + "COUNT(DISTINCT t.x)", + "PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY t.x)", + "COALESCE(SUM(t.x), 0)", + "CAST(DATE_TRUNC('month', t.d) AS DATE)", + "LAG(SUM(t.x), 1, 0) OVER (ORDER BY t.d ASC)", + "SUM(SUM(t.x)) OVER (PARTITION BY t.r ORDER BY t.d" + " ROWS BETWEEN 2 PRECEDING AND CURRENT ROW)", + "SUM(CASE WHEN t.a IN ('x', 'y') AND NOT (t.b IS NULL) THEN t.c END)", + ], + ) + def test_allowed(self, sql: str) -> None: + assert subset_violations(sql) == [] + + @pytest.mark.parametrize("name", sorted(_FUNCTIONS)) + def test_every_spec_function_sample_is_allowed(self, name: str) -> None: + # The samples define both the names and the node types; each must pass. + assert subset_violations(_FUNCTIONS[name]) == [] + + @pytest.mark.parametrize( + "sql", + [ + "ROUND(SUM(t.x), 2) + ABS(t.y) + CEIL(t.z) + TRUNCATE(t.x, 1) + LOG10(t.x)", + "DATEADD(day, 7, t.d) + DATEDIFF(month, t.a, t.b) + EXTRACT(YEAR FROM t.d)", + "IFF(t.c, 1, 0) + ZEROIFNULL(t.x) + NVL2(t.a, 1, 2) + CHARINDEX('a', t.s)", + "CAST(t.x AS STRING) + TRY_CAST(t.y AS DECIMAL(10, 2)) + TIMESTAMP_NTZ '2024-01-01'", + "t.a || t.b", + "t.a IS NOT DISTINCT FROM t.b", + "CASE WHEN t.a LIKE 'x%' THEN 'y' ELSE 'z' END", + ], + ) + def test_required_scalar_functions_and_syntax(self, sql: str) -> None: + assert subset_violations(sql) == [] + + @pytest.mark.parametrize( + "sql", + [ + # A string or a quoted name is not a keyword. + "SUM(CASE WHEN t.status = 'AS' THEN t.amount END)", + "SUM(CASE WHEN t.status = 'ESCAPE' THEN t.amount END)", + 'SUM(t."AS") + SUM(t."Escape") + SUM(t."Date")', + # A parenthesized argument is still one argument. + "TO_DATE((MAX(t.s)))", + # A leading TRUNCATE is a function, not a statement. + "TRUNCATE(t.x, 2) + CEILING(t.x)", + ], + ) + def test_no_false_violations(self, sql: str) -> None: + assert subset_violations(sql) == [] + + def test_a_clause_after_the_expression_is_refused(self) -> None: + assert subset_violations("SUM(t.x) FROM t") == ["the expression does not parse"] + + @pytest.mark.parametrize("name", sorted(_FUNCTIONS)) + def test_written_back_samples_stay_in_the_language(self, name: str) -> None: + # The import keeps sqlglot's spelling (VAR_POP as VARIANCE_POP); the + # export respells it through ossie_sql, or an imported metric loses its + # OSSIE_SQL_2026 form. + import sqlglot + + written = sqlglot.parse_one(f"SELECT {_FUNCTIONS[name]}").expressions[0].sql() + assert subset_violations(ossie_sql(written)) == [] + + @pytest.mark.parametrize( + ("sql", "reason"), + [ + ("MODE(t.x)", "Mode"), + ("LISTAGG(t.x, ',')", "LISTAGG"), + ("ANY_VALUE(t.x)", "AnyValue"), + ("t.a LIKE 'x' ESCAPE '\\'", "Escape"), + ("AVG(DISTINCT t.x)", "DISTINCT"), + ("SUM(t.x) + (SELECT 1)", "Subquery"), + ("CAST(t.x AS INTERVAL)", "INTERVAL"), + # Written forms sqlglot would normalize into allowed nodes. + ("SUM(SAFE_CAST(t.x AS INT))", "SAFE_CAST"), + ("CAST(t.x AS INT64)", "INT64"), + ("STR_POSITION(t.s, 'a')", "STR_POSITION"), + # Constructs the spec does not list. + ("TIMESTAMP '2024-01-01'", "TIMESTAMP"), + ("INTERVAL '1' DAY", "INTERVAL"), + ("t.a::int", "::"), + ("t.a[1]", "["), + ("CURRENT_USER", "CurrentUser"), + ("TO_DATE(t.s, 'YYYY')", "EXPERIMENTAL"), + ("EXTRACT(DECADE FROM t.d)", "EXTRACT"), + ("DATE_TRUNC('decade', t.d)", "DATE_TRUNC"), + ("SUM(", "does not parse"), + ], + ) + def test_rejected(self, sql: str, reason: str) -> None: + assert any(reason in v for v in subset_violations(sql)) diff --git a/converters/orionbelt/tests/test_ossie_v02_compat.py b/converters/orionbelt/tests/test_ossie_v02_compat.py index 5327d66a..a6a40b87 100644 --- a/converters/orionbelt/tests/test_ossie_v02_compat.py +++ b/converters/orionbelt/tests/test_ossie_v02_compat.py @@ -102,7 +102,7 @@ def walk(node: Any) -> None: { "vendor": "OBSL", "data": json.dumps( - {"obml_unique_keys": [["order_id"], ["order_id", "line_no"]]} + {"obml_unique_keys": [["Order ID"], ["Order ID", "Line Number"]]} ), } ], @@ -204,13 +204,14 @@ class TestPrimaryKey: def test_composite_pk_emitted_in_declaration_order(self) -> None: ossie = conv.OBMLtoOssie(_OBML_WITH_PK_AND_LABEL).convert() ds = ossie["datasets"][0] - # Two columns flagged primaryKey: emit composite in declaration order - assert ds["primary_key"] == ["order_id", "line_no"] + # Two columns flagged primaryKey: emit composite in declaration order, + # by field name (the OBML column name), not by physical code. + assert ds["primary_key"] == ["Order ID", "Line Number"] def test_pk_roundtrip_restores_per_column_flag(self) -> None: ossie = conv.OBMLtoOssie(_OBML_WITH_PK_AND_LABEL).convert() obml = conv.OssietoOBML(ossie).convert() - # Ossie fields are keyed by physical code; the OBML column names come back. + # Ossie fields are named by the OBML column names, which come back. cols = obml["dataObjects"]["Orders"]["columns"] assert cols["Order ID"].get("primaryKey") is True assert cols["Line Number"].get("primaryKey") is True @@ -249,7 +250,7 @@ class TestUniqueKeys: def test_unique_keys_roundtrip(self) -> None: ossie = conv.OBMLtoOssie(_OBML_WITH_PK_AND_LABEL).convert() ds = ossie["datasets"][0] - assert ds.get("unique_keys") == [["order_id"], ["order_id", "line_no"]] + assert ds.get("unique_keys") == [["Order ID"], ["Order ID", "Line Number"]] # --------------------------------------------------------------------------- @@ -261,7 +262,7 @@ class TestFieldLabel: def test_label_emitted_from_custom_extensions(self) -> None: ossie = conv.OBMLtoOssie(_OBML_WITH_PK_AND_LABEL).convert() fields = ossie["datasets"][0]["fields"] - order_id_field = next(f for f in fields if f["name"] == "order_id") + order_id_field = next(f for f in fields if f["name"] == "Order ID") assert order_id_field["label"] == "filter" def test_label_roundtrip(self) -> None: diff --git a/converters/orionbelt/uv.lock b/converters/orionbelt/uv.lock index 3d66da60..9bcb2001 100644 --- a/converters/orionbelt/uv.lock +++ b/converters/orionbelt/uv.lock @@ -45,6 +45,7 @@ dependencies = [ { name = "jsonschema" }, { name = "pyyaml" }, { name = "referencing" }, + { name = "sqlglot" }, ] [package.optional-dependencies] @@ -67,6 +68,7 @@ requires-dist = [ { name = "orionbelt-semantic-layer", marker = "extra == 'obml-validation'" }, { name = "pyyaml", specifier = ">=6.0" }, { name = "referencing", specifier = ">=0.30" }, + { name = "sqlglot", specifier = ">=30,<31" }, ] provides-extras = ["obml-validation"]