diff --git a/compliance/portability/README.md b/compliance/portability/README.md new file mode 100644 index 000000000..9039231ac --- /dev/null +++ b/compliance/portability/README.md @@ -0,0 +1,70 @@ + + +# Portability measurement ladder (M0-M3) + +A small set of portable fixtures, and a grader, for measuring how faithfully a +model survives a round-trip through each converter. This is issue #576 +(measurements for model portability across converters). The fixtures are the +inputs; `grader/` scores each returned document against its input, element by +element, into the outcome vocabulary in `manifest.yaml`. A thin runner that drives +the converters (modelled on `cube/tools/interop_matrix.py` in apache/ossie-converters) +turns the whole thing into one command; the grader is written to sit behind it. + +## What is here + +Each rung adds exactly one capability on top of the one below it, so a +difference in a converter's output localizes to a single construct: + +| Rung | Capability added | Construct | +|------|------------------|-----------| +| M0 | baseline | one dataset, one field, one `SUM` metric | +| M1 | a scalar field, no aggregation | `amount * 1.1` calculated column | +| M2 | the core aggregate set | `SUM`, `COUNT`, `COUNT(DISTINCT)`, `AVG`, `MIN`, `MAX` | +| M3 | a filtered aggregate | `SUM(CASE WHEN ... THEN x ELSE 0 END)` | + +`manifest.yaml` lists the fixtures, the element each rung exercises, and the +per-element outcome vocabulary the grader assigns (PORTABLE / TAG_PRESERVED, +REWRITTEN, SURVIVED-vendor-only, PLACEHOLDER, ADDED, RELABELED, +DROPPED-diagnosed, DROPPED-silent, REJECTED). `grader/grade.py` implements it +and `grader/test_grade.py` has one test per outcome plus an identity check that +grades the real fixtures all-PORTABLE. See `grader/README.md`. + +## Design rules + +- **Portable dialect only.** Every fixture is authored in `ANSI_SQL` or + `OSSIE_SQL_2026`. A vendor-tagged input would only exercise one converter, + which defeats a cross-converter measurement. +- **Qualified reference form is primary.** Where a metric references a dataset + field, the qualified `dataset.field` form is the primary fixture and the bare + form is a sibling variant, pending the name-resolution decisions in #486 and + #343. +- **Conformance is a floor, not the measurement.** Every fixture passes + `validation/validate.py` on current `main`. Fidelity is what the grader + measures on top of that. + +## Scope and placement + +Rungs M4-M8 (relationships and cross-dataset metrics, metric-to-metric +composition, date and time scalars, window and offset functions) are held +pending #354, #591 and #587, and are not in this set. + +Location is a starting point. If these should sit with the interoperability +harness from #348 or elsewhere once that lands, they can move; the fixtures are +self-contained and the manifest paths are relative. diff --git a/compliance/portability/grader/.gitignore b/compliance/portability/grader/.gitignore new file mode 100644 index 000000000..7a60b85e1 --- /dev/null +++ b/compliance/portability/grader/.gitignore @@ -0,0 +1,2 @@ +__pycache__/ +*.pyc diff --git a/compliance/portability/grader/README.md b/compliance/portability/grader/README.md new file mode 100644 index 000000000..0f699de97 --- /dev/null +++ b/compliance/portability/grader/README.md @@ -0,0 +1,66 @@ + + +# Portability fidelity grader + +`grade.py` is the scoring half of issue #576. Conformance (`validate.py` passing) +is a floor and says nothing about whether a model survived a converter round-trip; +this grader is the fidelity half. + +It does not drive the converters. It takes the input document and the document a +converter returned, and diffs them element by element into the outcome vocabulary +in `../manifest.yaml`. A thin runner (modelled on `cube/tools/interop_matrix.py` in +apache/ossie-converters) drives each converter and feeds the +grader, so the measurement table becomes one command. + +## Usage + +``` +python grade.py --input model.ossie.yaml --returned returned.ossie.yaml \ + --converter dbt [--diagnostics warnings.txt] [--json] +``` + +## Outcomes + +PORTABLE (with a TAG_PRESERVED sub-flag), REWRITTEN, SURVIVED_VENDOR_ONLY, +PLACEHOLDER, ADDED, RELABELED, DROPPED_DIAGNOSED, DROPPED_SILENT, REJECTED. + +## Design + +- **Standard library for the diff.** `sqlglot` (already used by the converters + and `validate.py`) is used when available to compare normalized expression + trees, so a REWRITTEN that is only a requalify, an unquote or a reorder is + marked cosmetic, while one it cannot prove equivalent is flagged for review. + Without `sqlglot` the grader still runs and marks every rewrite as needing + review. +- **Textual by default, semantic where it can be.** PORTABLE means the converter + returned the same expression it was given (modulo cosmetic whitespace). True + semantic equivalence of SQL is undecidable in general, so the grader auto-clears + only the changes a normalized tree proves equal and leaves the rest for a human. +- **Identity beyond name.** A dropped input element and an added output element + that carry an equivalent expression are paired as one element whose identity + (name) was not preserved, rather than scored as a drop plus an unrelated add. +- **Diagnosed vs silent, and PLACEHOLDER, need the converter's warnings.** Pass + them with `--diagnostics`. + +## Tests + +`test_grade.py` has one test per outcome plus an identity check that runs the real +M0-M3 fixtures through a no-op converter and asserts every element grades +PORTABLE. Run with `PYTHONPATH=. pytest test_grade.py`. diff --git a/compliance/portability/grader/grade.py b/compliance/portability/grader/grade.py new file mode 100644 index 000000000..7d8cd34a0 --- /dev/null +++ b/compliance/portability/grader/grade.py @@ -0,0 +1,388 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +"""Fidelity grader for the portability measurement ladder (issue #576). + +The grader answers one question per element: when a model is driven through a +converter and back, what happened to each metric and field expression it started +with? Conformance (``validate.py`` passing) is a floor and says nothing about +fidelity; this grader is the fidelity half. + +It does NOT drive the converters: that is the thin runner's job (modelled on +``cube/tools/interop_matrix.py`` in apache/ossie-converters). It takes the input document and the +document a converter returned, and diffs them element by element into the outcome +vocabulary in ``../manifest.yaml``: + +* PORTABLE returned under a portable tag, expression unchanged +* TAG_PRESERVED sub-flag of PORTABLE: an OSSIE_SQL_2026 input kept its tag +* REWRITTEN same portable tag, expression text changed. A change the + normalized comparison proves is only spelling (requalify, + unquote, reorder) is marked benign; anything it cannot + prove equivalent is flagged for review +* SURVIVED_VENDOR_ONLY returned in a vendor dialect only, as a real translation +* PLACEHOLDER replaced by a valid stand-in that is not the content + (e.g. ``BLANK()``), with a diagnostic: a diagnosed loss +* RELABELED same text, a different tag than authored +* DROPPED_DIAGNOSED gone, and a diagnostic names it +* DROPPED_SILENT gone, no diagnostic +* REJECTED the converter refused the whole document (document scope) +* ADDED an element in the output that was not in the input + +The diff is standard library. ``sqlglot`` (already used by the converters and by +``validate.py``) is used when available to compare normalized expression trees so +a cosmetic rewrite is told from a possible change in meaning; without it the +grader still runs and simply marks every rewrite as needing review. + +The diagnosed-versus-silent and PLACEHOLDER distinctions need the converter's own +warnings; pass them with ``--diagnostics``. +""" + +from __future__ import annotations + +import argparse +import json +import re +import sys +from dataclasses import dataclass, field + +PORTABLE_DIALECTS = frozenset({"ANSI_SQL", "OSSIE_SQL_2026"}) + +# Normalized expressions a converter emits as a stand-in when it cannot translate +# the authored one. Compared after _normalize + upper-casing. +PLACEHOLDER_TOKENS = frozenset({"BLANK()", "BLANK", "NULL"}) + +PORTABLE = "PORTABLE" +TAG_PRESERVED = "TAG_PRESERVED" +REWRITTEN = "REWRITTEN" +SURVIVED_VENDOR_ONLY = "SURVIVED_VENDOR_ONLY" +PLACEHOLDER = "PLACEHOLDER" +RELABELED = "RELABELED" +DROPPED_DIAGNOSED = "DROPPED_DIAGNOSED" +DROPPED_SILENT = "DROPPED_SILENT" +REJECTED = "REJECTED" +ADDED = "ADDED" + + +def _load(path): + """Load an Ossie document from a YAML or JSON file.""" + with open(path, "r", encoding="utf-8") as handle: + body = handle.read() + if path.endswith(".json"): + return json.loads(body) + try: + import yaml + except ImportError as exc: # pragma: no cover - environment dependent + raise SystemExit("PyYAML is needed to read a YAML document; pass JSON instead") from exc + return yaml.safe_load(body) + + +def _normalize(expression): + """Collapse insignificant whitespace, including around parentheses and commas, + so a reformatted expression still matches textually. Textual, not semantic.""" + if expression is None: + return "" + collapsed = re.sub(r"\s+", " ", str(expression)).strip() + return re.sub(r"\s*([(),])\s*", r"\1", collapsed) + + +def _equivalent_tree(left, right): + """Return True if two SQL expressions are equal after stripping table + qualifiers and quoting, False if they differ, or None if it cannot be decided + (sqlglot missing, or either side does not parse). Used only to tell a cosmetic + REWRITTEN from one that needs review.""" + try: + import sqlglot + from sqlglot import exp + except ImportError: # pragma: no cover - environment dependent + return None + + def canonical(text): + try: + tree = sqlglot.parse_one(text) + except Exception: + return None + for column in tree.find_all(exp.Column): + # Drop the table qualifier so sales.amount and amount compare equal. + column.set("table", None) + column.set("db", None) + column.set("catalog", None) + for identifier in tree.find_all(exp.Identifier): + # Unquote so "amount" and amount compare equal (sigma, orionbelt quote). + identifier.set("quoted", False) + # normalize_identifiers lower-cases identifiers consistently. + try: + from sqlglot.optimizer.normalize_identifiers import normalize_identifiers + + tree = normalize_identifiers(tree) + except Exception: + pass + return tree.sql(normalize=True) + + left_sql = canonical(left) + right_sql = canonical(right) + if left_sql is None or right_sql is None: + return None + return left_sql == right_sql + + +def _dialect_map(expression_obj): + result = {} + if not isinstance(expression_obj, dict): + return result + for entry in expression_obj.get("dialects") or []: + if isinstance(entry, dict) and entry.get("dialect") is not None: + result[entry["dialect"]] = _normalize(entry.get("expression")) + return result + + +@dataclass +class Element: + kind: str # "metric" or "field" + name: str + dialects: dict + + +def _collect(document): + """Collect metrics and field expressions, keyed by (kind, name).""" + elements = {} + if not isinstance(document, dict): + return elements + for metric in document.get("metrics") or []: + if isinstance(metric, dict) and metric.get("name"): + key = ("metric", metric["name"]) + elements[key] = Element("metric", metric["name"], _dialect_map(metric.get("expression"))) + for dataset in document.get("datasets") or []: + if not isinstance(dataset, dict): + continue + dataset_name = dataset.get("name") or "" + for field_def in dataset.get("fields") or []: + if isinstance(field_def, dict) and field_def.get("name"): + name = f"{dataset_name}.{field_def['name']}" + elements[("field", name)] = Element("field", name, _dialect_map(field_def.get("expression"))) + return elements + + +@dataclass +class Finding: + kind: str + name: str + outcome: str + tag_preserved: bool = False + detail: str = "" + + +@dataclass +class Report: + converter: str = "" + rejected: bool = False + findings: list = field(default_factory=list) + + def counts(self): + tally = {} + for finding in self.findings: + tally[finding.outcome] = tally.get(finding.outcome, 0) + 1 + return tally + + +def _authored_tag(dialects): + for dialect in dialects: + if dialect in PORTABLE_DIALECTS: + return dialect + return next(iter(dialects), None) + + +def _diagnosed(name, diagnostics): + return name in diagnostics or name.split(".")[-1] in diagnostics + + +def _classify_present(input_element, output_element): + """Classify an input element that still has a counterpart in the output.""" + in_dialects = input_element.dialects + out_dialects = output_element.dialects + authored_tag = _authored_tag(in_dialects) + authored_text = in_dialects.get(authored_tag, "") + + # PORTABLE: authored portable tag returned with the same text under that tag. + if authored_tag in PORTABLE_DIALECTS and out_dialects.get(authored_tag) == authored_text: + return Finding( + input_element.kind, + input_element.name, + PORTABLE, + tag_preserved=authored_tag == "OSSIE_SQL_2026", + detail=f"returned as {authored_tag}", + ) + + # RELABELED: the authored expression came back verbatim, under a different tag. + for dialect, text in out_dialects.items(): + if text == authored_text and dialect != authored_tag: + return Finding( + input_element.kind, input_element.name, RELABELED, + detail=f"authored {authored_tag}, returned {dialect}", + ) + + portable_out = {d: t for d, t in out_dialects.items() if d in PORTABLE_DIALECTS} + + # REWRITTEN: a portable tag is present but the expression text changed. + if portable_out: + returned_text = portable_out.get(authored_tag) or next(iter(portable_out.values())) + equivalent = _equivalent_tree(authored_text, returned_text) + if equivalent is True: + note = "cosmetic (normalized trees equal)" + elif equivalent is False: + note = "review: normalized trees differ" + else: + note = "review: could not compare trees" + return Finding( + input_element.kind, input_element.name, REWRITTEN, + detail=f"{note}; returned {returned_text!r}", + ) + + # No portable tag: either a placeholder stand-in or a real vendor-only return. + if out_dialects: + sole_text = next(iter(out_dialects.values())).upper() + if sole_text in PLACEHOLDER_TOKENS: + return Finding( + input_element.kind, input_element.name, PLACEHOLDER, + detail=f"stand-in {next(iter(out_dialects.values()))!r} replaced the content", + ) + return Finding( + input_element.kind, input_element.name, SURVIVED_VENDOR_ONLY, + detail=f"returned as {','.join(sorted(out_dialects))}", + ) + return Finding(input_element.kind, input_element.name, SURVIVED_VENDOR_ONLY, detail="no expression returned") + + +def _repair_by_expression(dropped, added): + """Pair a DROPPED input element with an ADDED output element that carries an + equivalent expression, so a converter that did not preserve an element's name + (orionbelt writes a field's expression as its name) scores as one element whose + identity was not preserved, not as a drop plus an unrelated addition. + + Returns a list of (input_element, output_element) pairs and mutates nothing. + """ + pairs = [] + used = set() + for in_el in dropped: + in_text = next(iter(in_el.dialects.values()), "") + for out_el in added: + if id(out_el) in used: + continue + out_text = next(iter(out_el.dialects.values()), "") + if in_text and (_normalize(out_el.name) == in_text or out_text == in_text + or _equivalent_tree(in_text, out_text) is True): + pairs.append((in_el, out_el)) + used.add(id(out_el)) + break + return pairs + + +def grade(input_document, output_document, diagnostics_text="", converter=""): + """Grade a returned document against its input, element by element.""" + report = Report(converter=converter) + input_elements = _collect(input_document) + output_elements = _collect(output_document) + + # A converter that returns nothing refused the whole document (document scope). + if not output_elements: + report.rejected = True + for element in input_elements.values(): + report.findings.append(Finding(element.kind, element.name, REJECTED)) + return report + + diagnostics = diagnostics_text or "" + + present = {} + dropped = [] + for key, element in input_elements.items(): + if key in output_elements: + present[key] = _classify_present(element, output_elements[key]) + else: + dropped.append(element) + + added_elements = [el for key, el in output_elements.items() if key not in input_elements] + + # Recover identity: an unnamed-preserving rename shows up as drop + add. + repaired = _repair_by_expression(dropped, added_elements) + repaired_in = {id(a): b for a, b in repaired} + repaired_out = {id(b) for _, b in repaired} + + report.findings.extend(present.values()) + + for element in dropped: + if id(element) in repaired_in: + report.findings.append(Finding( + element.kind, element.name, REWRITTEN, + detail="identity not preserved: returned under a different name, expression equivalent", + )) + continue + report.findings.append(Finding( + element.kind, element.name, + DROPPED_DIAGNOSED if _diagnosed(element.name, diagnostics) else DROPPED_SILENT, + )) + + for element in added_elements: + if id(element) in repaired_out: + continue # accounted for by its repaired partner + report.findings.append(Finding(element.kind, element.name, ADDED, detail="not in input")) + + return report + + +def _format(report): + lines = [f"converter: {report.converter}" if report.converter else "grade report"] + for finding in report.findings: + flag = " (tag preserved)" if finding.tag_preserved else "" + detail = f" -- {finding.detail}" if finding.detail else "" + lines.append(f" [{finding.outcome}{flag}] {finding.kind} {finding.name}{detail}") + tally = report.counts() + lines.append("summary: " + ", ".join(f"{name}={count}" for name, count in sorted(tally.items()))) + return "\n".join(lines) + + +def main(argv=None): + parser = argparse.ArgumentParser(description="Grade a converter round-trip for #576.") + parser.add_argument("--input", required=True, help="the input .ossie.yaml/.json document") + parser.add_argument("--returned", required=True, help="the document the converter returned") + parser.add_argument("--diagnostics", help="a text file of the converter's warnings/notices") + parser.add_argument("--converter", default="", help="converter name, for the report header") + parser.add_argument("--json", action="store_true", help="emit JSON instead of text") + args = parser.parse_args(argv) + + diagnostics_text = "" + if args.diagnostics: + with open(args.diagnostics, "r", encoding="utf-8") as handle: + diagnostics_text = handle.read() + + report = grade( + _load(args.input), _load(args.returned), + diagnostics_text=diagnostics_text, converter=args.converter, + ) + + if args.json: + print(json.dumps({ + "converter": report.converter, + "rejected": report.rejected, + "findings": [vars(f) for f in report.findings], + "summary": report.counts(), + }, indent=2)) + else: + print(_format(report)) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/compliance/portability/grader/test_grade.py b/compliance/portability/grader/test_grade.py new file mode 100644 index 000000000..1b35bb84f --- /dev/null +++ b/compliance/portability/grader/test_grade.py @@ -0,0 +1,159 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +"""Outcome tests for the #576 fidelity grader. + +Each test builds a tiny input document and a synthetic returned document that +exercises exactly one outcome. The last test runs the real posted M0-M3 fixtures +through a no-op converter: returned unchanged, every element must grade PORTABLE. +""" + +import glob +import os + +import grade + + +def _metric(name, dialect, expression): + return {"name": name, "expression": {"dialects": [{"dialect": dialect, "expression": expression}]}} + + +def _field(dataset, name, dialect, expression): + return { + "name": dataset, + "fields": [{"name": name, "expression": {"dialects": [{"dialect": dialect, "expression": expression}]}}], + } + + +def _doc(metrics=None, datasets=None): + document = {"version": "0.2.0.dev0", "name": "t"} + if metrics is not None: + document["metrics"] = metrics + if datasets is not None: + document["datasets"] = datasets + return document + + +def _by_name(report): + return {f.name: f for f in report.findings} + + +def _only(report): + assert len(report.findings) == 1, [f.outcome for f in report.findings] + return report.findings[0] + + +def test_portable_ansi(): + doc = _doc(metrics=[_metric("total", "ANSI_SQL", "SUM(sales.amount)")]) + assert _only(grade.grade(doc, doc)).outcome == grade.PORTABLE + + +def test_portable_ossiesql_sets_tag_preserved(): + doc = _doc(metrics=[_metric("total", "OSSIE_SQL_2026", "SUM(sales.amount)")]) + finding = _only(grade.grade(doc, doc)) + assert finding.outcome == grade.PORTABLE + assert finding.tag_preserved is True + + +def test_relabeled_when_tag_changes_but_text_is_kept(): + src = _doc(metrics=[_metric("total", "OSSIE_SQL_2026", "SUM(sales.amount)")]) + returned = _doc(metrics=[_metric("total", "ANSI_SQL", "SUM(sales.amount)")]) + assert _only(grade.grade(src, returned)).outcome == grade.RELABELED + + +def test_rewritten_cosmetic_requalify_is_marked_benign(): + src = _doc(metrics=[_metric("total", "ANSI_SQL", "SUM(amount)")]) + returned = _doc(metrics=[_metric("total", "ANSI_SQL", "SUM(sales.amount)")]) + finding = _only(grade.grade(src, returned)) + assert finding.outcome == grade.REWRITTEN + # When sqlglot is present the requalify is recognised as cosmetic. + if grade._equivalent_tree("SUM(amount)", "SUM(sales.amount)") is True: + assert "cosmetic" in finding.detail + + +def test_rewritten_structural_change_is_flagged_for_review(): + src = _doc(metrics=[_metric("total", "ANSI_SQL", "SUM(sales.amount)")]) + returned = _doc(metrics=[_metric("total", "ANSI_SQL", "SUM(SUM(sales.amount))")]) + finding = _only(grade.grade(src, returned)) + assert finding.outcome == grade.REWRITTEN + if grade._equivalent_tree("SUM(sales.amount)", "SUM(SUM(sales.amount))") is False: + assert "review" in finding.detail + + +def test_survived_vendor_only_real_translation(): + src = _doc(metrics=[_metric("total", "ANSI_SQL", "SUM(sales.amount)")]) + returned = _doc(metrics=[_metric("total", "DAX", "SUM('Sales'[Amount])")]) + assert _only(grade.grade(src, returned)).outcome == grade.SURVIVED_VENDOR_ONLY + + +def test_placeholder_blank_stand_in(): + src = _doc(datasets=[_field("sales", "amount_with_tax", "ANSI_SQL", "amount * 1.1")]) + returned = _doc(datasets=[_field("sales", "amount_with_tax", "DAX", "BLANK()")]) + assert _only(grade.grade(src, returned)).outcome == grade.PLACEHOLDER + + +def test_dropped_silent_and_added(): + src = _doc(metrics=[_metric("total", "ANSI_SQL", "SUM(sales.amount)")]) + returned = _doc(metrics=[_metric("other", "ANSI_SQL", "SUM(sales.cost)")]) + outcomes = {name: f.outcome for name, f in _by_name(grade.grade(src, returned)).items()} + assert outcomes["total"] == grade.DROPPED_SILENT + assert outcomes["other"] == grade.ADDED + + +def test_dropped_diagnosed_when_warning_names_it(): + src = _doc(metrics=[_metric("margin", "ANSI_SQL", "SUM(sales.amount)")]) + returned = _doc(metrics=[_metric("keep", "ANSI_SQL", "SUM(sales.cost)")]) + report = grade.grade(src, returned, diagnostics_text="measure 'margin' excluded, preserved in stash") + assert _by_name(report)["margin"].outcome == grade.DROPPED_DIAGNOSED + + +def test_rejected_when_output_has_no_elements(): + src = _doc(metrics=[_metric("total", "ANSI_SQL", "SUM(sales.amount)")]) + report = grade.grade(src, {"version": "0.2.0.dev0", "name": "t"}) + assert report.rejected is True + assert _only(report).outcome == grade.REJECTED + + +def test_identity_not_preserved_is_repaired_not_dropped_plus_added(): + # orionbelt writes a field's expression as its name on the way back. + src = _doc(datasets=[_field("sales", "amount_with_tax", "ANSI_SQL", "amount * 1.1")]) + returned = _doc(datasets=[_field("sales", "amount * 1.1", "ANSI_SQL", "amount * 1.1")]) + report = grade.grade(src, returned) + outcomes = [f.outcome for f in report.findings] + assert outcomes == [grade.REWRITTEN], outcomes + assert "identity not preserved" in report.findings[0].detail + + +def test_whitespace_is_not_a_difference(): + src = _doc(metrics=[_metric("total", "ANSI_SQL", "SUM(sales.amount)")]) + returned = _doc(metrics=[_metric("total", "ANSI_SQL", "SUM( sales.amount )")]) + assert _only(grade.grade(src, returned)).outcome == grade.PORTABLE + + +def test_identity_on_real_m0_m3_fixtures_is_all_portable(): + here = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) + files = sorted( + path + for rung in ("m0", "m1", "m2", "m3") + for path in glob.glob(os.path.join(here, rung, "*.ossie.yaml")) + ) + assert files, "no M0-M3 fixtures found" + for path in files: + document = grade._load(path) + report = grade.grade(document, document) + bad = [(f.name, f.outcome) for f in report.findings if f.outcome != grade.PORTABLE] + assert not bad, f"{os.path.basename(path)} graded non-PORTABLE: {bad}" diff --git a/compliance/portability/m0/m0_ansi_qualified.ossie.yaml b/compliance/portability/m0/m0_ansi_qualified.ossie.yaml new file mode 100644 index 000000000..b1df704c2 --- /dev/null +++ b/compliance/portability/m0/m0_ansi_qualified.ossie.yaml @@ -0,0 +1,42 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# M0 ladder rung -- one dataset, one field, one SUM metric. +# Variant: ANSI_SQL tag, QUALIFIED reference form (dataset.field). PRIMARY fixture. +# The qualified form is carried as primary at every rung per the open #486 +# (qualified dataset.field only) decision; the bare form is a sibling variant +# (see m0_sum_bare.ossie.yaml) pending #343. +version: "0.2.0.dev0" +name: m0_sales +description: M0 baseline -- single dataset, single field, single SUM metric. +datasets: + - name: sales + source: samples.sales.orders + fields: + - name: amount + expression: + dialects: + - dialect: ANSI_SQL + expression: amount + datatype: Decimal +metrics: + - name: total_amount + expression: + dialects: + - dialect: ANSI_SQL + expression: SUM(sales.amount) + datatype: Decimal diff --git a/compliance/portability/m0/m0_ossiesql_qualified.ossie.yaml b/compliance/portability/m0/m0_ossiesql_qualified.ossie.yaml new file mode 100644 index 000000000..04244cac6 --- /dev/null +++ b/compliance/portability/m0/m0_ossiesql_qualified.ossie.yaml @@ -0,0 +1,45 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# yaml-language-server: $schema=../../../../core-spec/ossie-schema.json +# +# M0 ladder rung -- one dataset, one field, one SUM metric. +# Variant: OSSIE_SQL_2026 tag, QUALIFIED reference form (dataset.field). +# This is the variant that RELABELED on kayemkim's M0 run (dbt, wisdom, +# honeydew, orionbelt returned it tagged ANSI_SQL with the text unchanged; +# only cube kept the OSSIE_SQL_2026 tag). "tag preserved" is tracked as a +# sub-metric under PORTABLE so that 1-of-N signal survives. +version: "0.2.0.dev0" +name: m0_sales +description: M0 baseline -- single dataset, single field, single SUM metric. +datasets: + - name: sales + source: samples.sales.orders + fields: + - name: amount + expression: + dialects: + - dialect: OSSIE_SQL_2026 + expression: amount + datatype: Decimal +metrics: + - name: total_amount + expression: + dialects: + - dialect: OSSIE_SQL_2026 + expression: SUM(sales.amount) + datatype: Decimal diff --git a/compliance/portability/m0/m0_sum_bare.ossie.yaml b/compliance/portability/m0/m0_sum_bare.ossie.yaml new file mode 100644 index 000000000..aac63ba18 --- /dev/null +++ b/compliance/portability/m0/m0_sum_bare.ossie.yaml @@ -0,0 +1,47 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# yaml-language-server: $schema=../../../../core-spec/ossie-schema.json +# +# M0 ladder rung -- one dataset, one field, one SUM metric. +# Variant: BARE reference form -- SUM(amount), a bare column name rather than +# dataset.field. ANSI_SQL tag. This is the "SUM(amount)" variant from the runs. +# Kept as a sibling variant to the qualified primary, not baked into the rung, +# because #343 (bare name means a source column inside a dataset-scoped metric) +# and #486 (qualified dataset.field only) are still settling which form is the +# model-level reference. sigma drops this as a cross-dataset metric on a +# one-dataset model. +version: "0.2.0.dev0" +name: m0_sales +description: M0 baseline -- single dataset, single field, single SUM metric. +datasets: + - name: sales + source: samples.sales.orders + fields: + - name: amount + expression: + dialects: + - dialect: ANSI_SQL + expression: amount + datatype: Decimal +metrics: + - name: total_amount + expression: + dialects: + - dialect: ANSI_SQL + expression: SUM(amount) + datatype: Decimal diff --git a/compliance/portability/m1/m1_scalar_field.ossie.yaml b/compliance/portability/m1/m1_scalar_field.ossie.yaml new file mode 100644 index 000000000..fb75cd69a --- /dev/null +++ b/compliance/portability/m1/m1_scalar_field.ossie.yaml @@ -0,0 +1,49 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# M1 ladder rung: baseline plus one scalar field expression (calculated +# column), no aggregation. Element under test: the amount_with_tax field. +# Reference-form note: a field expression resolves inside its own dataset, so +# it references the sibling column bare (amount). The qualified/bare axis that +# #486 and #343 settle applies to metric references, not to intra-dataset field +# expressions, so this rung has a single fixture. +version: "0.2.0.dev0" +name: m1_sales +description: "M1: single dataset, one scalar calculated-column field expression." +datasets: + - name: sales + source: samples.sales.orders + fields: + - name: amount + expression: + dialects: + - dialect: ANSI_SQL + expression: amount + datatype: Decimal + - name: amount_with_tax + expression: + dialects: + - dialect: ANSI_SQL + expression: amount * 1.1 + datatype: Decimal +metrics: + - name: total_amount + expression: + dialects: + - dialect: ANSI_SQL + expression: SUM(sales.amount) + datatype: Decimal diff --git a/compliance/portability/m2/m2_aggregates_bare.ossie.yaml b/compliance/portability/m2/m2_aggregates_bare.ossie.yaml new file mode 100644 index 000000000..bf5ca36c3 --- /dev/null +++ b/compliance/portability/m2/m2_aggregates_bare.ossie.yaml @@ -0,0 +1,81 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# M2 ladder rung: the core aggregate set as metrics. Variant: BARE reference +# form (column name only), kept as a sibling variant pending #343. +version: "0.2.0.dev0" +name: m2_sales +description: "M2: core aggregate set, bare reference form." +datasets: + - name: sales + source: samples.sales.orders + fields: + - name: amount + expression: + dialects: + - dialect: ANSI_SQL + expression: amount + datatype: Decimal + - name: order_id + expression: + dialects: + - dialect: ANSI_SQL + expression: order_id + datatype: Integer + - name: customer_id + expression: + dialects: + - dialect: ANSI_SQL + expression: customer_id + datatype: Integer +metrics: + - name: sum_amount + expression: + dialects: + - dialect: ANSI_SQL + expression: SUM(amount) + datatype: Decimal + - name: count_orders + expression: + dialects: + - dialect: ANSI_SQL + expression: COUNT(order_id) + datatype: Integer + - name: distinct_customers + expression: + dialects: + - dialect: ANSI_SQL + expression: COUNT(DISTINCT customer_id) + datatype: Integer + - name: avg_amount + expression: + dialects: + - dialect: ANSI_SQL + expression: AVG(amount) + datatype: Decimal + - name: min_amount + expression: + dialects: + - dialect: ANSI_SQL + expression: MIN(amount) + datatype: Decimal + - name: max_amount + expression: + dialects: + - dialect: ANSI_SQL + expression: MAX(amount) + datatype: Decimal diff --git a/compliance/portability/m2/m2_aggregates_qualified.ossie.yaml b/compliance/portability/m2/m2_aggregates_qualified.ossie.yaml new file mode 100644 index 000000000..8ab5b6252 --- /dev/null +++ b/compliance/portability/m2/m2_aggregates_qualified.ossie.yaml @@ -0,0 +1,82 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# M2 ladder rung: the core aggregate set as metrics (SUM, COUNT, +# COUNT DISTINCT, AVG, MIN, MAX). Variant: QUALIFIED reference form +# (dataset.field), carried as primary per #486. +version: "0.2.0.dev0" +name: m2_sales +description: "M2: core aggregate set, qualified reference form." +datasets: + - name: sales + source: samples.sales.orders + fields: + - name: amount + expression: + dialects: + - dialect: ANSI_SQL + expression: amount + datatype: Decimal + - name: order_id + expression: + dialects: + - dialect: ANSI_SQL + expression: order_id + datatype: Integer + - name: customer_id + expression: + dialects: + - dialect: ANSI_SQL + expression: customer_id + datatype: Integer +metrics: + - name: sum_amount + expression: + dialects: + - dialect: ANSI_SQL + expression: SUM(sales.amount) + datatype: Decimal + - name: count_orders + expression: + dialects: + - dialect: ANSI_SQL + expression: COUNT(sales.order_id) + datatype: Integer + - name: distinct_customers + expression: + dialects: + - dialect: ANSI_SQL + expression: COUNT(DISTINCT sales.customer_id) + datatype: Integer + - name: avg_amount + expression: + dialects: + - dialect: ANSI_SQL + expression: AVG(sales.amount) + datatype: Decimal + - name: min_amount + expression: + dialects: + - dialect: ANSI_SQL + expression: MIN(sales.amount) + datatype: Decimal + - name: max_amount + expression: + dialects: + - dialect: ANSI_SQL + expression: MAX(sales.amount) + datatype: Decimal diff --git a/compliance/portability/m3/m3_filtered_agg_bare.ossie.yaml b/compliance/portability/m3/m3_filtered_agg_bare.ossie.yaml new file mode 100644 index 000000000..244483381 --- /dev/null +++ b/compliance/portability/m3/m3_filtered_agg_bare.ossie.yaml @@ -0,0 +1,46 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# M3 ladder rung: a filtered aggregate in the spec's portable form +# SUM(CASE WHEN ... THEN x ELSE 0 END). Variant: BARE reference form, kept as a +# sibling variant pending #343. +version: "0.2.0.dev0" +name: m3_sales +description: "M3: conditional aggregation, bare reference form." +datasets: + - name: sales + source: samples.sales.orders + fields: + - name: amount + expression: + dialects: + - dialect: ANSI_SQL + expression: amount + datatype: Decimal + - name: status + expression: + dialects: + - dialect: ANSI_SQL + expression: status + datatype: String +metrics: + - name: completed_amount + expression: + dialects: + - dialect: ANSI_SQL + expression: SUM(CASE WHEN status = 'completed' THEN amount ELSE 0 END) + datatype: Decimal diff --git a/compliance/portability/m3/m3_filtered_agg_qualified.ossie.yaml b/compliance/portability/m3/m3_filtered_agg_qualified.ossie.yaml new file mode 100644 index 000000000..0edaa573b --- /dev/null +++ b/compliance/portability/m3/m3_filtered_agg_qualified.ossie.yaml @@ -0,0 +1,46 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# M3 ladder rung: a filtered aggregate in the spec's portable form +# SUM(CASE WHEN ... THEN x ELSE 0 END). Variant: QUALIFIED reference form, +# carried as primary per #486. +version: "0.2.0.dev0" +name: m3_sales +description: "M3: conditional aggregation, qualified reference form." +datasets: + - name: sales + source: samples.sales.orders + fields: + - name: amount + expression: + dialects: + - dialect: ANSI_SQL + expression: amount + datatype: Decimal + - name: status + expression: + dialects: + - dialect: ANSI_SQL + expression: status + datatype: String +metrics: + - name: completed_amount + expression: + dialects: + - dialect: ANSI_SQL + expression: SUM(CASE WHEN sales.status = 'completed' THEN sales.amount ELSE 0 END) + datatype: Decimal diff --git a/compliance/portability/manifest.yaml b/compliance/portability/manifest.yaml new file mode 100644 index 000000000..0382f779f --- /dev/null +++ b/compliance/portability/manifest.yaml @@ -0,0 +1,105 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# Portability measurement ladder, rungs M0-M3 (strawman for #576). +# +# One capability is added per rung so that a difference in a converter's +# round-trip localizes to a single construct. Every rung is authored in a +# PORTABLE dialect (ANSI_SQL / OSSIE_SQL_2026) so it is vendor-neutral and runs +# through every converter; a vendor-tagged input would only exercise one +# converter, which is not what a cross-converter measurement wants. +# +# This manifest lists the fixtures and the single element each rung exercises. +# It does NOT encode per-converter expected outputs: those are measured by a +# grader that diffs the returned document against the input element by element +# into the outcome vocabulary below. Conformance stays a floor (validate.py +# passing); scoring stays a harness job (see #348). +# +# Reference form: the qualified dataset.field form is carried as the primary +# fixture wherever a metric references a dataset field, with the bare form as a +# sibling variant, pending #486 and #343. M1's element is an intra-dataset field +# expression, so it carries a single fixture. +# +# All fixtures validate.py-pass on apache/ossie main (9531429) with sqlglot +# 29.0.1. Rungs M4-M8 (relationships, metric composition, date scalars, window +# and offset functions) are held pending #354, #591 and #587; they are not in +# this set. + +outcomes: + # per-element, assigned by the grader after diffing input vs returned output + - PORTABLE # returns tagged ANSI_SQL or OSSIE_SQL_2026, expression unchanged + - TAG_PRESERVED # sub-flag under PORTABLE: the OSSIE_SQL_2026 tag survived + - REWRITTEN # same portable tag, expression text changed. The grader must + # compare normalized expressions (e.g. sqlglot trees after + # qualification and unquoting) so a benign requalify/quote/reorder + # (cube, dbt, omni, sigma, orionbelt, honeydew on this run) is + # separated from a change in meaning (dbt SUM(SUM(CASE...)), #437) + - SURVIVED_VENDOR_ONLY # returns in a vendor dialect only + - PLACEHOLDER # replaced by a valid stand-in that is not the authored content + # (e.g. microsoft emits BLANK() with the original stashed in an + # annotation the import does not read back), with a diagnostic. A + # diagnosed loss of content, distinct from SURVIVED_VENDOR_ONLY + - ADDED # an element appears that was not in the input + - RELABELED # same text, different tag than authored + - DROPPED_DIAGNOSED # removed with a warning or notice + - DROPPED_SILENT # removed with no diagnostic + - REJECTED # the converter refuses the whole document (document-level, e.g. + # thoughtspot exits non-zero when any one element will not translate) + +rungs: + - id: M0 + description: one dataset, one field, one SUM metric + element_under_test: + kind: metric + name: total_amount + construct: single-column aggregate (SUM) + reference_form_primary: qualified + variants: + - {id: M0.ansi.qualified, file: m0/m0_ansi_qualified.ossie.yaml, tag: ANSI_SQL, reference_form: qualified, role: primary} + - {id: M0.ossiesql.qualified, file: m0/m0_ossiesql_qualified.ossie.yaml, tag: OSSIE_SQL_2026, reference_form: qualified, role: tag-sensitivity} + - {id: M0.sum.bare, file: m0/m0_sum_bare.ossie.yaml, tag: ANSI_SQL, reference_form: bare, role: reference-form-variant} + + - id: M1 + description: baseline plus one scalar calculated-column field expression, no aggregation + element_under_test: + kind: field + name: amount_with_tax + construct: scalar field expression (amount * 1.1) + reference_form_primary: n/a # intra-dataset field expression references its sibling column bare + variants: + - {id: M1.scalar_field, file: m1/m1_scalar_field.ossie.yaml, tag: ANSI_SQL, reference_form: intra-dataset, role: primary} + + - id: M2 + description: the core aggregate set (SUM, COUNT, COUNT DISTINCT, AVG, MIN, MAX) + element_under_test: + kind: metrics + construct: core aggregate set + reference_form_primary: qualified + variants: + - {id: M2.aggregates.qualified, file: m2/m2_aggregates_qualified.ossie.yaml, tag: ANSI_SQL, reference_form: qualified, role: primary} + - {id: M2.aggregates.bare, file: m2/m2_aggregates_bare.ossie.yaml, tag: ANSI_SQL, reference_form: bare, role: reference-form-variant} + + - id: M3 + description: a filtered aggregate in the portable form SUM(CASE WHEN ... THEN x ELSE 0 END) + element_under_test: + kind: metric + name: completed_amount + construct: conditional aggregation + reference_form_primary: qualified + variants: + - {id: M3.filtered_agg.qualified, file: m3/m3_filtered_agg_qualified.ossie.yaml, tag: ANSI_SQL, reference_form: qualified, role: primary} + - {id: M3.filtered_agg.bare, file: m3/m3_filtered_agg_bare.ossie.yaml, tag: ANSI_SQL, reference_form: bare, role: reference-form-variant}