From 663980c328ae0907c3889d56847fe65ea0b21138 Mon Sep 17 00:00:00 2001 From: Chris Eubank <108756251+christianeu-db@users.noreply.github.com> Date: Fri, 23 Oct 2026 18:05:45 +0000 Subject: [PATCH 01/10] test(compliance): add query compliance test layout with case A1 Add compliance/query/ with the shared SCOTT data, an Ossie model over it, and one case (A1, grand total) for the Relational Query Interface. The README defines the case format, the semantics of the expected results (standard SQL), the rules to compare results, and the steps to add a case. Co-authored-by: Isaac --- compliance/README.md | 5 + compliance/query/README.md | 139 +++++++++++++++++ compliance/query/cases/a01_grand_total.sql | 37 +++++ compliance/query/scott/scott.sql | 82 ++++++++++ compliance/query/scott/scott_model.ossie.yaml | 143 ++++++++++++++++++ 5 files changed, 406 insertions(+) create mode 100644 compliance/query/README.md create mode 100644 compliance/query/cases/a01_grand_total.sql create mode 100644 compliance/query/scott/scott.sql create mode 100644 compliance/query/scott/scott_model.ossie.yaml diff --git a/compliance/README.md b/compliance/README.md index 481db96f2..e6003b54c 100644 --- a/compliance/README.md +++ b/compliance/README.md @@ -18,3 +18,8 @@ --> This is for compliance tests + +- [`portability/`](portability/): fixtures and a grader that measure how well a + model survives a round trip through each converter. +- [`query/`](query/): query test cases for the + [Relational Query Interface](../core-spec/relational_semantics.md). diff --git a/compliance/query/README.md b/compliance/query/README.md new file mode 100644 index 000000000..1fad2cf8d --- /dev/null +++ b/compliance/query/README.md @@ -0,0 +1,139 @@ + + + +Query compliance tests +====================== + +Test cases for the +[Relational Query Interface](../../core-spec/relational_semantics.md). + +This directory sets up the layout and the case format. It starts with one +case, A1. Further cases are added in later pull requests, each with one case +or a small group of cases. + +Layout +------ + +| Path | Contents | +|------|----------| +| [`scott/scott.sql`](scott/scott.sql) | The SCOTT schema and its data. All cases use it. | +| [`scott/scott_model.ossie.yaml`](scott/scott_model.ossie.yaml) | An Ossie model over SCOTT, with dataset-scoped metrics. All cases use it. | +| [`cases/`](cases/) | One file for each case. | + +Run +--- + +Run the data file, then the cases, with DuckDB. DuckDB needs no extensions: + +```sh +cat compliance/query/scott/scott.sql compliance/query/cases/*.sql | duckdb +``` + +To run one case, give only that case file after `scott.sql`. + +To check the model: + +```sh +uv run validation/validate.py compliance/query/scott/scott_model.ossie.yaml +``` + +Case format +----------- + +Each file in `cases/` has one case with four parts: + +1. A header: the case ID and title, one or two sentences about the behavior + under test, and the spec sections (for example, `Spec: 6, 7.2`). +2. An Ossie query that evaluates measures with `AGG()` (commented out). The + query refers to the model relations by two-part name, for example + `scott_model.emp`. +3. A reference query in DuckDB SQL over the base tables (runnable). + - For a case on one relation, the reference query is the Ossie query with + each `AGG(m)` replaced by the expression of `m`, and each model relation + replaced by its source table. + - For a case with joins, the reference query applies the reference + evaluation strategy of Section 9 of the spec. It removes join duplicates + by row identity (the dataset's `primary_key`), and then it aggregates. +4. The expected result (commented out), in this format: + - Column headers are in uppercase. + - Numbers are right-aligned. Strings and dates are left-aligned. + - NULL is shown as an empty cell. + - Values that are not exact (see "Comparison of results") are rounded to + 2 decimal places. + +See [`cases/a01_grand_total.sql`](cases/a01_grand_total.sql) for an example. + +Semantics of the expected results +--------------------------------- + +The expected results use standard SQL semantics (ISO/IEC 9075) for aggregates +and for scalar operators. They do not use the defaults of a particular engine. +Section 8.1 of the spec refers to the "host dialect". For these tests, the host +dialect is standard SQL. For example, `SUM` and `AVG` over no rows give NULL, +and `COUNT` over no rows gives 0. + +If the defaults of an engine are different (for example, if `SUM` over no rows +gives 0), run the engine in a mode with standard SQL semantics. + +Comparison of results +--------------------- + +Compare an actual result to the expected result by these rules: + +- Compare column names without regard to case. +- Compare row order only if the Ossie query has `ORDER BY`. +- Do not compare data types. The SQL standard lets an implementation choose + some result types. For example, the precision and scale of `AVG` over an + exact numeric type are implementation-defined. +- Integers, and exact numerics whose scale the standard fixes, must be equal. + For example, `COUNT(*)`, and `SUM` over `DECIMAL(7, 2)`, which has scale 2. +- Other numeric values (for example, `AVG`, division, and approximate numerics) + are not exact. An actual value matches if it differs from the expected value + by 0.01 or less. Thus rounding and truncation to 2 decimal places both match. + +Adding a case +------------- + +- Put one case in each file in `cases/`. Use the lowercase case ID, with a + two-digit number, and a short name, for example `a02_group_by.sql`. The + two-digit number keeps the files in ID order. +- Use the ID prefix `A` for a case on one relation, and `B` for a case with + joins. +- Use only the data in `scott/scott.sql`. Do not change the data, because the + expected results of all cases depend on it. +- Use the model in `scott/scott_model.ossie.yaml`. If a case needs a new metric, + add the metric in the same pull request, and check the model with + `validation/validate.py`. +- Take the expected result from the output of the reference query. Do not + calculate it by hand. If possible, confirm it with a second, independent + query. For a case with joins, the second query can write the input set of + each measure directly from Section 8.1, for example with `EXISTS` over the + join condition. +- Prefer queries whose result does not depend on implementation-defined + behavior. If a result does depend on it, say so in the header. + +Future work +----------- + +These tests are a light version of the `.ost` test format +([#501](https://github.com/apache/ossie/issues/501)) and the view expander +([#502](https://github.com/apache/ossie/issues/502)). When those are +available, each case can become an `.ost` test, and a runner can check the +Ossie query directly instead of its reference query. diff --git a/compliance/query/cases/a01_grand_total.sql b/compliance/query/cases/a01_grand_total.sql new file mode 100644 index 000000000..4cc0e7f08 --- /dev/null +++ b/compliance/query/cases/a01_grand_total.sql @@ -0,0 +1,37 @@ +-- Licensed to the Apache Software Foundation (ASF) under one +-- or more contributor license agreements. See the NOTICE file +-- distributed with this work for additional information +-- regarding copyright ownership. The ASF licenses this file +-- to you under the Apache License, Version 2.0 (the +-- "License"); you may not use this file except in compliance +-- with the License. You may obtain a copy of the License at +-- +-- http://www.apache.org/licenses/LICENSE-2.0 +-- +-- Unless required by applicable law or agreed to in writing, +-- software distributed under the License is distributed on an +-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +-- KIND, either express or implied. See the License for the +-- specific language governing permissions and limitations +-- under the License. + +-- ---------------------------------------------------------------------------- +-- A1. Grand total. +-- AGG() is an aggregate. Without GROUP BY, the query returns one row. +-- Spec: 6, 7.2 +-- ---------------------------------------------------------------------------- +-- Ossie: +-- SELECT AGG(sum_sal) AS sum_sal, +-- AGG(emp_count) AS emp_count, +-- AGG(avg_sal) AS avg_sal +-- FROM scott_model.emp; +SELECT SUM(sal) AS sum_sal, + COUNT(*) AS emp_count, + AVG(sal) AS avg_sal +FROM scott.emp; +-- +----------+-----------+---------+ +-- | SUM_SAL | EMP_COUNT | AVG_SAL | +-- +----------+-----------+---------+ +-- | 29025.00 | 14 | 2073.21 | +-- +----------+-----------+---------+ +-- (1 row) diff --git a/compliance/query/scott/scott.sql b/compliance/query/scott/scott.sql new file mode 100644 index 000000000..414165dcd --- /dev/null +++ b/compliance/query/scott/scott.sql @@ -0,0 +1,82 @@ +-- Licensed to the Apache Software Foundation (ASF) under one +-- or more contributor license agreements. See the NOTICE file +-- distributed with this work for additional information +-- regarding copyright ownership. The ASF licenses this file +-- to you under the Apache License, Version 2.0 (the +-- "License"); you may not use this file except in compliance +-- with the License. You may obtain a copy of the License at +-- +-- http://www.apache.org/licenses/LICENSE-2.0 +-- +-- Unless required by applicable law or agreed to in writing, +-- software distributed under the License is distributed on an +-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +-- KIND, either express or implied. See the License for the +-- specific language governing permissions and limitations +-- under the License. + +-- ============================================================================ +-- Data: the SCOTT schema +-- +-- Shared input data for the query compliance cases in ../cases/. Every case +-- file runs after this file. Do not change the values: the expected results +-- of all cases depend on them. +-- +-- Values are the same as the SCOTT tables in Apache Calcite +-- (core/src/test/resources/sql/within-distinct.iq). +-- ============================================================================ + +DROP SCHEMA IF EXISTS scott CASCADE; +CREATE SCHEMA scott; + +CREATE TABLE scott.dept ( + deptno INTEGER NOT NULL PRIMARY KEY, + dname VARCHAR NOT NULL, + loc VARCHAR NOT NULL +); + +INSERT INTO scott.dept VALUES + (10, 'ACCOUNTING', 'NEW YORK'), + (20, 'RESEARCH', 'DALLAS'), + (30, 'SALES', 'CHICAGO'), + (40, 'OPERATIONS', 'BOSTON'); + +CREATE TABLE scott.emp ( + empno INTEGER NOT NULL PRIMARY KEY, + ename VARCHAR NOT NULL, + job VARCHAR NOT NULL, + mgr INTEGER, + hiredate DATE NOT NULL, + sal DECIMAL(7, 2) NOT NULL, + comm DECIMAL(7, 2), + deptno INTEGER NOT NULL +); + +INSERT INTO scott.emp VALUES + (7369, 'SMITH', 'CLERK', 7902, DATE '1980-12-17', 800.00, NULL, 20), + (7499, 'ALLEN', 'SALESMAN', 7698, DATE '1981-02-20', 1600.00, 300.00, 30), + (7521, 'WARD', 'SALESMAN', 7698, DATE '1981-02-22', 1250.00, 500.00, 30), + (7566, 'JONES', 'MANAGER', 7839, DATE '1981-02-04', 2975.00, NULL, 20), + (7654, 'MARTIN', 'SALESMAN', 7698, DATE '1981-09-28', 1250.00, 1400.00, 30), + (7698, 'BLAKE', 'MANAGER', 7839, DATE '1981-01-05', 2850.00, NULL, 30), + (7782, 'CLARK', 'MANAGER', 7839, DATE '1981-06-09', 2450.00, NULL, 10), + (7788, 'SCOTT', 'ANALYST', 7566, DATE '1987-04-19', 3000.00, NULL, 20), + (7839, 'KING', 'PRESIDENT', NULL, DATE '1981-11-17', 5000.00, NULL, 10), + (7844, 'TURNER', 'SALESMAN', 7698, DATE '1981-09-08', 1500.00, 0.00, 30), + (7876, 'ADAMS', 'CLERK', 7788, DATE '1987-05-23', 1100.00, NULL, 20), + (7900, 'JAMES', 'CLERK', 7698, DATE '1981-12-03', 950.00, NULL, 30), + (7902, 'FORD', 'ANALYST', 7566, DATE '1981-12-03', 3000.00, NULL, 20), + (7934, 'MILLER', 'CLERK', 7782, DATE '1982-01-23', 1300.00, NULL, 10); + +CREATE TABLE scott.salgrade ( + grade INTEGER NOT NULL PRIMARY KEY, + losal INTEGER NOT NULL, + hisal INTEGER NOT NULL +); + +INSERT INTO scott.salgrade VALUES + (1, 700, 1200), + (2, 1201, 1400), + (3, 1401, 2000), + (4, 2001, 3000), + (5, 3001, 9999); diff --git a/compliance/query/scott/scott_model.ossie.yaml b/compliance/query/scott/scott_model.ossie.yaml new file mode 100644 index 000000000..ddbdd6514 --- /dev/null +++ b/compliance/query/scott/scott_model.ossie.yaml @@ -0,0 +1,143 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# yaml-language-server: $schema=../../../core-spec/ossie-schema.json +# +# Ossie semantic model over the SCOTT schema (scott.sql), for the query +# compliance cases in ../cases/. +# +# The metrics are dataset-scoped (datasets[].metrics). Each dataset is exposed +# as a relation with the same name, and each metric is a measure column of that +# relation (relational_semantics.md, Section 3.4). All relations of the model +# are in one schema with the name of the model (Section 11.2). The cases refer +# to these relations by two-part name: scott_model.emp, scott_model.dept, and +# scott_model.salgrade. +# +# The primary_key of each dataset is the row identity of its measures +# (Section 3.2). +# +# The emp-to-salgrade join (e.sal BETWEEN s.losal AND s.hisal) is a range join. +# An Ossie relationship cannot express it, so a case that uses it writes the +# join in the query (Section 11.3). + +version: 0.2.0.dev0 +name: scott_model +description: The SCOTT schema with dataset-scoped metrics +datasets: + - name: emp + source: scott.emp + primary_key: [empno] + fields: + - name: empno + datatype: Integer + expression: {dialects: [{dialect: OSSIE_SQL_2026, expression: empno}]} + - name: ename + datatype: String + expression: {dialects: [{dialect: OSSIE_SQL_2026, expression: ename}]} + - name: job + datatype: String + expression: {dialects: [{dialect: OSSIE_SQL_2026, expression: job}]} + - name: mgr + datatype: Integer + expression: {dialects: [{dialect: OSSIE_SQL_2026, expression: mgr}]} + - name: hiredate + datatype: Date + expression: {dialects: [{dialect: OSSIE_SQL_2026, expression: hiredate}]} + - name: sal + datatype: Decimal + expression: {dialects: [{dialect: OSSIE_SQL_2026, expression: sal}]} + - name: comm + datatype: Decimal + expression: {dialects: [{dialect: OSSIE_SQL_2026, expression: comm}]} + - name: deptno + datatype: Integer + expression: {dialects: [{dialect: OSSIE_SQL_2026, expression: deptno}]} + metrics: + - name: sum_sal + description: Total salary + datatype: Decimal + expression: {dialects: [{dialect: OSSIE_SQL_2026, expression: SUM(sal)}]} + - name: avg_sal + description: Average salary + datatype: Float + expression: {dialects: [{dialect: OSSIE_SQL_2026, expression: AVG(sal)}]} + - name: emp_count + description: Number of employees + datatype: Integer + expression: {dialects: [{dialect: OSSIE_SQL_2026, expression: COUNT(*)}]} + - name: comm_count + description: Number of employees that have a commission + datatype: Integer + expression: {dialects: [{dialect: OSSIE_SQL_2026, expression: COUNT(comm)}]} + - name: sum_comm + description: Total commission + datatype: Decimal + expression: {dialects: [{dialect: OSSIE_SQL_2026, expression: SUM(comm)}]} + - name: avg_comm + description: Average commission, over employees that have a commission + datatype: Float + expression: {dialects: [{dialect: OSSIE_SQL_2026, expression: AVG(comm)}]} + + - name: dept + source: scott.dept + primary_key: [deptno] + fields: + - name: deptno + datatype: Integer + expression: {dialects: [{dialect: OSSIE_SQL_2026, expression: deptno}]} + - name: dname + datatype: String + expression: {dialects: [{dialect: OSSIE_SQL_2026, expression: dname}]} + - name: loc + datatype: String + expression: {dialects: [{dialect: OSSIE_SQL_2026, expression: loc}]} + metrics: + - name: dept_count + description: Number of departments + datatype: Integer + expression: {dialects: [{dialect: OSSIE_SQL_2026, expression: COUNT(*)}]} + + - name: salgrade + source: scott.salgrade + primary_key: [grade] + fields: + - name: grade + datatype: Integer + expression: {dialects: [{dialect: OSSIE_SQL_2026, expression: grade}]} + - name: losal + datatype: Integer + expression: {dialects: [{dialect: OSSIE_SQL_2026, expression: losal}]} + - name: hisal + datatype: Integer + expression: {dialects: [{dialect: OSSIE_SQL_2026, expression: hisal}]} + metrics: + - name: grade_count + description: Number of salary grades + datatype: Integer + expression: {dialects: [{dialect: OSSIE_SQL_2026, expression: COUNT(*)}]} + +relationships: + - name: emp_to_dept + from: emp + to: dept + from_columns: [deptno] + to_columns: [deptno] + - name: emp_to_manager + from: emp + to: emp + from_columns: [mgr] + to_columns: [empno] From 94fb8a572cc17a238f673a3712202464b8c31445 Mon Sep 17 00:00:00 2001 From: Chris Eubank <108756251+christianeu-db@users.noreply.github.com> Date: Fri, 23 Oct 2026 18:56:01 +0000 Subject: [PATCH 02/10] docs(compliance): tighten the query compliance README Co-authored-by: Isaac --- compliance/query/README.md | 19 ++++++++----------- 1 file changed, 8 insertions(+), 11 deletions(-) diff --git a/compliance/query/README.md b/compliance/query/README.md index 1fad2cf8d..969d21680 100644 --- a/compliance/query/README.md +++ b/compliance/query/README.md @@ -24,19 +24,19 @@ Query compliance tests Test cases for the [Relational Query Interface](../../core-spec/relational_semantics.md). -This directory sets up the layout and the case format. It starts with one -case, A1. Further cases are added in later pull requests, each with one case -or a small group of cases. +This directory sets up the layout and the case format. Layout ------ | Path | Contents | |------|----------| -| [`scott/scott.sql`](scott/scott.sql) | The SCOTT schema and its data. All cases use it. | -| [`scott/scott_model.ossie.yaml`](scott/scott_model.ossie.yaml) | An Ossie model over SCOTT, with dataset-scoped metrics. All cases use it. | +| [`scott/scott.sql`](scott/scott.sql) | The SCOTT schema and its data. | +| [`scott/scott_model.ossie.yaml`](scott/scott_model.ossie.yaml) | An Ossie model over SCOTT, with dataset-scoped metrics. | | [`cases/`](cases/) | One file for each case. | +All cases use the same data and the same model. + Run --- @@ -84,10 +84,9 @@ Semantics of the expected results --------------------------------- The expected results use standard SQL semantics (ISO/IEC 9075) for aggregates -and for scalar operators. They do not use the defaults of a particular engine. -Section 8.1 of the spec refers to the "host dialect". For these tests, the host -dialect is standard SQL. For example, `SUM` and `AVG` over no rows give NULL, -and `COUNT` over no rows gives 0. +and for scalar operators. Section 8.1 of the spec refers to the "host +dialect". For these tests, the host dialect is standard SQL. For example, `SUM` +and `AVG` over no rows give NULL, and `COUNT` over no rows gives 0. If the defaults of an engine are different (for example, if `SUM` over no rows gives 0), run the engine in a mode with standard SQL semantics. @@ -95,8 +94,6 @@ gives 0), run the engine in a mode with standard SQL semantics. Comparison of results --------------------- -Compare an actual result to the expected result by these rules: - - Compare column names without regard to case. - Compare row order only if the Ossie query has `ORDER BY`. - Do not compare data types. The SQL standard lets an implementation choose From 120eaf8b4c4047472be19ec8c866ded32a147243 Mon Sep 17 00:00:00 2001 From: Chris Eubank <108756251+christianeu-db@users.noreply.github.com> Date: Fri, 23 Oct 2026 20:34:27 +0000 Subject: [PATCH 03/10] test(compliance): split each query case into query, expected, and reference files Each case is now a directory with query.sql (the Ossie query), expected.csv (the expected result), and reference.sql (the DuckDB query that produces expected.csv). An implementation can submit query.sql and compare its result to expected.csv, end to end. Co-authored-by: Isaac --- compliance/query/README.md | 107 +++++++++++------- .../query/cases/a01_grand_total/expected.csv | 2 + .../query.sql} | 20 +--- .../query/cases/a01_grand_total/reference.sql | 22 ++++ compliance/query/scott/scott.sql | 6 +- 5 files changed, 98 insertions(+), 59 deletions(-) create mode 100644 compliance/query/cases/a01_grand_total/expected.csv rename compliance/query/cases/{a01_grand_total.sql => a01_grand_total/query.sql} (70%) create mode 100644 compliance/query/cases/a01_grand_total/reference.sql diff --git a/compliance/query/README.md b/compliance/query/README.md index 969d21680..bd841f67b 100644 --- a/compliance/query/README.md +++ b/compliance/query/README.md @@ -33,20 +33,71 @@ Layout |------|----------| | [`scott/scott.sql`](scott/scott.sql) | The SCOTT schema and its data. | | [`scott/scott_model.ossie.yaml`](scott/scott_model.ossie.yaml) | An Ossie model over SCOTT, with dataset-scoped metrics. | -| [`cases/`](cases/) | One file for each case. | +| [`cases/`](cases/) | One directory for each case. | All cases use the same data and the same model. +Each case directory has three files: + +| File | Contents | +|------|----------| +| `query.sql` | The Ossie query. It evaluates measures with `AGG()`, and it refers to the model relations by two-part name, for example `scott_model.emp`. | +| `expected.csv` | The expected result of the Ossie query. | +| `reference.sql` | A DuckDB query over the base tables in `scott/scott.sql`. Its result is `expected.csv`. | + +An implementation passes a case if it loads the data and the model, accepts +`query.sql`, and returns a result that matches `expected.csv` by the rules in +"Comparison of results". The implementation does not use `reference.sql`. + +See [`cases/a01_grand_total/`](cases/a01_grand_total/) for an example. + +Case files +---------- + +`query.sql` starts with a header in SQL comments: the case ID and title, one or +two sentences about the behavior under test, and the spec sections (for +example, `Spec: 6, 7.2`). The query follows the header. A harness can submit +the file as it is. + +`reference.sql` uses the same column names as `query.sql`. + +- For a case on one relation, the reference query is the Ossie query with each + `AGG(m)` replaced by the expression of `m`, and each model relation replaced + by its source table. +- For a case with joins, the reference query applies the reference evaluation + strategy of Section 9 of the spec. It removes join duplicates by row identity + (the dataset's `primary_key`), and then it aggregates. + +`expected.csv` is a CSV file (RFC 4180) with one header row: + +- The header row holds the column names of the query. +- NULL is an empty field without quotes. An empty string is `""`. +- Values are not rounded. Exact numerics keep their scale, for example + `29025.00`. +- If the query has `ORDER BY`, the rows are in that order. + +DuckDB writes this format with `COPY ... TO ... (HEADER)`. + Run --- -Run the data file, then the cases, with DuckDB. DuckDB needs no extensions: +DuckDB needs no extensions. To run all reference queries: ```sh -cat compliance/query/scott/scott.sql compliance/query/cases/*.sql | duckdb +cat compliance/query/scott/scott.sql compliance/query/cases/*/reference.sql | duckdb ``` -To run one case, give only that case file after `scott.sql`. +To write `expected.csv` for one case from its reference query: + +```sh +CASE=compliance/query/cases/a01_grand_total +duckdb -c ".read compliance/query/scott/scott.sql" \ + -c "SET VARIABLE q = (SELECT content FROM read_text('$CASE/reference.sql'))" \ + -c "COPY (FROM query(getvariable('q'))) TO '$CASE/expected.csv' (HEADER)" +``` + +To check `expected.csv` instead, write to `/dev/stdout` and compare with +`diff - "$CASE/expected.csv"`. To check the model: @@ -54,32 +105,6 @@ To check the model: uv run validation/validate.py compliance/query/scott/scott_model.ossie.yaml ``` -Case format ------------ - -Each file in `cases/` has one case with four parts: - -1. A header: the case ID and title, one or two sentences about the behavior - under test, and the spec sections (for example, `Spec: 6, 7.2`). -2. An Ossie query that evaluates measures with `AGG()` (commented out). The - query refers to the model relations by two-part name, for example - `scott_model.emp`. -3. A reference query in DuckDB SQL over the base tables (runnable). - - For a case on one relation, the reference query is the Ossie query with - each `AGG(m)` replaced by the expression of `m`, and each model relation - replaced by its source table. - - For a case with joins, the reference query applies the reference - evaluation strategy of Section 9 of the spec. It removes join duplicates - by row identity (the dataset's `primary_key`), and then it aggregates. -4. The expected result (commented out), in this format: - - Column headers are in uppercase. - - Numbers are right-aligned. Strings and dates are left-aligned. - - NULL is shown as an empty cell. - - Values that are not exact (see "Comparison of results") are rounded to - 2 decimal places. - -See [`cases/a01_grand_total.sql`](cases/a01_grand_total.sql) for an example. - Semantics of the expected results --------------------------------- @@ -108,9 +133,10 @@ Comparison of results Adding a case ------------- -- Put one case in each file in `cases/`. Use the lowercase case ID, with a - two-digit number, and a short name, for example `a02_group_by.sql`. The - two-digit number keeps the files in ID order. +- Add one directory in `cases/` for each case, with the three files above. + Name the directory with the lowercase case ID, with a two-digit number, and + a short name, for example `a02_group_by`. The two-digit number keeps the + directories in ID order. - Use the ID prefix `A` for a case on one relation, and `B` for a case with joins. - Use only the data in `scott/scott.sql`. Do not change the data, because the @@ -118,13 +144,14 @@ Adding a case - Use the model in `scott/scott_model.ossie.yaml`. If a case needs a new metric, add the metric in the same pull request, and check the model with `validation/validate.py`. -- Take the expected result from the output of the reference query. Do not - calculate it by hand. If possible, confirm it with a second, independent - query. For a case with joins, the second query can write the input set of - each measure directly from Section 8.1, for example with `EXISTS` over the - join condition. +- Write `expected.csv` from `reference.sql` with the command in "Run". Do not + write it by hand. If possible, confirm it with a second, independent query. + For a case with joins, the second query can write the input set of each + measure directly from Section 8.1, for example with `EXISTS` over the join + condition. - Prefer queries whose result does not depend on implementation-defined - behavior. If a result does depend on it, say so in the header. + behavior. If a result does depend on it, say so in the header of + `query.sql`. Future work ----------- @@ -133,4 +160,4 @@ These tests are a light version of the `.ost` test format ([#501](https://github.com/apache/ossie/issues/501)) and the view expander ([#502](https://github.com/apache/ossie/issues/502)). When those are available, each case can become an `.ost` test, and a runner can check the -Ossie query directly instead of its reference query. +Ossie query directly. diff --git a/compliance/query/cases/a01_grand_total/expected.csv b/compliance/query/cases/a01_grand_total/expected.csv new file mode 100644 index 000000000..f5f9f51a0 --- /dev/null +++ b/compliance/query/cases/a01_grand_total/expected.csv @@ -0,0 +1,2 @@ +sum_sal,emp_count,avg_sal +29025.00,14,2073.214285714286 diff --git a/compliance/query/cases/a01_grand_total.sql b/compliance/query/cases/a01_grand_total/query.sql similarity index 70% rename from compliance/query/cases/a01_grand_total.sql rename to compliance/query/cases/a01_grand_total/query.sql index 4cc0e7f08..d7151d70b 100644 --- a/compliance/query/cases/a01_grand_total.sql +++ b/compliance/query/cases/a01_grand_total/query.sql @@ -14,24 +14,12 @@ -- KIND, either express or implied. See the License for the -- specific language governing permissions and limitations -- under the License. - -- ---------------------------------------------------------------------------- -- A1. Grand total. -- AGG() is an aggregate. Without GROUP BY, the query returns one row. -- Spec: 6, 7.2 -- ---------------------------------------------------------------------------- --- Ossie: --- SELECT AGG(sum_sal) AS sum_sal, --- AGG(emp_count) AS emp_count, --- AGG(avg_sal) AS avg_sal --- FROM scott_model.emp; -SELECT SUM(sal) AS sum_sal, - COUNT(*) AS emp_count, - AVG(sal) AS avg_sal -FROM scott.emp; --- +----------+-----------+---------+ --- | SUM_SAL | EMP_COUNT | AVG_SAL | --- +----------+-----------+---------+ --- | 29025.00 | 14 | 2073.21 | --- +----------+-----------+---------+ --- (1 row) +SELECT AGG(sum_sal) AS sum_sal, + AGG(emp_count) AS emp_count, + AGG(avg_sal) AS avg_sal +FROM scott_model.emp; diff --git a/compliance/query/cases/a01_grand_total/reference.sql b/compliance/query/cases/a01_grand_total/reference.sql new file mode 100644 index 000000000..569446b95 --- /dev/null +++ b/compliance/query/cases/a01_grand_total/reference.sql @@ -0,0 +1,22 @@ +-- Licensed to the Apache Software Foundation (ASF) under one +-- or more contributor license agreements. See the NOTICE file +-- distributed with this work for additional information +-- regarding copyright ownership. The ASF licenses this file +-- to you under the Apache License, Version 2.0 (the +-- "License"); you may not use this file except in compliance +-- with the License. You may obtain a copy of the License at +-- +-- http://www.apache.org/licenses/LICENSE-2.0 +-- +-- Unless required by applicable law or agreed to in writing, +-- software distributed under the License is distributed on an +-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +-- KIND, either express or implied. See the License for the +-- specific language governing permissions and limitations +-- under the License. +-- Reference query for case A1 (query.sql), over the base tables in +-- ../../scott/scott.sql. Its result is expected.csv. +SELECT SUM(sal) AS sum_sal, + COUNT(*) AS emp_count, + AVG(sal) AS avg_sal +FROM scott.emp; diff --git a/compliance/query/scott/scott.sql b/compliance/query/scott/scott.sql index 414165dcd..b0358ea23 100644 --- a/compliance/query/scott/scott.sql +++ b/compliance/query/scott/scott.sql @@ -18,9 +18,9 @@ -- ============================================================================ -- Data: the SCOTT schema -- --- Shared input data for the query compliance cases in ../cases/. Every case --- file runs after this file. Do not change the values: the expected results --- of all cases depend on them. +-- Shared input data for the query compliance cases in ../cases/. Load this +-- file before a case runs. Do not change the values: the expected results of +-- all cases depend on them. -- -- Values are the same as the SCOTT tables in Apache Calcite -- (core/src/test/resources/sql/within-distinct.iq). From 88bde11d8ed1d6400ac9b8d1a17d0dd0f88f818d Mon Sep 17 00:00:00 2001 From: Chris Eubank <108756251+christianeu-db@users.noreply.github.com> Date: Fri, 23 Oct 2026 20:53:45 +0000 Subject: [PATCH 04/10] test(compliance): name query case files by SQL dialect Rename query.sql to query.ossie.sql and reference.sql to reference.duckdb.sql, so that the file name shows the dialect. Co-authored-by: Isaac --- compliance/query/README.md | 39 +++++++++++-------- .../{query.sql => query.ossie.sql} | 0 .../{reference.sql => reference.duckdb.sql} | 2 +- 3 files changed, 23 insertions(+), 18 deletions(-) rename compliance/query/cases/a01_grand_total/{query.sql => query.ossie.sql} (100%) rename compliance/query/cases/a01_grand_total/{reference.sql => reference.duckdb.sql} (92%) diff --git a/compliance/query/README.md b/compliance/query/README.md index bd841f67b..abd089a1e 100644 --- a/compliance/query/README.md +++ b/compliance/query/README.md @@ -41,25 +41,29 @@ Each case directory has three files: | File | Contents | |------|----------| -| `query.sql` | The Ossie query. It evaluates measures with `AGG()`, and it refers to the model relations by two-part name, for example `scott_model.emp`. | +| `query.ossie.sql` | The Ossie query. It evaluates measures with `AGG()`, and it refers to the model relations by two-part name, for example `scott_model.emp`. | | `expected.csv` | The expected result of the Ossie query. | -| `reference.sql` | A DuckDB query over the base tables in `scott/scott.sql`. Its result is `expected.csv`. | +| `reference.duckdb.sql` | A DuckDB query over the base tables in `scott/scott.sql`. Its result is `expected.csv`. | + +The part of the file name before `.sql` gives the SQL dialect of the file: +`.ossie.sql` for an Ossie query, and `.duckdb.sql` for DuckDB SQL. An implementation passes a case if it loads the data and the model, accepts -`query.sql`, and returns a result that matches `expected.csv` by the rules in -"Comparison of results". The implementation does not use `reference.sql`. +`query.ossie.sql`, and returns a result that matches `expected.csv` by the +rules in "Comparison of results". The implementation does not use +`reference.duckdb.sql`. See [`cases/a01_grand_total/`](cases/a01_grand_total/) for an example. Case files ---------- -`query.sql` starts with a header in SQL comments: the case ID and title, one or -two sentences about the behavior under test, and the spec sections (for -example, `Spec: 6, 7.2`). The query follows the header. A harness can submit -the file as it is. +`query.ossie.sql` starts with a header in SQL comments: the case ID and +title, one or two sentences about the behavior under test, and the spec +sections (for example, `Spec: 6, 7.2`). The query follows the header. A +harness can submit the file as it is. -`reference.sql` uses the same column names as `query.sql`. +`reference.duckdb.sql` uses the same column names as `query.ossie.sql`. - For a case on one relation, the reference query is the Ossie query with each `AGG(m)` replaced by the expression of `m`, and each model relation replaced @@ -84,7 +88,8 @@ Run DuckDB needs no extensions. To run all reference queries: ```sh -cat compliance/query/scott/scott.sql compliance/query/cases/*/reference.sql | duckdb +cat compliance/query/scott/scott.sql \ + compliance/query/cases/*/reference.duckdb.sql | duckdb ``` To write `expected.csv` for one case from its reference query: @@ -92,7 +97,7 @@ To write `expected.csv` for one case from its reference query: ```sh CASE=compliance/query/cases/a01_grand_total duckdb -c ".read compliance/query/scott/scott.sql" \ - -c "SET VARIABLE q = (SELECT content FROM read_text('$CASE/reference.sql'))" \ + -c "SET VARIABLE q = (SELECT content FROM read_text('$CASE/reference.duckdb.sql'))" \ -c "COPY (FROM query(getvariable('q'))) TO '$CASE/expected.csv' (HEADER)" ``` @@ -144,14 +149,14 @@ Adding a case - Use the model in `scott/scott_model.ossie.yaml`. If a case needs a new metric, add the metric in the same pull request, and check the model with `validation/validate.py`. -- Write `expected.csv` from `reference.sql` with the command in "Run". Do not - write it by hand. If possible, confirm it with a second, independent query. - For a case with joins, the second query can write the input set of each - measure directly from Section 8.1, for example with `EXISTS` over the join - condition. +- Write `expected.csv` from `reference.duckdb.sql` with the command in "Run". + Do not write it by hand. If possible, confirm it with a second, independent + query. For a case with joins, the second query can write the input set of + each measure directly from Section 8.1, for example with `EXISTS` over the + join condition. - Prefer queries whose result does not depend on implementation-defined behavior. If a result does depend on it, say so in the header of - `query.sql`. + `query.ossie.sql`. Future work ----------- diff --git a/compliance/query/cases/a01_grand_total/query.sql b/compliance/query/cases/a01_grand_total/query.ossie.sql similarity index 100% rename from compliance/query/cases/a01_grand_total/query.sql rename to compliance/query/cases/a01_grand_total/query.ossie.sql diff --git a/compliance/query/cases/a01_grand_total/reference.sql b/compliance/query/cases/a01_grand_total/reference.duckdb.sql similarity index 92% rename from compliance/query/cases/a01_grand_total/reference.sql rename to compliance/query/cases/a01_grand_total/reference.duckdb.sql index 569446b95..cfe477b8a 100644 --- a/compliance/query/cases/a01_grand_total/reference.sql +++ b/compliance/query/cases/a01_grand_total/reference.duckdb.sql @@ -14,7 +14,7 @@ -- KIND, either express or implied. See the License for the -- specific language governing permissions and limitations -- under the License. --- Reference query for case A1 (query.sql), over the base tables in +-- Reference query for case A1 (query.ossie.sql), over the base tables in -- ../../scott/scott.sql. Its result is expected.csv. SELECT SUM(sal) AS sum_sal, COUNT(*) AS emp_count, From 9a4d96df8e1a4af0201a92e3aadf24df011e87a4 Mon Sep 17 00:00:00 2001 From: Chris Eubank <108756251+christianeu-db@users.noreply.github.com> Date: Fri, 23 Oct 2026 23:17:58 +0000 Subject: [PATCH 05/10] test(compliance): use DOUBLE PRECISION money columns and a relative tolerance Make emp.sal and emp.comm DOUBLE PRECISION. For AVG over an exact numeric, the SQL standard leaves the scale of the result implementation-defined, so it cannot be compared precisely. Replace the absolute 0.01 tolerance with a relative tolerance of 1e-9 for approximate numerics; exact numerics must be equal. Co-authored-by: Isaac --- compliance/query/README.md | 24 +++++++++++-------- .../query/cases/a01_grand_total/expected.csv | 2 +- compliance/query/scott/scott.sql | 11 ++++++--- compliance/query/scott/scott_model.ossie.yaml | 8 +++---- 4 files changed, 27 insertions(+), 18 deletions(-) diff --git a/compliance/query/README.md b/compliance/query/README.md index abd089a1e..d56c2daa2 100644 --- a/compliance/query/README.md +++ b/compliance/query/README.md @@ -76,8 +76,7 @@ harness can submit the file as it is. - The header row holds the column names of the query. - NULL is an empty field without quotes. An empty string is `""`. -- Values are not rounded. Exact numerics keep their scale, for example - `29025.00`. +- Values are not rounded, for example `2073.214285714286`. - If the query has `ORDER BY`, the rows are in that order. DuckDB writes this format with `COPY ... TO ... (HEADER)`. @@ -127,13 +126,15 @@ Comparison of results - Compare column names without regard to case. - Compare row order only if the Ossie query has `ORDER BY`. - Do not compare data types. The SQL standard lets an implementation choose - some result types. For example, the precision and scale of `AVG` over an - exact numeric type are implementation-defined. -- Integers, and exact numerics whose scale the standard fixes, must be equal. - For example, `COUNT(*)`, and `SUM` over `DECIMAL(7, 2)`, which has scale 2. -- Other numeric values (for example, `AVG`, division, and approximate numerics) - are not exact. An actual value matches if it differs from the expected value - by 0.01 or less. Thus rounding and truncation to 2 decimal places both match. + some result types, for example the precision of `COUNT(*)`. +- A value is exact or approximate by the type that the SQL standard gives to + its expression. `COUNT(*)` is exact. `SUM(sal)` and `AVG(sal)` are + approximate, because `sal` is `DOUBLE PRECISION`. +- Exact numerics must be equal. +- An approximate numeric matches if + `|actual - expected| <= 1e-9 * max(|expected|, 1)`. This is a relative + tolerance of 1e-9. For an expected value between -1 and 1, it is an absolute + tolerance of 1e-9. Adding a case ------------- @@ -154,7 +155,10 @@ Adding a case query. For a case with joins, the second query can write the input set of each measure directly from Section 8.1, for example with `EXISTS` over the join condition. -- Prefer queries whose result does not depend on implementation-defined +- Do not use an expression whose result type the SQL standard leaves + implementation-defined, such as `AVG` over an exact numeric or the division + of two exact numerics. Use a `DOUBLE PRECISION` column or a `CAST` instead. +- Prefer queries whose result does not depend on other implementation-defined behavior. If a result does depend on it, say so in the header of `query.ossie.sql`. diff --git a/compliance/query/cases/a01_grand_total/expected.csv b/compliance/query/cases/a01_grand_total/expected.csv index f5f9f51a0..191badb49 100644 --- a/compliance/query/cases/a01_grand_total/expected.csv +++ b/compliance/query/cases/a01_grand_total/expected.csv @@ -1,2 +1,2 @@ sum_sal,emp_count,avg_sal -29025.00,14,2073.214285714286 +29025.0,14,2073.214285714286 diff --git a/compliance/query/scott/scott.sql b/compliance/query/scott/scott.sql index b0358ea23..8fa50d377 100644 --- a/compliance/query/scott/scott.sql +++ b/compliance/query/scott/scott.sql @@ -23,7 +23,12 @@ -- all cases depend on them. -- -- Values are the same as the SCOTT tables in Apache Calcite --- (core/src/test/resources/sql/within-distinct.iq). +-- (core/src/test/resources/sql/within-distinct.iq), but the money columns +-- (emp.sal and emp.comm) are DOUBLE PRECISION, not DECIMAL(7, 2). For AVG +-- over an exact numeric, the SQL standard leaves the scale of the result +-- implementation-defined, so the result cannot be compared precisely. SUM and +-- AVG over DOUBLE PRECISION give approximate numerics, which ../README.md +-- compares with a tolerance. -- ============================================================================ DROP SCHEMA IF EXISTS scott CASCADE; @@ -47,8 +52,8 @@ CREATE TABLE scott.emp ( job VARCHAR NOT NULL, mgr INTEGER, hiredate DATE NOT NULL, - sal DECIMAL(7, 2) NOT NULL, - comm DECIMAL(7, 2), + sal DOUBLE PRECISION NOT NULL, + comm DOUBLE PRECISION, deptno INTEGER NOT NULL ); diff --git a/compliance/query/scott/scott_model.ossie.yaml b/compliance/query/scott/scott_model.ossie.yaml index ddbdd6514..c41ab4e45 100644 --- a/compliance/query/scott/scott_model.ossie.yaml +++ b/compliance/query/scott/scott_model.ossie.yaml @@ -58,10 +58,10 @@ datasets: datatype: Date expression: {dialects: [{dialect: OSSIE_SQL_2026, expression: hiredate}]} - name: sal - datatype: Decimal + datatype: Float expression: {dialects: [{dialect: OSSIE_SQL_2026, expression: sal}]} - name: comm - datatype: Decimal + datatype: Float expression: {dialects: [{dialect: OSSIE_SQL_2026, expression: comm}]} - name: deptno datatype: Integer @@ -69,7 +69,7 @@ datasets: metrics: - name: sum_sal description: Total salary - datatype: Decimal + datatype: Float expression: {dialects: [{dialect: OSSIE_SQL_2026, expression: SUM(sal)}]} - name: avg_sal description: Average salary @@ -85,7 +85,7 @@ datasets: expression: {dialects: [{dialect: OSSIE_SQL_2026, expression: COUNT(comm)}]} - name: sum_comm description: Total commission - datatype: Decimal + datatype: Float expression: {dialects: [{dialect: OSSIE_SQL_2026, expression: SUM(comm)}]} - name: avg_comm description: Average commission, over employees that have a commission From 9466500666bf0ca8c4d11e6a46d033b6509792b3 Mon Sep 17 00:00:00 2001 From: Chris Eubank <108756251+christianeu-db@users.noreply.github.com> Date: Fri, 23 Oct 2026 23:39:35 +0000 Subject: [PATCH 06/10] test(compliance): use expected.sql and exact comparison for query cases Replace expected.csv with expected.sql, a standard SQL query over literals. Define the comparison: columns by position, bag semantics, NULL-aware equality, and row order only for a unique ORDER BY. All expected values are exact, with no tolerance, so the money columns are DECIMAL(7, 2) again and the model and case A1 have no AVG. Co-authored-by: Isaac --- compliance/query/README.md | 120 +++++++++++------- .../query/cases/a01_grand_total/expected.csv | 2 - .../query/cases/a01_grand_total/expected.sql | 19 +++ .../cases/a01_grand_total/query.ossie.sql | 3 +- .../a01_grand_total/reference.duckdb.sql | 5 +- compliance/query/scott/scott.sql | 11 +- compliance/query/scott/scott_model.ossie.yaml | 20 ++- 7 files changed, 108 insertions(+), 72 deletions(-) delete mode 100644 compliance/query/cases/a01_grand_total/expected.csv create mode 100644 compliance/query/cases/a01_grand_total/expected.sql diff --git a/compliance/query/README.md b/compliance/query/README.md index d56c2daa2..288ddd75e 100644 --- a/compliance/query/README.md +++ b/compliance/query/README.md @@ -42,19 +42,26 @@ Each case directory has three files: | File | Contents | |------|----------| | `query.ossie.sql` | The Ossie query. It evaluates measures with `AGG()`, and it refers to the model relations by two-part name, for example `scott_model.emp`. | -| `expected.csv` | The expected result of the Ossie query. | -| `reference.duckdb.sql` | A DuckDB query over the base tables in `scott/scott.sql`. Its result is `expected.csv`. | +| `expected.sql` | The expected result of the Ossie query, as a standard SQL query over literals. | +| `reference.duckdb.sql` | A DuckDB query over the base tables in `scott/scott.sql`. Its result is equal to the result of `expected.sql`. | The part of the file name before `.sql` gives the SQL dialect of the file: -`.ossie.sql` for an Ossie query, and `.duckdb.sql` for DuckDB SQL. +`.ossie.sql` for an Ossie query, and `.duckdb.sql` for DuckDB SQL. A file name +without a dialect part, such as `expected.sql`, is standard SQL (ISO/IEC 9075). An implementation passes a case if it loads the data and the model, accepts -`query.ossie.sql`, and returns a result that matches `expected.csv` by the -rules in "Comparison of results". The implementation does not use -`reference.duckdb.sql`. +`query.ossie.sql`, and returns a result that is equal to the result of +`expected.sql` by the rules in "Comparison of results". The implementation does +not use `reference.duckdb.sql`. See [`cases/a01_grand_total/`](cases/a01_grand_total/) for an example. +The key words "MUST", "MUST NOT", "SHOULD", "SHOULD NOT", and "MAY" in the rest +of this document are to be interpreted as described in BCP 14 +([RFC 2119](https://www.rfc-editor.org/rfc/rfc2119), +[RFC 8174](https://www.rfc-editor.org/rfc/rfc8174)) when, and only when, they +appear in all capitals, as shown here. + Case files ---------- @@ -63,6 +70,24 @@ title, one or two sentences about the behavior under test, and the spec sections (for example, `Spec: 6, 7.2`). The query follows the header. A harness can submit the file as it is. +`expected.sql` is one query of this form: + +```sql +SELECT * +FROM (VALUES (29025.00, 14)) AS expected (sum_sal, emp_count); +``` + +- The column names and their order MUST be the same as in `query.ossie.sql`. +- The values MUST be literals of standard SQL: `NULL`, character string + literals (`'SALES'`), exact numeric literals (`29025.00`), date and time + literals (`DATE '1981-12-03'`), and `TRUE` or `FALSE`. Approximate numeric + literals (`2.5E3`) MUST NOT be used. +- If every value of a column is NULL, the first row MUST give the type with + `CAST(NULL AS )`, because a bare NULL has no type. +- If the expected result has no rows, `expected.sql` MUST have one row that + gives the types, and it MUST end with `WHERE 1 = 0`. +- If `query.ossie.sql` has `ORDER BY`, the rows MUST be in that order. + `reference.duckdb.sql` uses the same column names as `query.ossie.sql`. - For a case on one relation, the reference query is the Ossie query with each @@ -72,15 +97,6 @@ harness can submit the file as it is. strategy of Section 9 of the spec. It removes join duplicates by row identity (the dataset's `primary_key`), and then it aggregates. -`expected.csv` is a CSV file (RFC 4180) with one header row: - -- The header row holds the column names of the query. -- NULL is an empty field without quotes. An empty string is `""`. -- Values are not rounded, for example `2073.214285714286`. -- If the query has `ORDER BY`, the rows are in that order. - -DuckDB writes this format with `COPY ... TO ... (HEADER)`. - Run --- @@ -91,17 +107,23 @@ cat compliance/query/scott/scott.sql \ compliance/query/cases/*/reference.duckdb.sql | duckdb ``` -To write `expected.csv` for one case from its reference query: +To check one case, compare the result of `reference.duckdb.sql` with the result +of `expected.sql`. The command prints the number of rows that are different; +for a correct case, it is 0: ```sh CASE=compliance/query/cases/a01_grand_total duckdb -c ".read compliance/query/scott/scott.sql" \ - -c "SET VARIABLE q = (SELECT content FROM read_text('$CASE/reference.duckdb.sql'))" \ - -c "COPY (FROM query(getvariable('q'))) TO '$CASE/expected.csv' (HEADER)" + -c "SET VARIABLE r = (SELECT content FROM read_text('$CASE/reference.duckdb.sql'))" \ + -c "SET VARIABLE e = (SELECT content FROM read_text('$CASE/expected.sql'))" \ + -c "SELECT count(*) AS differences FROM ( + (FROM query(getvariable('r')) EXCEPT ALL FROM query(getvariable('e'))) + UNION ALL + (FROM query(getvariable('e')) EXCEPT ALL FROM query(getvariable('r'))))" ``` -To check `expected.csv` instead, write to `/dev/stdout` and compare with -`diff - "$CASE/expected.csv"`. +This command does not compare row order. For a case with `ORDER BY`, also +compare the order of the rows. To check the model: @@ -115,26 +137,37 @@ Semantics of the expected results The expected results use standard SQL semantics (ISO/IEC 9075) for aggregates and for scalar operators. Section 8.1 of the spec refers to the "host dialect". For these tests, the host dialect is standard SQL. For example, `SUM` -and `AVG` over no rows give NULL, and `COUNT` over no rows gives 0. +over no rows gives NULL, and `COUNT` over no rows gives 0. If the defaults of an engine are different (for example, if `SUM` over no rows -gives 0), run the engine in a mode with standard SQL semantics. +gives 0), the engine MUST run in a mode with standard SQL semantics. + +Every expected value is exact, and the comparison has no tolerance. Thus a +case MUST NOT have a result column whose type is approximate numeric (`REAL`, +`DOUBLE PRECISION`, or `FLOAT`), or whose scale the SQL standard leaves +implementation-defined. Examples of the second kind are `AVG` over an exact +numeric and the division of two exact numerics. For this reason, the model has +no `AVG` metric. Comparison of results --------------------- -- Compare column names without regard to case. -- Compare row order only if the Ossie query has `ORDER BY`. -- Do not compare data types. The SQL standard lets an implementation choose - some result types, for example the precision of `COUNT(*)`. -- A value is exact or approximate by the type that the SQL standard gives to - its expression. `COUNT(*)` is exact. `SUM(sal)` and `AVG(sal)` are - approximate, because `sal` is `DOUBLE PRECISION`. -- Exact numerics must be equal. -- An approximate numeric matches if - `|actual - expected| <= 1e-9 * max(|expected|, 1)`. This is a relative - tolerance of 1e-9. For an expected value between -1 and 1, it is an absolute - tolerance of 1e-9. +The actual result is the result of `query.ossie.sql`. The expected result is +the result of `expected.sql`. + +- The two results MUST have the same number of columns. Columns are compared by + position. Column names MUST be equal without regard to case. +- The two results MUST be equal as bags (multisets): each row MUST occur the + same number of times in both results. +- Two values are equal if both are NULL, or if neither is NULL and the SQL `=` + operator gives TRUE. This is `IS NOT DISTINCT FROM`. Data types are not + compared, so `29025.00` is equal to `29025`. +- If `query.ossie.sql` has `ORDER BY`, the rows MUST also be in the same order. + The `ORDER BY` of a case MUST give one order only: its sort key MUST be + unique in the result. + +In SQL, `(actual EXCEPT ALL expected) UNION ALL (expected EXCEPT ALL actual)` +gives no rows if and only if the two results are equal as bags. Adding a case ------------- @@ -150,17 +183,14 @@ Adding a case - Use the model in `scott/scott_model.ossie.yaml`. If a case needs a new metric, add the metric in the same pull request, and check the model with `validation/validate.py`. -- Write `expected.csv` from `reference.duckdb.sql` with the command in "Run". - Do not write it by hand. If possible, confirm it with a second, independent - query. For a case with joins, the second query can write the input set of - each measure directly from Section 8.1, for example with `EXISTS` over the - join condition. -- Do not use an expression whose result type the SQL standard leaves - implementation-defined, such as `AVG` over an exact numeric or the division - of two exact numerics. Use a `DOUBLE PRECISION` column or a `CAST` instead. -- Prefer queries whose result does not depend on other implementation-defined - behavior. If a result does depend on it, say so in the header of - `query.ossie.sql`. +- Copy the values in `expected.sql` from the output of `reference.duckdb.sql`. + Do not calculate them by hand. Check the case with the command in "Run". If + possible, confirm the result with a second, independent query. For a case + with joins, the second query can write the input set of each measure + directly from Section 8.1, for example with `EXISTS` over the join + condition. +- A case SHOULD NOT depend on other implementation-defined behavior. If a + result does depend on it, say so in the header of `query.ossie.sql`. Future work ----------- diff --git a/compliance/query/cases/a01_grand_total/expected.csv b/compliance/query/cases/a01_grand_total/expected.csv deleted file mode 100644 index 191badb49..000000000 --- a/compliance/query/cases/a01_grand_total/expected.csv +++ /dev/null @@ -1,2 +0,0 @@ -sum_sal,emp_count,avg_sal -29025.0,14,2073.214285714286 diff --git a/compliance/query/cases/a01_grand_total/expected.sql b/compliance/query/cases/a01_grand_total/expected.sql new file mode 100644 index 000000000..6b0c1c0e5 --- /dev/null +++ b/compliance/query/cases/a01_grand_total/expected.sql @@ -0,0 +1,19 @@ +-- Licensed to the Apache Software Foundation (ASF) under one +-- or more contributor license agreements. See the NOTICE file +-- distributed with this work for additional information +-- regarding copyright ownership. The ASF licenses this file +-- to you under the Apache License, Version 2.0 (the +-- "License"); you may not use this file except in compliance +-- with the License. You may obtain a copy of the License at +-- +-- http://www.apache.org/licenses/LICENSE-2.0 +-- +-- Unless required by applicable law or agreed to in writing, +-- software distributed under the License is distributed on an +-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +-- KIND, either express or implied. See the License for the +-- specific language governing permissions and limitations +-- under the License. +-- Expected result for case A1 (query.ossie.sql), in standard SQL. +SELECT * +FROM (VALUES (29025.00, 14)) AS expected (sum_sal, emp_count); diff --git a/compliance/query/cases/a01_grand_total/query.ossie.sql b/compliance/query/cases/a01_grand_total/query.ossie.sql index d7151d70b..d2a7fcf49 100644 --- a/compliance/query/cases/a01_grand_total/query.ossie.sql +++ b/compliance/query/cases/a01_grand_total/query.ossie.sql @@ -20,6 +20,5 @@ -- Spec: 6, 7.2 -- ---------------------------------------------------------------------------- SELECT AGG(sum_sal) AS sum_sal, - AGG(emp_count) AS emp_count, - AGG(avg_sal) AS avg_sal + AGG(emp_count) AS emp_count FROM scott_model.emp; diff --git a/compliance/query/cases/a01_grand_total/reference.duckdb.sql b/compliance/query/cases/a01_grand_total/reference.duckdb.sql index cfe477b8a..a8ef0d6c3 100644 --- a/compliance/query/cases/a01_grand_total/reference.duckdb.sql +++ b/compliance/query/cases/a01_grand_total/reference.duckdb.sql @@ -15,8 +15,7 @@ -- specific language governing permissions and limitations -- under the License. -- Reference query for case A1 (query.ossie.sql), over the base tables in --- ../../scott/scott.sql. Its result is expected.csv. +-- ../../scott/scott.sql. Its result is equal to the result of expected.sql. SELECT SUM(sal) AS sum_sal, - COUNT(*) AS emp_count, - AVG(sal) AS avg_sal + COUNT(*) AS emp_count FROM scott.emp; diff --git a/compliance/query/scott/scott.sql b/compliance/query/scott/scott.sql index 8fa50d377..b0358ea23 100644 --- a/compliance/query/scott/scott.sql +++ b/compliance/query/scott/scott.sql @@ -23,12 +23,7 @@ -- all cases depend on them. -- -- Values are the same as the SCOTT tables in Apache Calcite --- (core/src/test/resources/sql/within-distinct.iq), but the money columns --- (emp.sal and emp.comm) are DOUBLE PRECISION, not DECIMAL(7, 2). For AVG --- over an exact numeric, the SQL standard leaves the scale of the result --- implementation-defined, so the result cannot be compared precisely. SUM and --- AVG over DOUBLE PRECISION give approximate numerics, which ../README.md --- compares with a tolerance. +-- (core/src/test/resources/sql/within-distinct.iq). -- ============================================================================ DROP SCHEMA IF EXISTS scott CASCADE; @@ -52,8 +47,8 @@ CREATE TABLE scott.emp ( job VARCHAR NOT NULL, mgr INTEGER, hiredate DATE NOT NULL, - sal DOUBLE PRECISION NOT NULL, - comm DOUBLE PRECISION, + sal DECIMAL(7, 2) NOT NULL, + comm DECIMAL(7, 2), deptno INTEGER NOT NULL ); diff --git a/compliance/query/scott/scott_model.ossie.yaml b/compliance/query/scott/scott_model.ossie.yaml index c41ab4e45..e01461197 100644 --- a/compliance/query/scott/scott_model.ossie.yaml +++ b/compliance/query/scott/scott_model.ossie.yaml @@ -30,6 +30,10 @@ # The primary_key of each dataset is the row identity of its measures # (Section 3.2). # +# The model has no AVG metric. For AVG over an exact numeric, the SQL standard +# leaves the scale of the result implementation-defined, so the cases cannot +# compare it exactly (../README.md, "Comparison of results"). +# # The emp-to-salgrade join (e.sal BETWEEN s.losal AND s.hisal) is a range join. # An Ossie relationship cannot express it, so a case that uses it writes the # join in the query (Section 11.3). @@ -58,10 +62,10 @@ datasets: datatype: Date expression: {dialects: [{dialect: OSSIE_SQL_2026, expression: hiredate}]} - name: sal - datatype: Float + datatype: Decimal expression: {dialects: [{dialect: OSSIE_SQL_2026, expression: sal}]} - name: comm - datatype: Float + datatype: Decimal expression: {dialects: [{dialect: OSSIE_SQL_2026, expression: comm}]} - name: deptno datatype: Integer @@ -69,12 +73,8 @@ datasets: metrics: - name: sum_sal description: Total salary - datatype: Float + datatype: Decimal expression: {dialects: [{dialect: OSSIE_SQL_2026, expression: SUM(sal)}]} - - name: avg_sal - description: Average salary - datatype: Float - expression: {dialects: [{dialect: OSSIE_SQL_2026, expression: AVG(sal)}]} - name: emp_count description: Number of employees datatype: Integer @@ -85,12 +85,8 @@ datasets: expression: {dialects: [{dialect: OSSIE_SQL_2026, expression: COUNT(comm)}]} - name: sum_comm description: Total commission - datatype: Float + datatype: Decimal expression: {dialects: [{dialect: OSSIE_SQL_2026, expression: SUM(comm)}]} - - name: avg_comm - description: Average commission, over employees that have a commission - datatype: Float - expression: {dialects: [{dialect: OSSIE_SQL_2026, expression: AVG(comm)}]} - name: dept source: scott.dept From 15ed782cf12f87434b954913d89dfbe4ead31c53 Mon Sep 17 00:00:00 2001 From: Chris Eubank <108756251+christianeu-db@users.noreply.github.com> Date: Fri, 23 Oct 2026 23:40:17 +0000 Subject: [PATCH 07/10] docs(compliance): state that the reference query is non-normative Co-authored-by: Isaac --- compliance/query/README.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/compliance/query/README.md b/compliance/query/README.md index 288ddd75e..b02fa4d98 100644 --- a/compliance/query/README.md +++ b/compliance/query/README.md @@ -88,7 +88,10 @@ FROM (VALUES (29025.00, 14)) AS expected (sum_sal, emp_count); gives the types, and it MUST end with `WHERE 1 = 0`. - If `query.ossie.sql` has `ORDER BY`, the rows MUST be in that order. -`reference.duckdb.sql` uses the same column names as `query.ossie.sql`. +`reference.duckdb.sql` is non-normative. It helps to derive `expected.sql` and +to check it again. The spec is normative: if the reference query and the spec +disagree, the case is wrong, and it MUST be corrected to follow the spec. The +reference query uses the same column names as `query.ossie.sql`. - For a case on one relation, the reference query is the Ossie query with each `AGG(m)` replaced by the expression of `m`, and each model relation replaced From ceacbebce93c8925c436b92538d4f669b3356292 Mon Sep 17 00:00:00 2001 From: Chris Eubank <108756251+christianeu-db@users.noreply.github.com> Date: Sat, 24 Oct 2026 01:18:39 +0000 Subject: [PATCH 08/10] docs(compliance): tighten the query README and allow approximate numerics Allow approximate numeric results, compared with a relative tolerance of 1e-9 and an absolute tolerance of 1e-9 near zero, and state that this rule is provisional. Replace the EXCEPT ALL check with a command that shows the reference result and the expected result. Co-authored-by: Isaac --- compliance/query/README.md | 229 +++++++++++++++---------------------- 1 file changed, 94 insertions(+), 135 deletions(-) diff --git a/compliance/query/README.md b/compliance/query/README.md index b02fa4d98..f4806b897 100644 --- a/compliance/query/README.md +++ b/compliance/query/README.md @@ -21,87 +21,101 @@ Query compliance tests ====================== -Test cases for the -[Relational Query Interface](../../core-spec/relational_semantics.md). +Compliance tests for the +[Relational Query Interface](../../core-spec/relational_semantics.md). All +cases use the same SCOTT data and Ossie model. -This directory sets up the layout and the case format. +The key words MUST, MUST NOT, SHOULD, SHOULD NOT, and MAY are to be +interpreted as described in BCP 14 +([RFC 2119](https://www.rfc-editor.org/rfc/rfc2119), +[RFC 8174](https://www.rfc-editor.org/rfc/rfc8174)) when they appear in +uppercase. Layout ------ | Path | Contents | |------|----------| -| [`scott/scott.sql`](scott/scott.sql) | The SCOTT schema and its data. | -| [`scott/scott_model.ossie.yaml`](scott/scott_model.ossie.yaml) | An Ossie model over SCOTT, with dataset-scoped metrics. | -| [`cases/`](cases/) | One directory for each case. | - -All cases use the same data and the same model. - -Each case directory has three files: - -| File | Contents | -|------|----------| -| `query.ossie.sql` | The Ossie query. It evaluates measures with `AGG()`, and it refers to the model relations by two-part name, for example `scott_model.emp`. | -| `expected.sql` | The expected result of the Ossie query, as a standard SQL query over literals. | -| `reference.duckdb.sql` | A DuckDB query over the base tables in `scott/scott.sql`. Its result is equal to the result of `expected.sql`. | +| [`scott/scott.sql`](scott/scott.sql) | SCOTT schema and data | +| [`scott/scott_model.ossie.yaml`](scott/scott_model.ossie.yaml) | Ossie model with dataset-scoped metrics | +| `cases//query.ossie.sql` | Ossie query under test | +| `cases//expected.sql` | Expected result, as standard SQL over literals | +| `cases//reference.duckdb.sql` | Non-normative DuckDB query that derives the expected result | -The part of the file name before `.sql` gives the SQL dialect of the file: -`.ossie.sql` for an Ossie query, and `.duckdb.sql` for DuckDB SQL. A file name -without a dialect part, such as `expected.sql`, is standard SQL (ISO/IEC 9075). +The file name suffix gives the SQL dialect: `.ossie.sql` for Ossie SQL, +`.duckdb.sql` for DuckDB SQL, and `.sql` for standard SQL (ISO/IEC 9075). -An implementation passes a case if it loads the data and the model, accepts -`query.ossie.sql`, and returns a result that is equal to the result of -`expected.sql` by the rules in "Comparison of results". The implementation does -not use `reference.duckdb.sql`. - -See [`cases/a01_grand_total/`](cases/a01_grand_total/) for an example. - -The key words "MUST", "MUST NOT", "SHOULD", "SHOULD NOT", and "MAY" in the rest -of this document are to be interpreted as described in BCP 14 -([RFC 2119](https://www.rfc-editor.org/rfc/rfc2119), -[RFC 8174](https://www.rfc-editor.org/rfc/rfc8174)) when, and only when, they -appear in all capitals, as shown here. +A case passes when an implementation loads the data and the model, runs +`query.ossie.sql`, and returns a result equal to the result of `expected.sql` +(see "Comparing results"). For an example, see +[`cases/a01_grand_total/`](cases/a01_grand_total/). Case files ---------- -`query.ossie.sql` starts with a header in SQL comments: the case ID and -title, one or two sentences about the behavior under test, and the spec -sections (for example, `Spec: 6, 7.2`). The query follows the header. A -harness can submit the file as it is. +`query.ossie.sql` starts with a comment header: the case ID and title, the +behavior under test, and the spec sections (for example, `Spec: 6, 7.2`). The +file can be submitted to an implementation as it is. -`expected.sql` is one query of this form: +`expected.sql` is one query: ```sql SELECT * FROM (VALUES (29025.00, 14)) AS expected (sum_sal, emp_count); ``` -- The column names and their order MUST be the same as in `query.ossie.sql`. -- The values MUST be literals of standard SQL: `NULL`, character string - literals (`'SALES'`), exact numeric literals (`29025.00`), date and time - literals (`DATE '1981-12-03'`), and `TRUE` or `FALSE`. Approximate numeric - literals (`2.5E3`) MUST NOT be used. -- If every value of a column is NULL, the first row MUST give the type with - `CAST(NULL AS )`, because a bare NULL has no type. -- If the expected result has no rows, `expected.sql` MUST have one row that - gives the types, and it MUST end with `WHERE 1 = 0`. +- Column names and order MUST match `query.ossie.sql`. +- Values MUST be standard SQL literals. An exact result MUST be an exact + numeric literal (`29025.00`). An approximate result MUST be an approximate + numeric literal (`2.073214285714286E3`). +- A column that is NULL in every row MUST give its type in the first row, with + `CAST(NULL AS )`. +- An empty result MUST use one typed row and `WHERE 1 = 0`. - If `query.ossie.sql` has `ORDER BY`, the rows MUST be in that order. -`reference.duckdb.sql` is non-normative. It helps to derive `expected.sql` and -to check it again. The spec is normative: if the reference query and the spec -disagree, the case is wrong, and it MUST be corrected to follow the spec. The -reference query uses the same column names as `query.ossie.sql`. - -- For a case on one relation, the reference query is the Ossie query with each - `AGG(m)` replaced by the expression of `m`, and each model relation replaced - by its source table. -- For a case with joins, the reference query applies the reference evaluation - strategy of Section 9 of the spec. It removes join duplicates by row identity - (the dataset's `primary_key`), and then it aggregates. - -Run ---- +`reference.duckdb.sql` is non-normative. If it disagrees with the spec, the +spec is correct, and the case MUST be fixed. It uses the column names of +`query.ossie.sql`. For one relation, it replaces each `AGG(m)` with the +expression of `m`, and each model relation with its source table. For joins, +it follows the reference evaluation strategy of Section 9: it removes join +duplicates by `primary_key`, and then it aggregates. + +Result semantics +---------------- + +Expected results use standard SQL (ISO/IEC 9075) semantics. For these tests, +the host dialect of Section 8.1 is standard SQL. For example, `SUM` over no +rows is NULL, and `COUNT` over no rows is 0. An implementation with other +defaults MUST run the tests with standard SQL semantics. + +A case MUST NOT return an exact numeric whose scale the standard leaves +implementation-defined, such as `AVG` over an exact numeric or the division of +two exact numerics. For this reason, the model has no `AVG` metric. A case +MUST NOT return NaN or an infinity. + +Comparing results +----------------- + +- Columns are matched by position. The column counts MUST be equal. Column + names are compared case-insensitively. +- Rows MUST match as multisets, including the number of duplicates. +- If `query.ossie.sql` has `ORDER BY`, the row order MUST also match, and the + sort key MUST order the result uniquely. +- Two values match if both are NULL, or if neither is NULL and they are equal. + Types are not compared: `29025.00` matches `29025`. +- Exact numerics MUST be equal. +- An approximate numeric `a` matches an expected value `e` if + `|a - e| <= 1e-9 * max(|a|, |e|, 1)`. This is a relative tolerance of 1e-9, + with an absolute tolerance of 1e-9 near zero. + +The rule for approximate numerics is provisional. How to compare +floating-point results across engines is an open question: the size of the +tolerance, the tolerance near zero, and how to match rows as multisets when +values match only within the tolerance. Until it is settled, cases SHOULD use +exact results. + +Running the tests +----------------- DuckDB needs no extensions. To run all reference queries: @@ -110,96 +124,41 @@ cat compliance/query/scott/scott.sql \ compliance/query/cases/*/reference.duckdb.sql | duckdb ``` -To check one case, compare the result of `reference.duckdb.sql` with the result -of `expected.sql`. The command prints the number of rows that are different; -for a correct case, it is 0: +To show the result of a reference query above the result of its +`expected.sql`: ```sh CASE=compliance/query/cases/a01_grand_total -duckdb -c ".read compliance/query/scott/scott.sql" \ - -c "SET VARIABLE r = (SELECT content FROM read_text('$CASE/reference.duckdb.sql'))" \ - -c "SET VARIABLE e = (SELECT content FROM read_text('$CASE/expected.sql'))" \ - -c "SELECT count(*) AS differences FROM ( - (FROM query(getvariable('r')) EXCEPT ALL FROM query(getvariable('e'))) - UNION ALL - (FROM query(getvariable('e')) EXCEPT ALL FROM query(getvariable('r'))))" +cat compliance/query/scott/scott.sql \ + "$CASE/reference.duckdb.sql" "$CASE/expected.sql" | duckdb ``` -This command does not compare row order. For a case with `ORDER BY`, also -compare the order of the rows. - -To check the model: +To validate the model: ```sh uv run validation/validate.py compliance/query/scott/scott_model.ossie.yaml ``` -Semantics of the expected results ---------------------------------- - -The expected results use standard SQL semantics (ISO/IEC 9075) for aggregates -and for scalar operators. Section 8.1 of the spec refers to the "host -dialect". For these tests, the host dialect is standard SQL. For example, `SUM` -over no rows gives NULL, and `COUNT` over no rows gives 0. - -If the defaults of an engine are different (for example, if `SUM` over no rows -gives 0), the engine MUST run in a mode with standard SQL semantics. - -Every expected value is exact, and the comparison has no tolerance. Thus a -case MUST NOT have a result column whose type is approximate numeric (`REAL`, -`DOUBLE PRECISION`, or `FLOAT`), or whose scale the SQL standard leaves -implementation-defined. Examples of the second kind are `AVG` over an exact -numeric and the division of two exact numerics. For this reason, the model has -no `AVG` metric. - -Comparison of results ---------------------- - -The actual result is the result of `query.ossie.sql`. The expected result is -the result of `expected.sql`. - -- The two results MUST have the same number of columns. Columns are compared by - position. Column names MUST be equal without regard to case. -- The two results MUST be equal as bags (multisets): each row MUST occur the - same number of times in both results. -- Two values are equal if both are NULL, or if neither is NULL and the SQL `=` - operator gives TRUE. This is `IS NOT DISTINCT FROM`. Data types are not - compared, so `29025.00` is equal to `29025`. -- If `query.ossie.sql` has `ORDER BY`, the rows MUST also be in the same order. - The `ORDER BY` of a case MUST give one order only: its sort key MUST be - unique in the result. - -In SQL, `(actual EXCEPT ALL expected) UNION ALL (expected EXCEPT ALL actual)` -gives no rows if and only if the two results are equal as bags. - Adding a case ------------- -- Add one directory in `cases/` for each case, with the three files above. - Name the directory with the lowercase case ID, with a two-digit number, and - a short name, for example `a02_group_by`. The two-digit number keeps the - directories in ID order. -- Use the ID prefix `A` for a case on one relation, and `B` for a case with - joins. -- Use only the data in `scott/scott.sql`. Do not change the data, because the - expected results of all cases depend on it. -- Use the model in `scott/scott_model.ossie.yaml`. If a case needs a new metric, - add the metric in the same pull request, and check the model with - `validation/validate.py`. -- Copy the values in `expected.sql` from the output of `reference.duckdb.sql`. - Do not calculate them by hand. Check the case with the command in "Run". If - possible, confirm the result with a second, independent query. For a case - with joins, the second query can write the input set of each measure - directly from Section 8.1, for example with `EXISTS` over the join - condition. -- A case SHOULD NOT depend on other implementation-defined behavior. If a - result does depend on it, say so in the header of `query.ossie.sql`. +- Add a directory under `cases/` with the three files. Name it with the + lowercase case ID, with a two-digit number, and a short name, for example + `a02_group_by`. Use the ID prefix `A` for one relation and `B` for joins. +- Do not change `scott/scott.sql`. Add any new metric to + `scott/scott_model.ossie.yaml` in the same pull request, and validate the + model. +- Copy `expected.sql` from the output of `reference.duckdb.sql`. Do not + calculate it by hand. Where possible, confirm it with a second, independent + query. For a join case, that query can build the input set of each measure + from Section 8.1, for example with `EXISTS` over the join condition. +- Cases SHOULD NOT depend on other implementation-defined behavior. Document + any such dependency in the header of `query.ossie.sql`. Future work ----------- -These tests are a light version of the `.ost` test format -([#501](https://github.com/apache/ossie/issues/501)) and the view expander -([#502](https://github.com/apache/ossie/issues/502)). When those are -available, each case can become an `.ost` test, and a runner can check the -Ossie query directly. +The cases can move to the `.ost` test format +([#501](https://github.com/apache/ossie/issues/501)) and use the view expander +([#502](https://github.com/apache/ossie/issues/502)) to run Ossie queries +directly. From 1afff0137f2eb1279387ee1ac5fdc9896c6c1538 Mon Sep 17 00:00:00 2001 From: Chris Eubank <108756251+christianeu-db@users.noreply.github.com> Date: Sat, 24 Oct 2026 01:27:39 +0000 Subject: [PATCH 09/10] docs(compliance): allow AVG over an approximate argument State why the tolerance does not cover AVG over an exact numeric, and that a case can use AVG over a cast to DOUBLE PRECISION. Co-authored-by: Isaac --- compliance/query/README.md | 5 ++++- compliance/query/scott/scott_model.ossie.yaml | 7 ++++--- 2 files changed, 8 insertions(+), 4 deletions(-) diff --git a/compliance/query/README.md b/compliance/query/README.md index f4806b897..199d5c849 100644 --- a/compliance/query/README.md +++ b/compliance/query/README.md @@ -90,7 +90,10 @@ defaults MUST run the tests with standard SQL semantics. A case MUST NOT return an exact numeric whose scale the standard leaves implementation-defined, such as `AVG` over an exact numeric or the division of -two exact numerics. For this reason, the model has no `AVG` metric. A case +two exact numerics. The tolerance does not cover these results: for `AVG` over +`DECIMAL(7, 2)`, an implementation can return a scale of 2, which differs from +the true value by more than 1e-9. To use `AVG`, cast its argument to an +approximate type, for example `AVG(CAST(sal AS DOUBLE PRECISION))`. A case MUST NOT return NaN or an infinity. Comparing results diff --git a/compliance/query/scott/scott_model.ossie.yaml b/compliance/query/scott/scott_model.ossie.yaml index e01461197..442f0ab4d 100644 --- a/compliance/query/scott/scott_model.ossie.yaml +++ b/compliance/query/scott/scott_model.ossie.yaml @@ -30,9 +30,10 @@ # The primary_key of each dataset is the row identity of its measures # (Section 3.2). # -# The model has no AVG metric. For AVG over an exact numeric, the SQL standard -# leaves the scale of the result implementation-defined, so the cases cannot -# compare it exactly (../README.md, "Comparison of results"). +# The model has no AVG metric yet. For AVG over an exact numeric, the SQL +# standard leaves the scale of the result implementation-defined, so a case +# that needs AVG adds a metric over an approximate argument, for example +# AVG(CAST(sal AS DOUBLE PRECISION)) (../README.md, "Result semantics"). # # The emp-to-salgrade join (e.sal BETWEEN s.losal AND s.hisal) is a range join. # An Ossie relationship cannot express it, so a case that uses it writes the From f0b6c368eb38312d64336ce3c4eb75ff84f849de Mon Sep 17 00:00:00 2001 From: Chris Eubank <108756251+christianeu-db@users.noreply.github.com> Date: Sat, 24 Oct 2026 01:29:37 +0000 Subject: [PATCH 10/10] docs(compliance): drop the second-query advice from the query README Co-authored-by: Isaac --- compliance/query/README.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/compliance/query/README.md b/compliance/query/README.md index 199d5c849..98886a28f 100644 --- a/compliance/query/README.md +++ b/compliance/query/README.md @@ -152,9 +152,7 @@ Adding a case `scott/scott_model.ossie.yaml` in the same pull request, and validate the model. - Copy `expected.sql` from the output of `reference.duckdb.sql`. Do not - calculate it by hand. Where possible, confirm it with a second, independent - query. For a join case, that query can build the input set of each measure - from Section 8.1, for example with `EXISTS` over the join condition. + calculate it by hand. - Cases SHOULD NOT depend on other implementation-defined behavior. Document any such dependency in the header of `query.ossie.sql`.