diff --git a/compliance/README.md b/compliance/README.md index 481db96f..e6003b54 100644 --- a/compliance/README.md +++ b/compliance/README.md @@ -18,3 +18,8 @@ --> This is for compliance tests + +- [`portability/`](portability/): fixtures and a grader that measure how well a + model survives a round trip through each converter. +- [`query/`](query/): query test cases for the + [Relational Query Interface](../core-spec/relational_semantics.md). diff --git a/compliance/query/README.md b/compliance/query/README.md new file mode 100644 index 00000000..98886a28 --- /dev/null +++ b/compliance/query/README.md @@ -0,0 +1,165 @@ + + + +Query compliance tests +====================== + +Compliance tests for the +[Relational Query Interface](../../core-spec/relational_semantics.md). All +cases use the same SCOTT data and Ossie model. + +The key words MUST, MUST NOT, SHOULD, SHOULD NOT, and MAY are to be +interpreted as described in BCP 14 +([RFC 2119](https://www.rfc-editor.org/rfc/rfc2119), +[RFC 8174](https://www.rfc-editor.org/rfc/rfc8174)) when they appear in +uppercase. + +Layout +------ + +| Path | Contents | +|------|----------| +| [`scott/scott.sql`](scott/scott.sql) | SCOTT schema and data | +| [`scott/scott_model.ossie.yaml`](scott/scott_model.ossie.yaml) | Ossie model with dataset-scoped metrics | +| `cases//query.ossie.sql` | Ossie query under test | +| `cases//expected.sql` | Expected result, as standard SQL over literals | +| `cases//reference.duckdb.sql` | Non-normative DuckDB query that derives the expected result | + +The file name suffix gives the SQL dialect: `.ossie.sql` for Ossie SQL, +`.duckdb.sql` for DuckDB SQL, and `.sql` for standard SQL (ISO/IEC 9075). + +A case passes when an implementation loads the data and the model, runs +`query.ossie.sql`, and returns a result equal to the result of `expected.sql` +(see "Comparing results"). For an example, see +[`cases/a01_grand_total/`](cases/a01_grand_total/). + +Case files +---------- + +`query.ossie.sql` starts with a comment header: the case ID and title, the +behavior under test, and the spec sections (for example, `Spec: 6, 7.2`). The +file can be submitted to an implementation as it is. + +`expected.sql` is one query: + +```sql +SELECT * +FROM (VALUES (29025.00, 14)) AS expected (sum_sal, emp_count); +``` + +- Column names and order MUST match `query.ossie.sql`. +- Values MUST be standard SQL literals. An exact result MUST be an exact + numeric literal (`29025.00`). An approximate result MUST be an approximate + numeric literal (`2.073214285714286E3`). +- A column that is NULL in every row MUST give its type in the first row, with + `CAST(NULL AS )`. +- An empty result MUST use one typed row and `WHERE 1 = 0`. +- If `query.ossie.sql` has `ORDER BY`, the rows MUST be in that order. + +`reference.duckdb.sql` is non-normative. If it disagrees with the spec, the +spec is correct, and the case MUST be fixed. It uses the column names of +`query.ossie.sql`. For one relation, it replaces each `AGG(m)` with the +expression of `m`, and each model relation with its source table. For joins, +it follows the reference evaluation strategy of Section 9: it removes join +duplicates by `primary_key`, and then it aggregates. + +Result semantics +---------------- + +Expected results use standard SQL (ISO/IEC 9075) semantics. For these tests, +the host dialect of Section 8.1 is standard SQL. For example, `SUM` over no +rows is NULL, and `COUNT` over no rows is 0. An implementation with other +defaults MUST run the tests with standard SQL semantics. + +A case MUST NOT return an exact numeric whose scale the standard leaves +implementation-defined, such as `AVG` over an exact numeric or the division of +two exact numerics. The tolerance does not cover these results: for `AVG` over +`DECIMAL(7, 2)`, an implementation can return a scale of 2, which differs from +the true value by more than 1e-9. To use `AVG`, cast its argument to an +approximate type, for example `AVG(CAST(sal AS DOUBLE PRECISION))`. A case +MUST NOT return NaN or an infinity. + +Comparing results +----------------- + +- Columns are matched by position. The column counts MUST be equal. Column + names are compared case-insensitively. +- Rows MUST match as multisets, including the number of duplicates. +- If `query.ossie.sql` has `ORDER BY`, the row order MUST also match, and the + sort key MUST order the result uniquely. +- Two values match if both are NULL, or if neither is NULL and they are equal. + Types are not compared: `29025.00` matches `29025`. +- Exact numerics MUST be equal. +- An approximate numeric `a` matches an expected value `e` if + `|a - e| <= 1e-9 * max(|a|, |e|, 1)`. This is a relative tolerance of 1e-9, + with an absolute tolerance of 1e-9 near zero. + +The rule for approximate numerics is provisional. How to compare +floating-point results across engines is an open question: the size of the +tolerance, the tolerance near zero, and how to match rows as multisets when +values match only within the tolerance. Until it is settled, cases SHOULD use +exact results. + +Running the tests +----------------- + +DuckDB needs no extensions. To run all reference queries: + +```sh +cat compliance/query/scott/scott.sql \ + compliance/query/cases/*/reference.duckdb.sql | duckdb +``` + +To show the result of a reference query above the result of its +`expected.sql`: + +```sh +CASE=compliance/query/cases/a01_grand_total +cat compliance/query/scott/scott.sql \ + "$CASE/reference.duckdb.sql" "$CASE/expected.sql" | duckdb +``` + +To validate the model: + +```sh +uv run validation/validate.py compliance/query/scott/scott_model.ossie.yaml +``` + +Adding a case +------------- + +- Add a directory under `cases/` with the three files. Name it with the + lowercase case ID, with a two-digit number, and a short name, for example + `a02_group_by`. Use the ID prefix `A` for one relation and `B` for joins. +- Do not change `scott/scott.sql`. Add any new metric to + `scott/scott_model.ossie.yaml` in the same pull request, and validate the + model. +- Copy `expected.sql` from the output of `reference.duckdb.sql`. Do not + calculate it by hand. +- Cases SHOULD NOT depend on other implementation-defined behavior. Document + any such dependency in the header of `query.ossie.sql`. + +Future work +----------- + +The cases can move to the `.ost` test format +([#501](https://github.com/apache/ossie/issues/501)) and use the view expander +([#502](https://github.com/apache/ossie/issues/502)) to run Ossie queries +directly. diff --git a/compliance/query/cases/a01_grand_total/expected.sql b/compliance/query/cases/a01_grand_total/expected.sql new file mode 100644 index 00000000..6b0c1c0e --- /dev/null +++ b/compliance/query/cases/a01_grand_total/expected.sql @@ -0,0 +1,19 @@ +-- Licensed to the Apache Software Foundation (ASF) under one +-- or more contributor license agreements. See the NOTICE file +-- distributed with this work for additional information +-- regarding copyright ownership. The ASF licenses this file +-- to you under the Apache License, Version 2.0 (the +-- "License"); you may not use this file except in compliance +-- with the License. You may obtain a copy of the License at +-- +-- http://www.apache.org/licenses/LICENSE-2.0 +-- +-- Unless required by applicable law or agreed to in writing, +-- software distributed under the License is distributed on an +-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +-- KIND, either express or implied. See the License for the +-- specific language governing permissions and limitations +-- under the License. +-- Expected result for case A1 (query.ossie.sql), in standard SQL. +SELECT * +FROM (VALUES (29025.00, 14)) AS expected (sum_sal, emp_count); diff --git a/compliance/query/cases/a01_grand_total/query.ossie.sql b/compliance/query/cases/a01_grand_total/query.ossie.sql new file mode 100644 index 00000000..d2a7fcf4 --- /dev/null +++ b/compliance/query/cases/a01_grand_total/query.ossie.sql @@ -0,0 +1,24 @@ +-- Licensed to the Apache Software Foundation (ASF) under one +-- or more contributor license agreements. See the NOTICE file +-- distributed with this work for additional information +-- regarding copyright ownership. The ASF licenses this file +-- to you under the Apache License, Version 2.0 (the +-- "License"); you may not use this file except in compliance +-- with the License. You may obtain a copy of the License at +-- +-- http://www.apache.org/licenses/LICENSE-2.0 +-- +-- Unless required by applicable law or agreed to in writing, +-- software distributed under the License is distributed on an +-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +-- KIND, either express or implied. See the License for the +-- specific language governing permissions and limitations +-- under the License. +-- ---------------------------------------------------------------------------- +-- A1. Grand total. +-- AGG() is an aggregate. Without GROUP BY, the query returns one row. +-- Spec: 6, 7.2 +-- ---------------------------------------------------------------------------- +SELECT AGG(sum_sal) AS sum_sal, + AGG(emp_count) AS emp_count +FROM scott_model.emp; diff --git a/compliance/query/cases/a01_grand_total/reference.duckdb.sql b/compliance/query/cases/a01_grand_total/reference.duckdb.sql new file mode 100644 index 00000000..a8ef0d6c --- /dev/null +++ b/compliance/query/cases/a01_grand_total/reference.duckdb.sql @@ -0,0 +1,21 @@ +-- Licensed to the Apache Software Foundation (ASF) under one +-- or more contributor license agreements. See the NOTICE file +-- distributed with this work for additional information +-- regarding copyright ownership. The ASF licenses this file +-- to you under the Apache License, Version 2.0 (the +-- "License"); you may not use this file except in compliance +-- with the License. You may obtain a copy of the License at +-- +-- http://www.apache.org/licenses/LICENSE-2.0 +-- +-- Unless required by applicable law or agreed to in writing, +-- software distributed under the License is distributed on an +-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +-- KIND, either express or implied. See the License for the +-- specific language governing permissions and limitations +-- under the License. +-- Reference query for case A1 (query.ossie.sql), over the base tables in +-- ../../scott/scott.sql. Its result is equal to the result of expected.sql. +SELECT SUM(sal) AS sum_sal, + COUNT(*) AS emp_count +FROM scott.emp; diff --git a/compliance/query/scott/scott.sql b/compliance/query/scott/scott.sql new file mode 100644 index 00000000..b0358ea2 --- /dev/null +++ b/compliance/query/scott/scott.sql @@ -0,0 +1,82 @@ +-- Licensed to the Apache Software Foundation (ASF) under one +-- or more contributor license agreements. See the NOTICE file +-- distributed with this work for additional information +-- regarding copyright ownership. The ASF licenses this file +-- to you under the Apache License, Version 2.0 (the +-- "License"); you may not use this file except in compliance +-- with the License. You may obtain a copy of the License at +-- +-- http://www.apache.org/licenses/LICENSE-2.0 +-- +-- Unless required by applicable law or agreed to in writing, +-- software distributed under the License is distributed on an +-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +-- KIND, either express or implied. See the License for the +-- specific language governing permissions and limitations +-- under the License. + +-- ============================================================================ +-- Data: the SCOTT schema +-- +-- Shared input data for the query compliance cases in ../cases/. Load this +-- file before a case runs. Do not change the values: the expected results of +-- all cases depend on them. +-- +-- Values are the same as the SCOTT tables in Apache Calcite +-- (core/src/test/resources/sql/within-distinct.iq). +-- ============================================================================ + +DROP SCHEMA IF EXISTS scott CASCADE; +CREATE SCHEMA scott; + +CREATE TABLE scott.dept ( + deptno INTEGER NOT NULL PRIMARY KEY, + dname VARCHAR NOT NULL, + loc VARCHAR NOT NULL +); + +INSERT INTO scott.dept VALUES + (10, 'ACCOUNTING', 'NEW YORK'), + (20, 'RESEARCH', 'DALLAS'), + (30, 'SALES', 'CHICAGO'), + (40, 'OPERATIONS', 'BOSTON'); + +CREATE TABLE scott.emp ( + empno INTEGER NOT NULL PRIMARY KEY, + ename VARCHAR NOT NULL, + job VARCHAR NOT NULL, + mgr INTEGER, + hiredate DATE NOT NULL, + sal DECIMAL(7, 2) NOT NULL, + comm DECIMAL(7, 2), + deptno INTEGER NOT NULL +); + +INSERT INTO scott.emp VALUES + (7369, 'SMITH', 'CLERK', 7902, DATE '1980-12-17', 800.00, NULL, 20), + (7499, 'ALLEN', 'SALESMAN', 7698, DATE '1981-02-20', 1600.00, 300.00, 30), + (7521, 'WARD', 'SALESMAN', 7698, DATE '1981-02-22', 1250.00, 500.00, 30), + (7566, 'JONES', 'MANAGER', 7839, DATE '1981-02-04', 2975.00, NULL, 20), + (7654, 'MARTIN', 'SALESMAN', 7698, DATE '1981-09-28', 1250.00, 1400.00, 30), + (7698, 'BLAKE', 'MANAGER', 7839, DATE '1981-01-05', 2850.00, NULL, 30), + (7782, 'CLARK', 'MANAGER', 7839, DATE '1981-06-09', 2450.00, NULL, 10), + (7788, 'SCOTT', 'ANALYST', 7566, DATE '1987-04-19', 3000.00, NULL, 20), + (7839, 'KING', 'PRESIDENT', NULL, DATE '1981-11-17', 5000.00, NULL, 10), + (7844, 'TURNER', 'SALESMAN', 7698, DATE '1981-09-08', 1500.00, 0.00, 30), + (7876, 'ADAMS', 'CLERK', 7788, DATE '1987-05-23', 1100.00, NULL, 20), + (7900, 'JAMES', 'CLERK', 7698, DATE '1981-12-03', 950.00, NULL, 30), + (7902, 'FORD', 'ANALYST', 7566, DATE '1981-12-03', 3000.00, NULL, 20), + (7934, 'MILLER', 'CLERK', 7782, DATE '1982-01-23', 1300.00, NULL, 10); + +CREATE TABLE scott.salgrade ( + grade INTEGER NOT NULL PRIMARY KEY, + losal INTEGER NOT NULL, + hisal INTEGER NOT NULL +); + +INSERT INTO scott.salgrade VALUES + (1, 700, 1200), + (2, 1201, 1400), + (3, 1401, 2000), + (4, 2001, 3000), + (5, 3001, 9999); diff --git a/compliance/query/scott/scott_model.ossie.yaml b/compliance/query/scott/scott_model.ossie.yaml new file mode 100644 index 00000000..442f0ab4 --- /dev/null +++ b/compliance/query/scott/scott_model.ossie.yaml @@ -0,0 +1,140 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +# yaml-language-server: $schema=../../../core-spec/ossie-schema.json +# +# Ossie semantic model over the SCOTT schema (scott.sql), for the query +# compliance cases in ../cases/. +# +# The metrics are dataset-scoped (datasets[].metrics). Each dataset is exposed +# as a relation with the same name, and each metric is a measure column of that +# relation (relational_semantics.md, Section 3.4). All relations of the model +# are in one schema with the name of the model (Section 11.2). The cases refer +# to these relations by two-part name: scott_model.emp, scott_model.dept, and +# scott_model.salgrade. +# +# The primary_key of each dataset is the row identity of its measures +# (Section 3.2). +# +# The model has no AVG metric yet. For AVG over an exact numeric, the SQL +# standard leaves the scale of the result implementation-defined, so a case +# that needs AVG adds a metric over an approximate argument, for example +# AVG(CAST(sal AS DOUBLE PRECISION)) (../README.md, "Result semantics"). +# +# The emp-to-salgrade join (e.sal BETWEEN s.losal AND s.hisal) is a range join. +# An Ossie relationship cannot express it, so a case that uses it writes the +# join in the query (Section 11.3). + +version: 0.2.0.dev0 +name: scott_model +description: The SCOTT schema with dataset-scoped metrics +datasets: + - name: emp + source: scott.emp + primary_key: [empno] + fields: + - name: empno + datatype: Integer + expression: {dialects: [{dialect: OSSIE_SQL_2026, expression: empno}]} + - name: ename + datatype: String + expression: {dialects: [{dialect: OSSIE_SQL_2026, expression: ename}]} + - name: job + datatype: String + expression: {dialects: [{dialect: OSSIE_SQL_2026, expression: job}]} + - name: mgr + datatype: Integer + expression: {dialects: [{dialect: OSSIE_SQL_2026, expression: mgr}]} + - name: hiredate + datatype: Date + expression: {dialects: [{dialect: OSSIE_SQL_2026, expression: hiredate}]} + - name: sal + datatype: Decimal + expression: {dialects: [{dialect: OSSIE_SQL_2026, expression: sal}]} + - name: comm + datatype: Decimal + expression: {dialects: [{dialect: OSSIE_SQL_2026, expression: comm}]} + - name: deptno + datatype: Integer + expression: {dialects: [{dialect: OSSIE_SQL_2026, expression: deptno}]} + metrics: + - name: sum_sal + description: Total salary + datatype: Decimal + expression: {dialects: [{dialect: OSSIE_SQL_2026, expression: SUM(sal)}]} + - name: emp_count + description: Number of employees + datatype: Integer + expression: {dialects: [{dialect: OSSIE_SQL_2026, expression: COUNT(*)}]} + - name: comm_count + description: Number of employees that have a commission + datatype: Integer + expression: {dialects: [{dialect: OSSIE_SQL_2026, expression: COUNT(comm)}]} + - name: sum_comm + description: Total commission + datatype: Decimal + expression: {dialects: [{dialect: OSSIE_SQL_2026, expression: SUM(comm)}]} + + - name: dept + source: scott.dept + primary_key: [deptno] + fields: + - name: deptno + datatype: Integer + expression: {dialects: [{dialect: OSSIE_SQL_2026, expression: deptno}]} + - name: dname + datatype: String + expression: {dialects: [{dialect: OSSIE_SQL_2026, expression: dname}]} + - name: loc + datatype: String + expression: {dialects: [{dialect: OSSIE_SQL_2026, expression: loc}]} + metrics: + - name: dept_count + description: Number of departments + datatype: Integer + expression: {dialects: [{dialect: OSSIE_SQL_2026, expression: COUNT(*)}]} + + - name: salgrade + source: scott.salgrade + primary_key: [grade] + fields: + - name: grade + datatype: Integer + expression: {dialects: [{dialect: OSSIE_SQL_2026, expression: grade}]} + - name: losal + datatype: Integer + expression: {dialects: [{dialect: OSSIE_SQL_2026, expression: losal}]} + - name: hisal + datatype: Integer + expression: {dialects: [{dialect: OSSIE_SQL_2026, expression: hisal}]} + metrics: + - name: grade_count + description: Number of salary grades + datatype: Integer + expression: {dialects: [{dialect: OSSIE_SQL_2026, expression: COUNT(*)}]} + +relationships: + - name: emp_to_dept + from: emp + to: dept + from_columns: [deptno] + to_columns: [deptno] + - name: emp_to_manager + from: emp + to: emp + from_columns: [mgr] + to_columns: [empno]