Skip to content

Commit 40f5d7e

Browse files
kyleconroyclaude
andauthored
googlesql: Spanner schemas, arrays, and a function seed from Spanner (#4645)
* googlesql: convert ALTER TABLE, views, renames and HIDDEN columns ALTER TABLE was dropped as unsupported, so columns it added, dropped or retyped never reached the catalog. It now converts ADD/DROP COLUMN, Spanner's ALTER COLUMN (which restates the type and nullability), ALTER COLUMN ... DROP NOT NULL, RENAME COLUMN and RENAME TO; RENAME TABLE converts too. CREATE [OR REPLACE] VIEW and DROP VIEW convert, and a HIDDEN column is left out of SELECT * as Spanner does. The new googlesql cases' schemas were checked against Spanner Omni. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L8QoGi6VRtADkb4jL3BFHP * googlesql: type arrays, UNNEST, subscripts and typed literals The converter now handles typed literals (DATE, TIMESTAMP, NUMERIC, JSON, bytes), array constructors, INTERVAL, the OFFSET/ORDINAL/SAFE_* subscripts and "x IN UNNEST(array)", which becomes "x = ANY(array)". In the analysis core: - A subscript is the array's element, nullable, with an integer index. - ARRAY(subquery) is an array of the column, not its first value. - "x = ANY(array)" types a placeholder as an array of x's type, as PostgreSQL does; sqlc.slice() still gets the element. - A polymorphic function taking anyarray and returning anyelement, as unnest does, returns the array's element, and a function in FROM takes that resolved type, with its alias naming the column. GoogleSQL gains an unnest seed. analyze_arrays is checked against Spanner Omni; the rest are in array_unnest, where Spanner's wire types lose the nullability and lengths the golden pins. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L8QoGi6VRtADkb4jL3BFHP * googlesql: seed Spanner's functions and operators functions.jsonl held only count, so every other function typed as any. It now lists 291 signatures across aggregates, math, strings, dates and timestamps, JSON, arrays, hashing, search and vector distance. Their return types, and those of the operators, were read from Spanner Omni by compiling each call in PLAN mode; the search functions and PENDING_COMMIT_TIMESTAMP, which cannot be compiled in a select list, are typed from Spanner's reference. Spanner has no window functions, so none are listed. GoogleSQL now propagates NULL through function arguments, with the functions that return NULL from non-null input marked nullable. The converter maps COALESCE, IFNULL, NULLIF, IF, GREATEST and LEAST onto the nodes the core types them by, makes SAFE.f(...) nullable, passes date parts (DAY, WEEK(MONDAY)) as strings rather than columns, and converts EXTRACT. In the core, an operator a dialect lists now replaces the one its arithmetic rule generates for the same operands, which is how INT64 / INT64 is a FLOAT64; and a placeholder passed for a polymorphic parameter takes its type from the other arguments, so the value in array_includes(array, @v) is the array's element. analyze_functions is checked against Spanner Omni; func_nullability covers what Spanner's wire types cannot report. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L8QoGi6VRtADkb4jL3BFHP * goldeneye: generate Spanner's types, functions and operators Spanner keeps no catalog of its library, so goldeneye now reads it by asking. spanner/library.jsonl lists the function signatures to try; each is compiled in PLAN mode against a probe database, and the ones Spanner accepts are written to functions.jsonl with the type it reports. A polymorphic signature is tried with INT64 and STRING and its result must hold for both. Calls a select list cannot make are compiled the way they can be: SEARCH in a WHERE clause, SCORE over a search index, PENDING_COMMIT_TIMESTAMP in an INSERT. operators.jsonl is every binary operator Spanner accepts over the operands tried, and types.jsonl the types the dialect lists, each one Spanner has checked by a cast; the GoogleSQL types only BigQuery has are kept as they are. The generated files hold the same records the seed already did; they are rewritten in goldeneye's compact form, and TestDialect now checks all four files against Spanner. The dialect package's Function gains never_null, which the seed already reads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L8QoGi6VRtADkb4jL3BFHP --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent c69308b commit 40f5d7e

79 files changed

Lines changed: 3570 additions & 57 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎internal/compiler/parse_core.go‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -201,5 +201,13 @@ func coreParamColumn(p core.Parameter, params *named.ParamSet) *Column {
201201
col.IsSqlcSlice = param.IsSqlcSlice()
202202
col.IsNamedParam = true
203203
}
204+
// sqlc.slice() already says the argument is a list of values, so a
205+
// slice the analyzer typed as an array, as it does "= ANY(sqlc.slice(x))",
206+
// is a slice of the array's elements.
207+
if col.IsSqlcSlice && col.TypeExpr.IsArray() {
208+
col.TypeExpr = col.TypeExpr.Element()
209+
col.IsArray = false
210+
describeType(col, col.TypeExpr)
211+
}
204212
return col
205213
}

‎internal/core/analyzer/expr.go‎

Lines changed: 127 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -79,6 +79,9 @@ func (a *analyzer) typeExpr(n ast.Node) (exprType, error) {
7979
case *ast.A_ArrayExpr:
8080
return a.typeArrayExpr(e)
8181

82+
case *ast.A_Indirection:
83+
return a.typeIndirection(e)
84+
8285
case *ast.SubLink:
8386
return a.typeSubLink(e)
8487

@@ -366,7 +369,17 @@ func (a *analyzer) typeQuantifiedExpr(e *ast.A_Expr) (exprType, error) {
366369
if err != nil {
367370
return exprType{}, err
368371
}
369-
if err := a.typeOperands(e.Rexpr, leftT); err != nil {
372+
// "x = ANY(SELECT ...)" compares x against the subquery's column, but
373+
// "x = ANY(array)" against the elements of an array, so a placeholder
374+
// there is an array of x's type.
375+
right := leftT
376+
if _, sub := e.Rexpr.(*ast.SubLink); !sub {
377+
right = a.arrayOf(leftT)
378+
right.sourceClassOID = leftT.sourceClassOID
379+
right.sourceAttributeOID = leftT.sourceAttributeOID
380+
right.sourceTableAlias = leftT.sourceTableAlias
381+
}
382+
if err := a.typeOperands(e.Rexpr, right); err != nil {
370383
return exprType{}, err
371384
}
372385
return a.boolType(false)
@@ -524,6 +537,51 @@ func (a *analyzer) typeArrayExpr(e *ast.A_ArrayExpr) (exprType, error) {
524537
return a.lookupType(core.Array(element.WithNullable(false))), nil
525538
}
526539

540+
// typeIndirection types an array subscript, arr[i], as the array's element,
541+
// and a slice, arr[i:j], as the array itself. An element may be NULL, and a
542+
// subscript past the end of the array yields NULL in the dialects that do not
543+
// raise an error for it. The subscripts are integers.
544+
func (a *analyzer) typeIndirection(e *ast.A_Indirection) (exprType, error) {
545+
t, err := a.typeExpr(e.Arg)
546+
if err != nil {
547+
return exprType{}, err
548+
}
549+
intT := exprType{}
550+
if oid, err := a.cat.ConstTypeOID(core.ConstInteger); err == nil {
551+
intT.typeOID = oid
552+
}
553+
for _, item := range listItems(e.Indirection) {
554+
idx, ok := item.(*ast.A_Indices)
555+
if !ok {
556+
return exprType{}, fmt.Errorf("typeExpr: unsupported indirection %T", item)
557+
}
558+
for _, bound := range []ast.Node{idx.Lidx, idx.Uidx} {
559+
if err := a.typeOperands(bound, intT); err != nil {
560+
return exprType{}, err
561+
}
562+
}
563+
if idx.IsSlice {
564+
continue
565+
}
566+
element := a.exprOf(t).Element()
567+
if element == nil {
568+
return exprType{nullable: true}, nil
569+
}
570+
t = a.lookupType(element)
571+
t.nullable = true
572+
}
573+
return t, nil
574+
}
575+
576+
// arrayOf is the type of an array of t's type.
577+
func (a *analyzer) arrayOf(t exprType) exprType {
578+
element := a.exprOf(t)
579+
if element == nil {
580+
return exprType{}
581+
}
582+
return a.lookupType(core.Array(element.WithNullable(false)))
583+
}
584+
527585
// lookupType is the type an expression refers to: its row when the catalog
528586
// holds one, its family's row when it holds only that, and the expression
529587
// alone when it holds neither.
@@ -619,14 +677,21 @@ func (a *analyzer) typeSubLink(e *ast.SubLink) (exprType, error) {
619677
return exprType{}, err
620678
}
621679
switch e.SubLinkType {
622-
case ast.EXPR_SUBLINK, ast.ARRAY_SUBLINK:
680+
case ast.EXPR_SUBLINK:
623681
if len(cols) == 0 {
624682
return exprType{nullable: true}, nil
625683
}
626684
// A subquery that matches no row yields NULL.
627685
t := columnExprType(cols[0])
628686
t.nullable = true
629687
return t, nil
688+
case ast.ARRAY_SUBLINK:
689+
// ARRAY(SELECT ...) collects the column into an array, which is
690+
// empty rather than NULL when no row matches.
691+
if len(cols) == 0 {
692+
return exprType{}, nil
693+
}
694+
return a.arrayOf(columnExprType(cols[0])), nil
630695
default:
631696
return a.boolType(false)
632697
}
@@ -831,12 +896,20 @@ func (a *analyzer) typeFuncCall(f *ast.FuncCall) (exprType, error) {
831896
}
832897
p := a.pickOverload(overloads, argOIDs)
833898
// An argument that is a bare placeholder takes the parameter's type,
834-
// unless the parameter is polymorphic and says nothing.
899+
// unless the parameter is polymorphic, when it takes what the other
900+
// arguments say: the element of the array passed for an anyarray, or
901+
// an array of the value passed for an anyelement, as
902+
// array_includes(anyarray, anyelement) relates them.
835903
for i, arg := range args {
836904
if i >= len(p.ArgTypes) {
837905
break
838906
}
839907
if a.isPolymorphicOID(p.ArgTypes[i]) {
908+
if t, ok := a.polymorphicArg(p, i, argTypes); ok {
909+
if err := a.typeOperands(arg, t); err != nil {
910+
return exprType{}, err
911+
}
912+
}
840913
continue
841914
}
842915
if err := a.typeOperands(arg, exprType{typeOID: p.ArgTypes[i]}); err != nil {
@@ -916,11 +989,61 @@ func (a *analyzer) returnType(p core.ProcOverload, argTypes []exprType) exprType
916989
return exprType{}
917990
}
918991
if isPolymorphic(name) && (argTypes[0].typeOID != 0 || argTypes[0].expr != nil) {
919-
return exprType{typeOID: argTypes[0].typeOID, expr: argTypes[0].expr}
992+
arg := exprType{typeOID: argTypes[0].typeOID, expr: argTypes[0].expr}
993+
// An element taken from an array argument, unnest(anyarray)
994+
// returning anyelement, is the array's element; an array built from
995+
// an element argument is an array of it.
996+
paramName := ""
997+
if len(p.ArgTypes) > 0 {
998+
paramName, _ = a.cat.TypeName(p.ArgTypes[0])
999+
}
1000+
switch {
1001+
case isPolymorphicArray(paramName) && !isPolymorphicArray(name):
1002+
if element := a.exprOf(arg).Element(); element != nil {
1003+
return a.lookupType(element)
1004+
}
1005+
case !isPolymorphicArray(paramName) && isPolymorphicArray(name) && isPolymorphic(paramName):
1006+
return a.arrayOf(arg)
1007+
}
1008+
return arg
9201009
}
9211010
return exprType{typeOID: p.ReturnTypeOID}
9221011
}
9231012

1013+
// polymorphicArg is the type the other arguments of a call give its i-th,
1014+
// polymorphic, parameter: the element of an array passed for an anyarray
1015+
// when the parameter is an element, or an array of an element passed for
1016+
// an anyelement when the parameter is an array.
1017+
func (a *analyzer) polymorphicArg(p core.ProcOverload, i int, argTypes []exprType) (exprType, bool) {
1018+
name, err := a.cat.TypeName(p.ArgTypes[i])
1019+
if err != nil || !isPolymorphic(name) {
1020+
return exprType{}, false
1021+
}
1022+
for j, other := range p.ArgTypes {
1023+
if j == i || j >= len(argTypes) || (argTypes[j].typeOID == 0 && argTypes[j].expr == nil) {
1024+
continue
1025+
}
1026+
otherName, err := a.cat.TypeName(other)
1027+
if err != nil || !isPolymorphic(otherName) {
1028+
continue
1029+
}
1030+
switch {
1031+
case isPolymorphicArray(otherName) && !isPolymorphicArray(name):
1032+
if element := a.exprOf(argTypes[j]).Element(); element != nil {
1033+
return a.lookupType(element), true
1034+
}
1035+
case !isPolymorphicArray(otherName) && isPolymorphicArray(name):
1036+
return a.arrayOf(argTypes[j]), true
1037+
}
1038+
}
1039+
return exprType{}, false
1040+
}
1041+
1042+
// isPolymorphicArray reports whether a polymorphic type stands for an array.
1043+
func isPolymorphicArray(typeName string) bool {
1044+
return typeName == "anyarray" || typeName == "anycompatiblearray"
1045+
}
1046+
9241047
// argIndex reads a seed's "$n" pseudo-type as the zero-based index of the
9251048
// argument whose type it stands for.
9261049
func argIndex(typeName string) (int, bool) {

‎internal/core/analyzer/scope.go‎

Lines changed: 16 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -100,19 +100,28 @@ func (a *analyzer) bindRangeFunction(rf *ast.RangeFunction) (scopeRel, error) {
100100
}
101101

102102
// The arguments are typed against the scope built so far, which is what
103-
// gives a placeholder passed to the function its type.
104-
if _, err := a.typeFuncCall(call); err != nil {
103+
// gives a placeholder passed to the function its type, and what
104+
// resolves a polymorphic result: unnest of an array of strings yields
105+
// strings.
106+
t, err := a.typeFuncCall(call)
107+
if err != nil {
105108
return scopeRel{}, err
106109
}
107110

108-
p := overloads[0]
109-
rel := scopeRel{
110-
alias: name,
111-
cols: []core.ClassColumn{{Name: name, TypeOID: p.ReturnTypeOID, NotNull: !p.ReturnNullable}},
112-
}
111+
// The relation's alias names its single column too, as "unnest(tags)
112+
// AS tag" does.
113+
colName := name
114+
rel := scopeRel{alias: name}
113115
if rf.Alias != nil && rf.Alias.Aliasname != nil && *rf.Alias.Aliasname != "" {
114116
rel.alias = *rf.Alias.Aliasname
117+
colName = rel.alias
118+
}
119+
col := core.ClassColumn{Name: colName, TypeOID: t.typeOID, Type: t.expr, NotNull: !t.nullable}
120+
if t.typeOID == 0 && t.expr == nil {
121+
p := overloads[0]
122+
col.TypeOID, col.NotNull = p.ReturnTypeOID, !p.ReturnNullable
115123
}
124+
rel.cols = []core.ClassColumn{col}
116125
return rel, nil
117126
}
118127

‎internal/core/seed/seed.go‎

Lines changed: 22 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -320,12 +320,24 @@ func apply(cat *core.Catalog, fsys fs.FS, settings Settings) error {
320320
if err := b.rules(); err != nil {
321321
return err
322322
}
323-
if err := b.categoryOperators(); err != nil {
323+
// An operator the dialect lists replaces the one its rules would
324+
// generate for the same operands, as GoogleSQL's INT64 / INT64 is a
325+
// FLOAT64, so the list is read before the rules apply.
326+
var operators []Operator
327+
if err := stream(fsys, OperatorsFile, func(op Operator) error {
328+
operators = append(operators, op)
329+
return nil
330+
}); err != nil {
324331
return err
325332
}
326-
if err := stream(fsys, OperatorsFile, b.addOperator); err != nil {
333+
if err := b.categoryOperators(operators); err != nil {
327334
return err
328335
}
336+
for i, op := range operators {
337+
if err := b.addOperator(op); err != nil {
338+
return fmt.Errorf("seed: %s: record %d: %w", OperatorsFile, i+1, err)
339+
}
340+
}
329341
if err := stream(fsys, CastsFile, b.addCast); err != nil {
330342
return err
331343
}
@@ -713,8 +725,12 @@ func (b *builder) rules() error {
713725
return nil
714726
}
715727

716-
func (b *builder) categoryOperators() error {
728+
func (b *builder) categoryOperators(listed []Operator) error {
717729
s := b.settings
730+
overridden := map[[3]string]bool{}
731+
for _, op := range listed {
732+
overridden[[3]string{op.Name, strings.ToLower(op.Left), strings.ToLower(op.Right)}] = true
733+
}
718734
boolOID, ok := b.oids[strings.ToLower(s.Bool)]
719735
if len(s.Comparison) > 0 && !ok {
720736
return fmt.Errorf("seed %s: comparison operators need a bool type", s.Dialect)
@@ -729,6 +745,9 @@ func (b *builder) categoryOperators() error {
729745
}
730746
if strings.Contains(s.ArithmeticCategories, t.category) {
731747
for _, op := range s.Arithmetic {
748+
if overridden[[3]string{op, strings.ToLower(t.name), strings.ToLower(t.name)}] {
749+
continue
750+
}
732751
if err := b.operator(op, t.name, t.name, t.name, b.oids[t.name]); err != nil {
733752
return err
734753
}
Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
{
2+
"command": "analyze",
3+
"args": ["--dialect", "googlesql", "--schema", "schema.sql", "query.sql"],
4+
"contexts": ["base"]
5+
}
Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
-- name: AlbumIDs :one
2+
SELECT singer_id, ARRAY(SELECT album_id FROM albums WHERE albums.singer_id = singers.singer_id) AS album_ids
3+
FROM singers WHERE singer_id = @singer_id;
4+
5+
-- name: Literals :one
6+
SELECT [1, 2, 3] AS ints, ARRAY<STRING>['a'] AS strs, DATE '2024-01-02' AS d,
7+
TIMESTAMP '2024-01-02 03:04:05+00' AS ts, NUMERIC '1.5' AS n, JSON '{"a": 1}' AS j,
8+
b'abc' AS raw, INTERVAL 3 DAY AS iv;
Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
CREATE TABLE singers (
2+
singer_id INT64 NOT NULL,
3+
name STRING(MAX) NOT NULL,
4+
tags ARRAY<STRING(MAX)>,
5+
scores ARRAY<INT64> NOT NULL,
6+
) PRIMARY KEY (singer_id);
7+
8+
CREATE TABLE albums (
9+
singer_id INT64 NOT NULL,
10+
album_id INT64 NOT NULL,
11+
title STRING(MAX),
12+
) PRIMARY KEY (singer_id, album_id),
13+
INTERLEAVE IN PARENT singers ON DELETE CASCADE;

0 commit comments

Comments
 (0)