Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 39 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -1848,6 +1848,45 @@ true until the next version shipped.
`moved s4d1's layout`, `no row was lost from s4d2` -- for the same reason.

No ledger change: `hilbert_cluster` is not one of the two suites the ledger covers.
- The object-storage loops name the case each iteration tests, so six checks stop
sharing three ledger keys (#982, second of eight).

`objstore_module` lost the most records of any suite to key collapsing: two loops of
three iterations each, where the check names did not carry the thing the iteration
varies. Measured against a control run on clean `main`:

clean main 30 records 24 distinct keys 3 colliding 6 lost
this branch 30 records 30 distinct keys 0 colliding 0 lost

The record count is unchanged, so this adds and removes no checks.

**The two loops show both sub-shapes of the same defect.** In the first, the headline
already interpolated the scheme and only the continuation was short:

check "a s3 URL reports an object-storage error, not a missing file"
check_num "and does NOT report it as a missing file" <- identical three times

In the second, *neither* name carried the metacharacter, so three iterations produced
one key for each of **two** checks -- the headline collided as well.

Both are fixed by the rule #984 proposed: the continuation carries the same
discriminator its headline names, and where the headline does not name one either, it
gains it. The discriminators come from the loop variable by parameter expansion
(`${url%%:*}` and stripping the fixed prefix and suffix off the pattern), so they are
already in scope:

a remote glob (*) is handled remotely (an object-storage error, not a local one)
and the * glob is NOT reported as a local filesystem miss
and the https URL is NOT reported as a missing file

The first loop's headline now reads its scheme from a variable rather than a `cut`
subshell, because both names need it. The resulting check name is byte-identical, so
no ledger row moves on account of it.

No ledger change at all: `objstore_module` is not one of the two suites the ledger
covers, so its check names have no rows. That is also why the twenty-four collisions
matter for #432 rather than for the census today -- twenty-one of them are in suites
that become covered only when the 240 are seeded.

## [1.0-alpha3] - 2026-09-02

Expand Down
17 changes: 13 additions & 4 deletions test/objstore_module.sh
Original file line number Diff line number Diff line change
Expand Up @@ -102,10 +102,14 @@ check_num "positive control: it IS defined in the module, so nm really looked" \

# A remote path must report a remote error, from the reader, without a connection.
for url in "s3://bucket/key.parquet" "gs://bucket/key.parquet" "https://host/key.parquet"; do
# The scheme, by parameter expansion rather than a `cut` subshell, because BOTH
# names need it now: the headline already carried it and the continuation did not,
# which is why three iterations produced one ledger key (#982).
scheme="${url%%:*}"
out=$(psql_run "SELECT * FROM pgcolumnar.read_parquet('$url') AS (a int)" 2>&1)
check "a $(cut -d: -f1 <<<"$url") URL reports an object-storage error, not a missing file" \
check "a $scheme URL reports an object-storage error, not a missing file" \
"$([ "$(grep -c 'object storage is not implemented\|is not supported\|requires the object-store module\|requires AWS_\|could not resolve\|could not connect\|objstore_allowed_endpoints' <<<"$out")" -ge 1 ] && echo yes || echo no)" "yes"
check_num "and does NOT report it as a missing file" \
check_num "and the $scheme URL is NOT reported as a missing file" \
"$(grep -c 'No such file or directory' <<<"$out")" "0"
done

Expand All @@ -119,10 +123,15 @@ done
# storage error, and the invariant this arm still guards is that it is NEVER a
# local filesystem miss. `*`, `?` and `[` are all legal in an S3 key.
for pat in "s3://bucket/a*.parquet" "s3://bucket/a?.parquet" "s3://bucket/a[0-9].parquet"; do
# The metacharacter under test, which is what distinguishes the three iterations.
# Here BOTH names collided: unlike the loop above, the headline did not carry it
# either, so three iterations produced one key for each of two checks (#982).
meta="${pat#s3://bucket/a}"
meta="${meta%.parquet}"
out=$(psql_run "SELECT * FROM pgcolumnar.read_parquet('$pat') AS (a int)" 2>&1)
check "a remote glob is handled remotely (an object-storage error, not a local one)" \
check "a remote glob ($meta) is handled remotely (an object-storage error, not a local one)" \
"$([ "$(grep -c 'requires AWS_\|object storage\|objstore_allowed_endpoints\|requires the object-store module\|could not resolve\|could not connect\|is not in pgcolumnar' <<<"$out")" -ge 1 ] && echo yes || echo no)" "yes"
check_num "and it is NOT reported as a local filesystem miss" \
check_num "and the $meta glob is NOT reported as a local filesystem miss" \
"$(grep -c 'no files match pattern\|matched no regular files\|No such file or directory' <<<"$out")" "0"
done

Expand Down
Loading