Fix load_tabular_data.sh depending on undefined behavior (which has started breaking)

[dmannarino]

The symptom:

test_table_source_asset_minimal (and any tabular data load using load_tabular_data.sh) intermittently failed with the batch job reporting status: failed. The job logs showed BEGIN, CREATE TABLE, and both ALTER TABLE statements succeeding, then immediately unexpected COPY_IN result, aborting connection from psql, followed by download failed: ... [Errno 32] Broken pipe from the aws s3 cp side of the pipe — with no data ever actually loaded.

The issue:

load_tabular_data.sh piped aws s3 cp ... - into psql -c "BEGIN; ...; COPY \"$TEMP_TABLE\" FROM STDIN ...; ...; INSERT ...; COMMIT;" — a single -c string bundling multiple statements with COPY FROM STDIN in the middle rather than last. This is a documented limitation of psql -c: bundling more than one command, with a COPY not in the final position, produces "unexpected results" (per PostgreSQL's own maintainers), and here it manifested as psql aborting the connection the instant it entered copy mode, before reading any data. psql closing its end of the pipe first is also what produced the "Broken pipe" on the aws s3 cp side — a downstream symptom, not the root cause. Since create_tabular_schema.sh has no COPY statement, it was unaffected, which is why only the load step failed.

The fix:

Restructure the script to feed psql one continuous script over its real stdin instead of via -c: header SQL ending in the COPY statement, then the streamed S3 data, then an explicit \. end-of-copy marker, then the trailer SQL (INSERT ... COMMIT) — the same pattern as psql -f <(cat header.sql data.csv footer.sql), kept streaming rather than buffered. The data is piped through sed -e '$a\' first so it always ends in exactly one newline (verified against both a normally-terminated and a missing-trailing-newline fixture), which keeps the \. marker on its own line without ever risking a spurious blank row that would break COPY on a multi-column table. Also added set -o pipefail, since without it a failed aws s3 cp could be silently masked by a later stage in the same pipe succeeding, and psql -v ON_ERROR_STOP=1 to preserve fail-fast behavior on SQL errors.

Why now?:

The code has been (at least mostly) working for a few years but just started CONSISTENTLY breaking now. I would guess, but am not sure, that Ubuntu upgraded the PostgreSQL package to one which changed the undefined behavior in practice. Actually, Claude says that Ubuntu Noble did just go from PostgreSQL 15 to PostgreSQL 16.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 76.18%. Comparing base (30a2a39) to head (2518652).

Additional details and impacted files

@@           Coverage Diff           @@
##           master     #770   +/-   ##
=======================================
  Coverage   76.17%   76.18%           
=======================================
  Files         144      144           
  Lines        6834     6835    +1     
=======================================
+ Hits         5206     5207    +1     
  Misses       1628     1628           

Flag	Coverage Δ
unittests	76.18% <100.00%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.