Repository navigation
test(TC-887): native at-rest harness with RED plaintext-at-rest checks - #324
Merged
Merged
Conversation
S0 of TC-887. scripts/at-rest-harness/harness.sh starts a real tinycloud node on native PostgreSQL and native MinIO (pinned, sha256-verified community release; no containers), all on 127.0.0.1 free ports with per-run secrets in 0600 files and a state file; down stops only the recorded PIDs. tests/at_rest_harness.rs (all #[ignore]d): green checks prove an SDK-signed KV put reaches MinIO and a SQL write reaches database_artifact; RED checks assert raw S3 bytes/keys and database_artifact payloads are sealed and fail today (plaintext, space id in keys, SQLite magic). CI runs green as a gate and RED as continue-on-error.
Contributor
Author
|
REQUEST_CHANGES: see TC-887 (head 7142fb9) |
…rministic transforms Review round on #324 (B1, B2, S1): - KV RED also rejects blake3(value) in object names (b64url padded and unpadded, standard base64, hex; raw digest and multihash). - Every check writes the same plaintext into two fresh spaces; RED requires the stored bytes to differ and share no 32-byte run, and a 64 KiB repeated-marker value (and every SQL payload/delta) to stay incompressible (zlib ratio > 0.95). Catches base64/XOR/zlib and any other deterministic transform that hides the marker. - SQL precondition requires a non-NULL delta_payload; RED covers the WAL magic and markers in delta_payload. - Offline selftest proves the predicates flag base64/XOR/zlib and pass sealed-like random blobs. flate2 is a dev-dependency already in the lock (via ssi-status).
…-RED gate Review round on #324 (B3, S1-S4, nits): - A failed, interrupted or timed-out up stops every process group it started (EXIT/INT/TERM trap; die triggers it) and keeps the logs. - PIDs are recorded with their kernel start time; down signals whole process groups and only if the leader is still the recorded process. - down/test/status/logs require an explicit STATE; the shared harness-latest link is gone. - Node runs with storage.sql.memory_threshold=0 so the WAL-delta path runs. - test --expect green|red: exit 0 only on the expected outcome, with distinct UNEXPECTEDLY PASSED (3) and ERRORED (4) results; CI runs RED as --expect red instead of continue-on-error. - TC887_AT_REST_MODE / TC887_AT_REST_ESCROW map to the S1-owned TINYCLOUD_AT_REST__MODE / __ESCROW in one marked block; with encrypt/strict, test red expects green. - Nits: temp file removed on failed download, non-linux-amd64 refused, CI job permissions contents:read, persist-credentials false, scratch root under RUNNER_TEMP.
Contributor
Author
|
APPROVE: see TC-887 (head |
This was referenced Oct 17, 2026
samgbafa
added a commit
that referenced
this pull request
Oct 17, 2026
… harness follow-ups (#331) * ci(TC-887): allow manual dispatch of the Rust workflow for stacked branches * docs(TC-887): put the at-rest interfaces in the repo; one generation ceiling docs/at-rest/INTERFACES.md carries the normative TCE1 format, the TCE2 versioning rule, the fail-closed and header-trust notes and the KV storage decisions; the module docs now point to it. PayloadEncoding reads 2..=9 as an unsupported later generation (the range the header can express) and anything above 9 as invalid. * test(TC-887): at-rest RED size floor and order-free KV pairing Review notes on #324 (N2, N3): - Every blob that must hold a plaintext (each KV put's objects in total, every SQL payload and delta) must be at least that large: TCE1 never shrinks data, so a smaller blob is compressed, not sealed. Catches xz/bz2-style compression of a salted WAL delta that the shared-run and compressibility checks miss. - KV cross-space comparison requires equal object counts per put and compares every pair, instead of zipping objects sorted by name. - The offline selftest also proves a salted deflate-9 blob is flagged by the floor and sealed-like blobs pass it. * test(TC-887): harness ESCROW_NONE mapping and per-check expectations Review notes on #324 (N1, N4): - TC887_AT_REST_ESCROW_NONE=true|false maps to S1's boolean TINYCLOUD_AT_REST__ESCROW_NONE (replaces the scalar ESCROW, which S1's deny_unknown_fields section rejects); mode is validated as off|encrypt|strict. State schema v3. - test --expect takes green|red or kv=green|red,sql=green|red. Each check runs alone and is judged on its own; exit 0 only if all match, 3 per mismatch (KV/SQL-specific UNEXPECTEDLY PASSED / still RED), 4 if any check errored. With mode encrypt|strict, test red defaults to kv=green,sql=red (S3 before S4); CI stays --expect red. - README: SIGKILL of up can't be trapped; run down STATE.
This was referenced Oct 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of TC-887 (encryption at rest), slice S0: a test harness. No product code changes.
What changed
scripts/at-rest-harness/harness.sh(+ README): starts a realtinycloudnode on native PostgreSQL and native MinIO, no containers.binsfetches pinned MinIO/mc community releases from GitHub release assets and checks a sha256 pinned in the script (dl.min.io now answers 410). Linux x86_64 only.upinitdb's PostgreSQL (scram, 127.0.0.1 only, no unix socket), starts MinIO and creates the bucket, then runs a private copy of the node with PG as database, MinIO as S3 block store,TINYCLOUD_STORAGE__SQL__MEMORY_THRESHOLD=0(so SQL saves take the WAL-delta path) and a fresh Static root. Per-run credentials live in 0600 files and reach processes through env files, never argv. Prints thestate.jsonpath (PIDs + start times, ports, paths; no secrets).uptears down everything it started (EXIT/INT/TERM traps; process groups;pg_ctl stopthen group-kill fallback).test green|red [--expect green|red] STATE,status STATE,logs STATE,down [--purge] STATE: an explicit state path is required (no shared "latest" link).downsignals a process group only if its leader is still the recorded process.TC887_AT_REST_MODE/TC887_AT_REST_ESCROWmap toTINYCLOUD_AT_REST__MODE/TINYCLOUD_AT_REST__ESCROW(one marked block; names follow the key-continuity slice). Withencrypt/strict,test redexpects green.tinycloud-node-server/tests/at_rest_harness.rs: signs in like the SDK (SIWE host delegation + session delegation) with fresh wallets and spaces, writes through/invoke.#[ignore], run by the harness): KV writes land in MinIO; SQL writes land indatabase_artifactwith a non-NULLdelta_payload.#[ignore = "TC-887 RED: …"]): for KV objects and SQLpayload/delta_payload: no marker; no SQLite/WAL magic; object keys carry no space id, wallet address or plaintext fingerprint (blake3 of the value as digest or multihash, b64url ±padding, base64 ±padding, hex); the same plaintext written into two spaces gives stored bytes that differ and share no 32-byte run; a 64 KiB repeated value and every SQL artifact don't compress (zlib ratio > 0.95). Each RED test repeats the green "write landed" check first.selftest_predicates_flag_deterministic_transforms_not_ciphertext(runs in plaincargo test): identity, base64, XOR and zlib are flagged; random blobs with a shared TCE1-style header are not.flate2dev-dependency (already inCargo.lock)..github/workflows/rust.yml:at-rest-harnessjob (no service containers,contents: read,persist-credentials: false, state under$RUNNER_TEMP): bins → up →test green→test red --expect red→ logs on failure →down --purge. The RED step passes only on the expected RED outcome and fails distinctly on "unexpectedly passed" (exit 3) or "errored" (exit 4).agent.dev.md/agent.changelog.md: pointer to the harness.How it's tested
Local runs from a clean env (PostgreSQL 14.24, MinIO RELEASE.2025-09-07T16-13-09Z); evidence in the TC-887 Linear thread:
upexit 0 →test greenexit 0 (KV and SQL writes landed; SQLrevision=2,payload73728 B,delta_payload78312 B in both spaces) →test red --expect redexit 0 "RED as expected" (20 KV + 16 SQL findings: space id/address/fingerprint in keys, marker in raw bytes, identical bytes across spaces, ratios 0.004–0.013, SQLite magic) →down --purgeexit 0, nothing left.TC887_NODE_BIN=/bin/false up(exit 2) andtimeout 12 upwith a node that never becomes healthy (exit 124): no process or listener from that run remains.--expectoutcomes shown: expected RED → 0, unexpected pass → 3, node stopped mid-test → 4.TC887_AT_REST_MODE=encrypttoday → exit 3 ("still RED but expected green").GITHUB_ACTIONS=true: all exit 0.state.jsonor outputs.cargo fmt --all -- --check,cargo clippy -p tinycloud-node --test at_rest_harness -- -D warnings,cargo test -p tinycloud-node --test at_rest_harness(self-test passes, 4 ignored),bash -n: clean.Not covered: DuckDB artifacts (opt-in feature; S4 follow-up), no dstack simulator (Static root), PostgreSQL version not pinned (native distro package).
Merge order: after the current main release is cut (TC-887 plan §7).