feat(admin): read-only GET /admin/stats owner list (TC-1482)

[samgbafa]

TC-1482 (parent TC-1479). Read-only GET /admin/stats for the scoreboard's protocol-user count. Consumer: OpenKey openkey#296 (dba4e5a).

Release: targets the next node release after 1.21.2 (not 1.21.2). Opened against main; do not merge into 1.21.2.

What is counted (differs from usage_bytes > 0)

This is the complete counting rule. The README's "Admin stats" section carries the same text.

• Counts: an owner holding live (deleted = false) KV in any space except account, or database artifacts (SQL/DuckDB database_artifact rows) in any space except account and secrets, except KV in the public space whose key starts with .well-known/vault- (vault metadata).
• Excluded (automatic sign-in records, not use of the node): the account space entirely (KV and databases); the secrets table created by OpenKey's bootstrap (CREATE TABLE secret_records; database artifacts in secrets, while KV stored in secrets still counts); and the vault metadata written by vault.unlock() (.well-known/vault-pubkey, .well-known/vault-version, .well-known/vault-space in public). An app may unlock the vault automatically, so counting those rows would inflate "users with data" the same way the secrets table did. (Naomi's rulings: review round 1 H1 for account/secrets, then the vault metadata.)
• App first-use tables count. A database an app creates on first use (for example listen's conversations database, feed-learn's tables) is a person actually using an app that stores on TinyCloud. Artifact rows are never deleted, so a created-then-emptied/dropped database still counts; database_legacy_artifact (fence markers) is not read.

Matching is exact. The space name is the last : segment of the space id (the id parser splits at the last :, so a name never contains one): xsecrets, secrets2, my-account, Secrets, Account are ordinary spaces (tested, including on SQLite where LIKE would be case-insensitive). The vault exclusion is exactly the space public and keys that start with .well-known/vault-, case-sensitive, compared with SUBSTR(key, 1, 18) = '.well-known/vault-' (no LIKE, so no wildcard or case folding). Everything wider still counts: other .well-known/* keys (including .well-known/vault, no dash), any other KV in public, the same prefix in any other space, .Well-Known/vault-x. A public space with at least one live KV row that is not vault metadata makes its owner count. So an owner who only signed in, or only signed in and unlocked the vault, is not listed.

Owner = the address in tinycloud:pkh:eip155:<chain>:0x<addr>:<name>, lowercased, deduped across spaces and chains. Every entry is exactly 0x + 40 lowercase hex. Spaces whose owner is not such an address (did:key owner, wrong length/non-hex, non-eip155, unparsable id) are left out.

Response

200 application/json, Cache-Control: no-store: {"owners":["0x…"],"count":N,"generatedAt":"<rfc3339>"}, sorted. No truncation flag. OpenKey makes no partial use of a response, so the route never returns a partial 200: more than 200,000 owners (or 400,000 distinct counted spaces, the memory bound, ~110 B each; the excluded sign-in rows are not counted, so the cap no longer fills at ~200,000 signed-in accounts) is a bare 507; a DB failure is a bare 500.

The single query

SELECT u.id FROM (
  SELECT s.id AS id FROM space s
  WHERE SUBSTR(s.id, LENGTH(s.id) - 7) <> ':account'
    AND EXISTS (SELECT 1 FROM current_kv k WHERE k.space = s.id AND NOT k.deleted
      AND NOT (SUBSTR(s.id, LENGTH(s.id) - 6) = ':public'
               AND SUBSTR(k.key, 1, 18) = '.well-known/vault-'))
  UNION
  SELECT a.space AS id FROM database_artifact a
  WHERE SUBSTR(a.space, LENGTH(a.space) - 7) <> ':account'
    AND SUBSTR(a.space, LENGTH(a.space) - 7) <> ':secrets'
) u LIMIT 400001

(tinycloud-core/src/data_owners.rs; owner parsing/dedupe in Rust.) One statement, no per-space store_size loop (the TC-1274 risk in /admin/usage). Plan on PG14, 20k spaces / 400k kv rows: semi join, one current_kv_pkey probe per space that stops at the first live row, index-only scan of space, one pass over database_artifact (PK covers space, no payload read); 85 ms. Runs on SQLite, PostgreSQL and MySQL (no dialect-specific functions). Not wrapped in a read-only transaction: BEGIN/COMMIT would add round trips and the handler issues only this SELECT.

The round-trip test counts 2 wire round trips for the request: the statement and sqlx's release ping on every connection return (the harness counts it for all flows), exactly 1 statement, identical with 3 and 303 spaces.

Auth (security-critical)

• New NODE_STATS_READ_TOKEN, read per request like TINYCLOUD_ADMIN_SECRET (routes/admin_stats.rs, its own guard StatsAuth).
• Unset / empty / whitespace-only / shorter than 32 bytes / equal to TINYCLOUD_ADMIN_SECRET: route disabled, 404 identical to an unknown route (tested for status, body, content-type, with and without any credential).
• Generate with openssl rand -hex 32 (64 hex characters; Bobbie's 32-byte hex token works). Review round 1 L1: a shorter token counts as disabled, because the route is internet-reachable with no rate limit.
• Missing/malformed/wrong credential (including the admin secret): bare 401, identical for every case, no detail.
• Comparison: both sides SHA-256'd, digests compared with subtle::ConstantTimeEq (neither content nor length leaks).
• The admin secret does not open this route; the stats token is refused by every AdminAuth route (/admin/quota*, /admin/usage), tested with and without the admin secret configured. GET only.
• No token/address in logs, errors, traces, metrics: constant guard/log strings, no DB error text logged, no observe_space. The existing request logger logs the route template only.

Tests

• tinycloud-core data_owners (SQLite and PostgreSQL): account-only excluded; account + data once; two chains/spaces once; mixed case deduped; deleted-only excluded, live beside tombstone included; artifact-only included; malformed ids excluded; bounds are errors not truncation; owner-parsing cases. New: exact-name exclusion matrix (secrets KV counted, secrets/account artifacts and account KV not; xsecrets, secrets2, my-secrets, Secrets, SECRETS, xaccount, account2, my-account, Account, ACCOUNT, default, applications, public counted); distinct-space bound (20 sign-in-only accounts fit a bound of 1).
• Sign-in replay (sign_in_only_owners_are_not_listed, both backends): replays OpenKey's bootstrap for 12 accounts (5 space rows each; the 10 ACCOUNT_INDEX_SCHEMA statements, the 5 spaces rows and the secret_records schema through the real SqlService + SeaOrmDatabaseArtifactRepository, so the database_artifact rows are the node's own; the 7 account KV rows). It first checks the replay produced 1 artifact in account, 1 in secrets and 7 account KV rows per owner, then asserts: sign-in only => not listed; sign-in + vault unlock (the three .well-known/vault-* rows in public, checked present and live) for 7 accounts => still not listed; a stored secret (secrets KV), KV in default, or a SQL table in default => listed. Each unlocked account is then given one more row, one at a time, and must appear (and only it): notes/x in public; .well-known/vault-pubkey in default (prefix in another space); .well-known/other in public; .well-known/vault (no dash) in public; .Well-Known/vault-x in public (case). A deleted notes/gone in public beside the vault rows does not count. Mutation checks of the vault filter (each fails on SQLite and PostgreSQL): removing it fails the "unlock only" assertion (7 owners wrongly listed); dropping the public name condition fails the prefix-in-another-space case; dropping the prefix condition (all public KV excluded) fails the notes/x case and exclusions_are_exact (public must count); widening the prefix to .well-known/ fails the .well-known/other case; a case-insensitive prefix fails the .Well-Known/vault-x case. KV rows are inserted directly (the KV write path needs signed delegations; data_owners reads only space and current_kv).
• tests/admin_stats.rs (SQLite + PostgreSQL scenario incl. a sign-in-only secrets artifact owner and a secrets KV owner; SQLite auth matrix; disabled route == unknown route incl. a 31-byte token with the correct header, a 32-byte token works; 200,000 owners → 200, 200,001 → 507).
• tests/admin_stats_logs.rs: runs the real binary (normal, debug, trace), asserts neither the token, admin secret, presented wrong token, address nor space id appears in stdout/stderr, and one /admin/stats request event per call. Mutation-checked: logging the owner list fails it.
• tests/postgres_round_trip_budget.rs (existing wire-counting proxy harness): one statement, round trips independent of space count.
• test_support: PostgreSQL test schema names now carry a process counter; two tests could read the same clock tick (schema ... already exists, seen once in 15 runs; 0 in 40 after).
• Local (head e6ed475): fmt, clippy --workspace --all-targets -D warnings, tinycloud-core data_owners 11/11 (SQLite and PostgreSQL 14), admin_stats 5/5, admin_stats_logs 3/3, postgres_round_trip_budget 4/4. The earlier note about postgres_space_filter_survives_a_collation_that_ignores_the_sentinel (needs PG 16 ICU rules) was from the full core lib run, which was not repeated for this change.

Measured cost (reviewer)

4.35 s warm on local PostgreSQL 14 at 300,000 spaces / 1.5M keys (50k owners x 6 bootstrap spaces, parallel seq scan of space plus one index-only probe per space). Measured before the vault-metadata condition, which adds one prefix comparison per current_kv row a probe reads; it was not re-measured. The cost follows all space rows, not only listed owners. No query redesign in this PR; the README documents the scaling limit.

Not done / follow-ups

• Not deployed: docker.yml and the compose files do not pass NODE_STATS_READ_TOKEN, so the route stays 404 in production until a reviewed change adds the secret, the PHALA_ENV_ pass-through and a compose line (changes the reviewed compose hash). Documented in docs/releasing.md. That file's bullet ("not passed to the CVM by docker.yml...") belongs to this PR and becomes stale once the deploy-wiring PR (TC-1482: wire optional NODE_STATS_READ_TOKEN into the production deploy (companion to #370) #371) lands; whoever merges second updates it.
• log-shipper/fields.json (canonical copy of tinycloud-dev ops/logging/fields.json) and deploy/ingress/redacted-logging.conf are untouched: the shipper drops an unlisted route and the ingress logs it as <other>, both safe. Add /admin/stats there in a coordinated change if the route should be visible in shipped logs.
• Docs: README (endpoint table, env var table, "Admin stats" section), CHANGELOG, docs/releasing.md.

Overlap with other open PRs

CHANGELOG.md (top of Unreleased, same spot as #353/#356/#357): textual conflict, trivial. tinycloud-core/src/db.rs: one hunk after list_space_ids (~l.722), no overlap with #353/#357. tinycloud-node-server/src/lib.rs: import at ~l.123 and route list at ~l.481; #359's nearest hunk is at l.107 (adjacent, likely clean); #355 touches other lines.

[samgbafa]

CHANGES NEEDED: see TC-1482 (head 5fcb331)

Independent security review (Opus gate) of exact head 5fcb3313cc1e85399d35f38cfda289386335ec50.

Blocker (H1): OpenKey's sign-in bootstrap runs the secret_records schema in <addr>:secrets (openkey tinycloud-bootstrap.ts runSecretRecordsSchema, js-sdk bootstrap step secrets:secret-records-schema). The node persists that as a database_artifact row, and data_owners counts any artifact row outside account. So every bootstrapped OpenKey account is listed, which is what Naomi's account-only ruling excludes. Reproduced through the real SqlService path: an owner who only bootstrapped is returned. Fix: don't count artifacts in the bootstrap-written databases (account, secrets), keep counting live KV in secrets (secret values), confirm the set with Naomi, and add a test that replays the real bootstrap shape.

Medium (M1): the 400k bound counts output rows, which today are about 2 per bootstrapped account. So the route returns 507 at ~200k accounts whatever the real owner count. Cost scales with all space rows (6 per account): 4.35 s warm on PG14 with 300k spaces / 1.5M kv. It fails closed, but the bootstrap spaces should be filtered in SQL.

Low: no minimum token length on a public /admin/* with no rate limit (suggest disabling below 32 bytes). Path aliases (/admin/stats/, //admin/stats, %73) and HEAD reach the guarded handler, so there is no bypass. CORS * + credentials on /admin/* (pre-existing, not exploitable). Unlocked set_var in the multi-thread round-trip test.

Auth OK: disabled == unknown 404; the admin secret never opens the route and the stats token opens no AdminAuth route; constant-time compare; no DB access before the guard; nothing logged. The log test is mutation-meaningful: a trace-level owner log fails it.

Run locally (SQLite + local PostgreSQL 14, no Docker): data_owners 5/5, admin_stats 5/5, admin_stats_logs 3/3, PG round-trip test (1 statement + release ping), EXPLAIN ANALYZE, live-binary curl probes. Not run: MySQL, production data.

CI at 5fcb331 when posting: pass: Build dstack ingress image, Version guard, at-rest harness, build (workspace), capabilities, clippy (workspace), fmt (workspace), m1-realdata, plan, separate workspace lockfiles, w5 workspace lockfile. Pending: build, test (workspace, PostgreSQL-backed). Skipped: announce, build-dstack, build-global-artifacts, build-local-artifacts, host, publish-homebrew-formula, rollback-audit. The verdict is code-only for the pending checks.

[samgbafa]

APPROVE: see TC-1482 (reviewed exact head e6ed475)

The full counting rule in the README ("What is counted") and in the PR body matches Naomi's ruling word for word: live KV in any space except account, or database artifacts in any space except account and secrets, except KV in public whose key starts with exactly .well-known/vault-; app first-use tables (listen conversations, feed-learn) count. CI at e6ed475: 13 pass (incl. build, build (workspace), test (workspace, PostgreSQL-backed), at-rest harness), 9 skipped.

[samgbafa]

APPROVE: see TC-1482 (main-merge only; +/- lines byte-identical vs e6ed475; head a746d2c)

Detail: merged origin/main a8ddd90 (1.21.2 + #371 + #372 + #163 pin) into the approved head e6ed475 with a merge commit. The only conflict was CHANGELOG.md (our Unreleased entry vs main's Unreleased + [1.21.2] sections): resolved by keeping both Unreleased entries (main's #371 entry first, then this PR's GET /admin/stats entry) and main's [1.21.2] section untouched. docs/releasing.md and tinycloud-core/src/db.rs auto-merged. The PR diff against main has the same 13 files as before; the patch-id differs (58df35c0… vs cbc3f6ee…) only because main moved the context; the added/removed lines are byte-identical per file for all 13 files (sha256 of the +/- lines: CHANGELOG.md, README.md, docs/releasing.md, data_owners.rs, db.rs, lib.rs (core + server), test_support.rs, admin_stats.rs, routes/mod.rs, the three test files). Known and tracked: the docs/releasing.md bullet saying the token is "not passed to the CVM" is stale now that #371 is merged: fixed in the post-release docs follow-up (no re-verdict). CI must be green at this exact head before merge.