DecodingUs Grid — the AppView side, end to end - #32
Merged
Conversation
The DecodingUs Grid publishes public-ENA work units that volunteer Navigator
instances lease, compute, and return as a signed digest. This is the coordination
half — the catalogue, the lease, the submission, the credit ledger — which is the
part with no existing analogue anywhere in the three repos. Design:
`documents/design/distributed-compute-grid.md` in the DUNavigator repo, §4 and §12.
Reuses rather than rebuilds: `fed.pds_node` for the node registry (built in
0008_fed.sql for almost exactly this and never wired), `fed.device_key` plus
`sig::verify_signed{,_fresh}` for auth, and `ident.users` for credit attribution.
`fed.pds_submission` is deliberately NOT reused — its status lifecycle means curator
review of a proposed variant call, which is a different thing from digest quorum,
and overloading it would leave both meanings unreadable.
**Claimability is derived, not stored.** The design gives the work unit the states
AVAILABLE → LEASED → SUBMITTED → CANONICAL, but sets `required_replicas` to 2 two
paragraphs later — so a unit routinely needs a second independent result while a
first node still holds a lease, and a unit flipped to LEASED can never reach that
second node. LEASED would have to mean "…and also still claimable", which is not a
state. So `work_unit.state` carries only the genuinely exclusive milestones and
`claim` derives the rest: state is AVAILABLE or CONTESTED, active leases plus
non-divergent submissions are below `required_replicas`, and the calling DID holds
neither a lease nor a submission on the unit. One SELECT … FOR UPDATE SKIP LOCKED
answers that. No replica counter exists, so none can drift from the rows it would
summarise, and a claim never mutates the work unit at all.
That last clause is a correctness rule and not a rate limit: quorum means
*independent* results, so a contributor must never be re-offered a unit it already
holds or has already answered.
The credit ledger is BIGINT milli-cobblestones. NUMERIC has no Rust mapping here —
sqlx is built without a decimal feature and nothing else in the repo uses one — and
adding a workspace-wide dependency for one column is out of proportion, while f64 is
the wrong shape for a value SUMmed over every contribution ever made. NUMERIC(12,3)
had already chosen three decimals, so the integer is that value and it sums exactly.
The digest carries no mt_terminal: `App::analyze_biosample` declines to assign mtDNA
because it "is not final on CHM13", and the Grid realigns to CHM13, so requiring an
exact match on it would have blocked P1 behind a research question.
Migration comments carry the same reasoning, because `sqlx::migrate!` checksums
applied migrations — the SQL that ran can never be edited, so its comments are the
one explanation that cannot drift away from it.
Tests: two unit tests pin the canonical signed strings, which are a cross-repo
contract with the Navigator edge — an accidental reformat now fails here rather than
as a 403 against a released desktop build. Six live-Postgres tests cover replica
bounds, self-replication, the data-kind filter, lease reclamation, submit/resubmit
idempotence, and credit. THOSE SIX HAVE NOT BEEN RUN: the development host has no
reachable Postgres, so the claim SQL is unverified against a real database. Two type
ambiguities were removed pre-emptively for that reason — LIMIT takes a bigint, and
`bigint * interval` has no operator, so make_interval is used instead. Run them
before building on this:
DATABASE_URL='postgres://…@localhost:5432/postgres?sslmode=disable' \
cargo test -p du-db --test grid -- --nocapture
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SequryftjcFkLnjQRMKCx3
…ored `run-once grid-curate` publishes `grid.work_unit` rows, and it makes no network calls at all. The design (§4.5) budgeted an ENA integration here; it turned out not to need one. `crawl_project` has already enumerated every run of a study, grouped the runs by sample, and written each file's URL, md5 and size into `genomics.sequence_file`. Curation is one query over tables we already have. That is also the ENA fair-use control §6.2 asks for, arrived at by accident: a node is handed a finished manifest and never goes discovering files for itself, so a fleet of any size costs the ENA portal nothing. `data_kind` is decided per sample and the manifest is then filtered to match it — any CRAM/BAM makes it a passthrough unit carrying only aligned files, otherwise a FASTQ unit carrying only reads. `build_libraries` already prefers aligned over FASTQ per sample, so the two normally agree; deciding it again here means a manifest can never list a file the data kind says the node will not open, and the download budget cannot be inflated by files nobody fetches. Refreshing a unit updates its manifest and sizes but never its state, required_replicas or canonical digest. Curation describes the input and validation owns the lifecycle, so a re-crawl must not silently un-canonicalise a finished unit — there is a test for exactly that. **This exposed a hole in the credit formula.** `est_bases` is `reads × read_length`, and the crawl leaves `read_length` unset: ENA's filereport exposes `base_count`, but du-external's RUN_FIELDS does not request it and `sequence_library` has nowhere to put it. So the per-Gbp term of §6.3 has nothing to weigh a FASTQ unit by. Deriving an estimate from byte sizes was the tempting fix and is the wrong one — a fabricated number in a ledger that pays people is worse than an honest null. Curation publishes the null and the job warns with a count, so the hole is visible rather than silent. Fixing it means adding `base_count` to RUN_FIELDS and carrying it through the crawl, which belongs with that code and not here. Three curation tests seed through `biosample::upsert_by_accession` + `sequence::ingest_libraries` rather than writing their own rows. The query reads JSONB paths whose shape only `ingest_libraries` defines, so a test that hand-wrote rows could agree with the query while both disagreed with what the crawl stores. Like the substrate commit before it, these tests compile but HAVE NOT BEEN RUN — no reachable Postgres on the dev host. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SequryftjcFkLnjQRMKCx3
…replica slot
Running the suite against a real Postgres for the first time. Both findings are the
kind that only a database can produce.
**`est_bases` decoded as NUMERIC, not INT8.** `SUM()` over bigint widens to NUMERIC
in Postgres; the operands were cast and the result was not. Review could not have
caught it — `total_bytes` on the very next line carries the `::bigint` cast, so the
omission read as consistent with its neighbour.
**The reaper does not free the replica slot, and never did.** `claim` already
ignores any lease past `expires_at`, so a unit held by a node that crashed becomes
claimable by another contributor the moment the lease lapses, with no job run in
between. That is a strictly better property than the design claims for it in §4.3:
the catalogue keeps flowing even while the reaper is down.
The test asserted the design's story rather than the code's behaviour, and the code
was right. Rewritten to pin what actually happens, and a second test added for the
reaper's real purpose — which is the two things that do need a row write:
1. recording the EXPIRED outcome, since trust tiering must tell "timed out" from
"gave it back" and no query over live state recovers that afterwards;
2. letting the *same* node take a fresh lease after overrunning, because the
self-replication guard keys on an unreleased lease regardless of expiry.
That guard is deliberate: relaxing it would let a node re-claim a unit it already
has a row for, and the partial unique index would then turn ON CONFLICT DO NOTHING
into a silently empty result with no explanation. So the node waits for the reaper,
which is the honest ordering — its first attempt really is over.
Worth stating plainly, since it is the argument for running these before building on
them: `grid-validate`'s trust tiering was about to be written on top of a wrong
model of when a lease stops counting.
All ten tests now pass. The Apple `container` published port is unusable for this —
it completes the handshake and resets on the first protocol byte — so connect to the
container's own vmnet address, which changes on every recreate. Recipe is in the
design doc §12.5.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SequryftjcFkLnjQRMKCx3
`run-once grid-validate` clusters the digests on a unit, canonicalizes when one cluster clears both the replica bar and the trust policy, and credits whoever agreed. `run-once grid-reap` closes lapsed leases — which, per the last commit, is not what returns a unit to the pool. Design §6.1 and §12.6. Trust tiers derive from `grid.submission` history and never from the social reputation score. Those are different claims: social standing says a person participates well in the community, grid trust has to say their machine produces correct results. Letting the first vouch for the second would let a well-regarded member canonicalize bad output on reputation alone, which is the attack adaptive replication exists to stop. §6.4's capped reputation event still fires *for* grid work; the arrow only points that way. **Bucketing moved to the server.** §5.2 had the node submit values already bucketed, which would make the bucket function a cross-repo contract: every Navigator release would have to round coverage exactly as the AppView expects, and drift would surface as unexplained DIVERGENT verdicts against honest nodes. Server side, the rule exists once and can be retuned without redeploying a client. The node still signs the raw digest it sends, so the signature covers what it computed, and boundary sensitivity is unchanged by where the rounding happens. **The spot-check needed no schema.** A trusted node's lone submission that draws the 5% simply does not canonicalize yet: required_replicas rises to 2 and the unit stays claimable, so the shadow arrives through the ordinary claim path and the next pass confirms or contests. One code path instead of a SHADOW state and a second one. The draw is Postgres `random()` rather than a hash of the unit id, because §6.2 requires spot-checks to be AppView-chosen — anything derived from the unit or the digest is a rule a contributor could compute in advance and route around. It also avoids adding a random-number crate for a single coin flip. Two rules worth stating because they are easy to get wrong in the other direction: - A contested unit blames nobody. Two conflicting clusters are no evidence about which is wrong, so nothing is marked DIVERGENT and no reputation is docked; the bar rises and the tie-breaker assigns blame. A coin-flip penalty would punish honest work, and under MAX_DIVERGENCE_FOR_TRUSTED = 0 a wrongly-marked contributor loses its tier permanently. - Two independent contributors agreeing suffice whatever their tier. Requiring a trusted node on top of independent agreement would deadlock a young fleet where nobody is trusted yet; the tier rule only governs whether one submission stands alone. Results against different references, or different stack majors, are not clustered together at all — they are answers to different questions, and pooling them would manufacture divergence out of nothing. Constants are conservative placeholders, as §9 asks: promoting too slowly costs duplicated compute, promoting too quickly costs a wrong canonical result, and only one of those is recoverable. 16 unit tests and 13 live-Postgres tests pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SequryftjcFkLnjQRMKCx3
Nine endpoints under /api/v1/grid: five signed mutations (node/register, claim,
heartbeat, release, submit), one signed read (mine), three public (leaderboard,
work/{accession}, stats). Auth is the same Ed25519 device-key path the D1 exchange
and the recruitment Edge use. Design §4.4 and §12.7.
**submit recomputes the digest hash rather than trusting it.** The signed message
covers a hash of the digest; the handler recomputes that hash from the body that
arrived. Without it a node could sign the hash of a good result and post a different
one, and the stored digest_sig would still look valid to an auditor later. The
canonicalization is the smallest contract available — serde_json::to_vec, key-sorted
and whitespace-free because this workspace does not enable preserve_order, and
Navigator uses the same crate under the same default. No field order to agree on and
no float formatting rules. A test pins it, so switching preserve_order on anywhere
fails there rather than as unexplained 400s against shipped desktop clients.
**claim normalizes before it verifies.** data_kinds is uppercased, deduplicated and
sorted, and the signed message covers the normalized form; otherwise ["CRAM","cram"]
and ["cram","CRAM"] are different signed strings for one request and a node whose
ordering differs gets an unexplained 403. The signature covers what the node asked
for, not what the server clamped it to — a node cannot know our bounds, and making
it guess them to produce a valid signature would be an unusable API.
Two endpoints from §4.4 are deliberately absent. /grid/node/heartbeat is redundant:
register_node already stamps last_heartbeat and is idempotent, so re-registering is
the node heartbeat, and two endpoints writing one row is drift waiting to happen.
/grid/heartbeat does not extend a lease, per §4.3 and §12.5.
/grid/mine is new. Every other endpoint either mutates or is public, which left
messages::poll with no caller, and §7.1's Grid panel needs the contributor's own
leases, history, credit and rank — which is not public data. One signed read closes
both gaps.
An uncredited contributor's rank is null rather than a number. The query returns the
position such a contributor would hold, but to a node with its first unit still in
flight that renders as "you are last" — about a board it does not yet appear on.
Null means unranked, which is what is true.
19 unit tests and 16 live-Postgres tests pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SequryftjcFkLnjQRMKCx3
By the time grid-validate shipped, the est_bases gap had stopped being theoretical.
The validator pays BASE_CREDIT + per-Gbp, and est_bases was NULL for essentially
every unit — so a 90 Gbp realignment earned exactly what a CRAM passthrough earned.
The ledger would have gone live quietly wrong. Design §12.8.
ENA had the figure all along: `filereport` publishes base_count on read_run, and
RUN_FIELDS simply never asked for it. It does now, and migration 0076 adds
genomics.sequence_library.base_count to hold it.
A typed column rather than the `atproto` JSONB slot, which was the other option since
that slot already carries {source, run_accession} for crawled runs. But that slot is
provenance — where a row came from. base_count is a measurement of the library, the
same kind of fact as the reads and read_length columns beside it, and it is summed
into an aggregate feeding a ledger that pays real people. In a JSON blob a missing
key reads identically to a zero; in a typed column a NULL is visible.
est_bases now prefers the measurement and falls back to reads × read_length only for
rows predating the column. That product is a mean-length approximation and is wrong
outright for variable-length reads, which is to say for every long-read platform.
Existing rows needed a job rather than a re-crawl: ingest_libraries is idempotent at
*sample* granularity and skips a sample that already has files, which is what keeps
re-crawls cheap and is not worth weakening for one column. So `run-once
ena-base-count` fills the column directly, one run at a time — filereport filters on
whatever accession it is given, so a run accession returns just that run — in
bounded batches with the study crawl's politeness gap. Re-run until nothing is
examined.
Two properties made explicit rather than incidental, both about not corrupting a
ledger:
- the backfill only ever fills a NULL, so re-running cannot overwrite a
measurement and silently change what a contributor was already paid for;
- an empty base_count from ENA leaves the NULL rather than writing a zero. A
submitter who never supplied the figure is not a run that sequenced nothing.
Where no honest figure exists the null survives and grid-curate still warns with a
count. That was right before and is still right; this makes the null rare instead of
universal.
18 live-Postgres tests pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SequryftjcFkLnjQRMKCx3
Picks up `du_domain::fed::Provenance`, which the Grid's fed records need to say that a result is *about* an ownerless public ENA sample while being *authored by* the volunteer who computed it (design §5.1). All three pins move together — du-domain, du-atproto and du-bio are one repository, and letting them drift to different revs of it would be a way to get two versions of the same type in one build. The change du-domain landed is additive: the four Grid records gained an optional, defaulted `provenance` block that is skipped when absent, so nothing in this repo had to change to keep compiling. That is what an additive contract change is supposed to look like from the consumer's side. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SequryftjcFkLnjQRMKCx3
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The coordination half of the distributed compute grid:
the AppView publishes public-ENA work units, volunteer Navigator instances lease them,
compute, and submit signed digests; agreeing results canonicalize and pay compute credit.
This is everything except the Navigator edge. 19 unit tests and 18 live-Postgres tests pass.
4671ad90075_grid.sql+du_db::grid— claim, lease, submit, credit, leaderboardcf3ad2arun-once grid-curate— the work list324a07817cdbb3run-once grid-validate+grid-reap— adaptive replicationd4ae531/api/v1/grid/*endpoints9b8acc40076—base_count, so per-gigabase credit means somethingfc6f8ecb11ba77(Provenance)The design was written well ahead of the code, and four of its mechanisms didn't survive contact
Each is corrected in the design doc (§12) as well as here.
The unit state machine can't express replication. §3 gives the work unit
AVAILABLE → LEASED → SUBMITTED → CANONICAL, and setsrequired_replicas = 2two paragraphslater. A unit routinely needs a second independent result while a first node holds a lease, so
LEASEDwould have to mean "…and also still claimable", which is not a state.work_unit.statenow carries only the genuinely exclusive milestones and claimability is derived from the lease
and submission rows. No replica counter exists, so none can drift.
Curation needed no ENA integration. §4.5 budgeted one.
crawl_projecthad already resolvedevery run at file level, so curation is a projection of existing tables and makes zero network
calls — which also delivers §6.2's ENA fair-use control for free.
The reaper doesn't free replica slots.
claimalready ignores any lease pastexpires_at, soa slot frees itself the instant a lease lapses — the catalogue keeps flowing even while the reaper
is down, which is better than the design claimed. What the reaper is actually for is recording the
EXPIREDoutcome (trust tiering needs it; no derived query recovers it) and letting the same noderetry after overrunning.
The spot-check needed no schema. A trusted node's lone submission that draws the 5% simply
doesn't canonicalize yet:
required_replicasrises to 2 and the unit stays claimable, so the shadowarrives through the ordinary claim path.
Three places a contract was avoided rather than created
bucket function a cross-repo contract — a Navigator release rounding differently would produce
unexplained
DIVERGENTverdicts against honest nodes. Server-side it exists once and can beretuned without redeploying a client.
submitrecomputes the digest hash rather than trusting it. A signature over a hash provesonly that the signer knew the hash; without recomputation a node could sign a good result, post a
different one, and still audit clean later.
claimnormalizesdata_kindsbefore verifying, so["CRAM","cram"]and["cram","CRAM"]aren't different signed strings for one request.
Judgement calls worth a look
claims — community standing must not vouch for a machine's output.
wrong, and
MAX_DIVERGENCE_FOR_TRUSTED = 0makes a wrongDIVERGENTmark permanent.fleet where nobody is trusted yet could never canonicalize anything.
NUMERIChas no Rust mapping here (sqlx isbuilt without a decimal feature) and
f64is the wrong shape for a column summed over everycontribution ever made.
est_basesprefers ENA's measuredbase_countoverreads × read_length, which is amean-length approximation and wrong for every long-read platform. Where neither exists the value
stays NULL and the job warns — a fabricated number in a ledger that pays people is worse than an
honest absence. Old rows need
run-once ena-base-count; a re-crawl can't fix them, becauseingest_librariesis idempotent at sample granularity.Verifying
Live-DB tests skip without
DATABASE_URL, socargo test --workspaceis green without one. To runthem for real, note the Apple
containerpublished port is unusable — it completes the TCPhandshake and resets on the first protocol byte — so use the container's vmnet address:
Serially, with ≥4 GB: each test migrates its own database, and ten concurrent
CREATE EXTENSION postgiscalls against 1 GB kill the backends at migration 1.CI will likely be red, and not because of this branch
origin/mainalready failscargo clippy --workspace -- -D warningsunder current stable clippy,in files this branch never touches:
du-db/src/place.rs:297(is_none_or, stable in 1.82 vs thedeclared MSRV
1.80) anddu-web/src/origins_layout.rs:505,512. All arrived with0dabbab. Happyto fix them here or in a separate PR — say which you'd prefer.
🤖 Generated with Claude Code
https://claude.ai/code/session_01SequryftjcFkLnjQRMKCx3