Skip to content

feat(clusters): record run-scoped computed candidate generations - #268

Merged
hannesrudolph merged 5 commits into
mainfrom
feat/cluster-candidate-generations
Oct 22, 2026
Merged

hannesrudolph merged 5 commits into
mainfrom
feat/cluster-candidate-generations

Conversation

@hannesrudolph

@hannesrudolph hannesrudolph commented Oct 22, 2026 •

Copy link
Copy Markdown
Member

The native available-subset producer keeps durable cluster IDs stable by quarantining ambiguous candidates: merges, splits and protected targets. On run 888 that was 2,461 of 9,411 selected threads in 86 components. The current computation for those threads is not visible anywhere, and v2 cannot publish them without rewriting identity history (see the identity analysis, option 1).

This adds a versioned, run-scoped computed-candidate domain without touching durable identity.

Writer. In the same native transaction, after v2 publication and scope validation, NativePublishClusters records the quarantine plan's complete candidate partition as an immutable gitcrawl.cluster-candidates.v3 generation (SaveClusterCandidateGeneration). It then moves the per-repository selection pointer to the new generation and rechecks the quarantine postimage hash. Each generation records:

  • its bindings: run, algorithm, input hash, quarantine proof hash and partition hash;
  • each candidate's members and representative;
  • each candidate's v2 durable_status (published with durable_cluster_id, or identity_quarantined);
  • observed overlaps with active durable groups.

Blocked-identity runs still record a generation; no-op runs record none. The six history tables (cluster_groups, cluster_memberships, cluster_overrides, cluster_closures, cluster_aliases, cluster_events), the strict v2 scope and v2 reader semantics are unchanged.

Invariants.

  • Candidate IDs are GENERATION:ORDINAL and never alias durable IDs. Overlaps are lineage observations, not equivalence.
  • Each generation must partition exactly the selected cohort. A thread in two candidates, a missing or foreign thread, an outside representative, another repository, or an owner-excluded thread are each refused.
  • Triggers make generation rows immutable. The one exception: once owner purge has deleted a thread, its member rows may be deleted, and readers derive counts and representatives from the remaining rows.
  • RollbackClusterCandidateGeneration selects the previous generation; the next accepted run selects its own.
  • Portable exports drop the new tables, children first.

Readers. gitcrawl cluster-candidates owner/repo [--generation N|--generations] [--limit N] [--members] [--json] reads a generation. --rollback is available only in the main CLI; the dedicated native cluster reader permits reads.

Tests.

  • Store tests cover merge/split visibility, durable history byte-equality across generations, selection, rollback, immutability, invalid partitions, the outside-transaction refusal, owner purge and exclusion refusal.
  • The overlay native quarantine test now asserts that the blocked run selects a fully computed generation, a no-op adds none, full recovery yields no quarantined candidates, and rollback works.
  • CLI and reader tests cover rollback refusal in the reader.
  • Portable shaping and identity tests cover the dropped tables.

The Clawmmunity selection-contract change needed to consume this is specified separately; Claw is owned elsewhere.

Owner decision (Hannes delegated): immutable computed-candidate history and its selection pointer are a supported, versioned source contract. Clawmmunity consumption is specified separately and activated by its owner.

Every accepted native cluster run now records its complete computed
partition as an immutable gitcrawl.cluster-candidates.v3 generation in the
same transaction, with each candidate's members, v2 durable status and
observed overlaps with active durable groups, and selects it. Durable IDs
and the six history tables are untouched and the quarantine postimage is
rechecked after the write, so quarantined candidates become visible in an
explicit run-scoped domain without asserting identity equivalence.

Generations are immutable except for owner removal. Rollback moves the
per-repository selection to the previous generation. Portable exports drop
the new tables. Add cluster-candidates to the CLI and the native reader.
Key the immutability exception on the thread no longer existing instead of
on thread_exclusions, so schema changes to the exclusion tables cannot
break the trigger; purge deletes the thread before its member rows.
@clawsweeper

clawsweeper Bot commented Oct 22, 2026

Copy link
Copy Markdown

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

…ble scope

Resolve each published candidate's durable ID after the v2 write, so
identities created by the run are recorded instead of a zero target.
Drop the derived candidate tables from a disposable portable snapshot
before repository filtering, which would otherwise hit their immutability
triggers.
Keep ordinary store opens free of the new DDL; readers, purge and portable
shaping already treat absent tables as no generations.
Wrap generation listing, selection lookup and candidate reads in one read
transaction so a concurrent rollback or owner purge cannot produce two
selected generations or counts that disagree with members.
@hannesrudolph

Copy link
Copy Markdown
Member Author

Local ClawSweeper (PR #1756 build) on the final range 04d7441..d7c43db: patch is correct, no findings, security cleared. Earlier rounds found three issues, each fixed with tests:

  • newly published durable IDs were not recorded;
  • portable repository filtering hit the immutability triggers;
  • CLI reads could straddle a rollback or purge (they now use one snapshot).

Local gates at d7c43db all passed:

  • gofmt, vet and deadcode clean; govulncheck found no vulnerabilities;
  • make test-coverage (85.0%), smoke and docs;
  • overlay native producer tests (-tags native_cluster_overlay);
  • race tests on internal/store and the cluster CLI tests.

The runtime proof will come from the first ordinary native maintenance run after deployment; I will add it here.

@hannesrudolph
hannesrudolph merged commit c6c87b4 into main Oct 22, 2026
13 checks passed
@hannesrudolph
hannesrudolph deleted the feat/cluster-candidate-generations branch October 22, 2026 17:00
@hannesrudolph

Copy link
Copy Markdown
Member Author

Deployed and proven on the live archive. The first ordinary native maintenance run after deployment (producer built from 38d6aee, which includes #269's pair-cache predecessor fix) exited 0 with combined_published.

  • Run 896 (v2 durable scope, unchanged semantics): selected 9,384; published 6,907; identity_quarantined 2,477 across 97 components. The previous run 895 had the same 2,477 quarantined.
  • v3 generation 1, bound to run 896 and selected with reason accepted, covers the full partition: 5,249 candidates and 9,384 members. Of those, 4,957 candidates (6,907 threads) were published under durable IDs, all with durable_cluster_id recorded. The other 292 candidates (2,477 threads) were durably quarantined and are now visible in the run-scoped domain, with 5,516 observed overlaps.
  • The shipped reader read it: gitcrawl-cluster-reader --json cluster-candidates openclaw/openclaw --limit 3 returned 1:215, 1:221 and 1:647 (40 members each, identity_quarantined, with 6, 5 and 2 overlaps). --rollback is refused by the reader.
  • The quarantine postimage recheck passed. The durable tables only grew through ordinary v2 publication (groups 13,085 to 13,092).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant