Decision: ADR-095. The full-text index is a disposable node-local derived index with exact source-cut/policy/schema manifests and the selected ZoneTree.FullTextSearch provider. Leased generations deliver publication and retirement; online maintenance delivers base-cut capture, delta catch-up and background build outside the canonical read gate, with canonical publication of the online generation. The capture/lifetime contract is RuntimeReadCuts under ADR-097. Files are owned by the node-local ZoneTree owner and never move with grain activations; distributed search merges results from several partitions through Orleans calls and does not share index files.
| Requirement | Acceptance and mapped tests |
|---|---|
| REQ-LEASE-001: publication retains borrowed old generations | AC-LEASE-001: an actual generation lease remains usable and its owned files remain present while another source-cut generation publishes; current acquisition selects only the fully verified replacement; the last old release retires only that generation. TextGenerationLifetimeTests |
| REQ-LEASE-002: finite lifetime and admission | AC-LEASE-002: at most 2 active leases, one builder, one retired leased generation and one current generation; at most 3 physical generations including the staged builder; total disk/file/metadata/posting limits account every owned generation. Exact saturation/cancellation releases admission and a following request succeeds |
| REQ-LEASE-003: cleanup and shutdown preserve truth | AC-LEASE-003: failed refresh/close/delete retains the unsettled owner/handle for joined retry, both primary and cleanup failures survive, repeated release/dispose cannot double-release or delete another generation; shutdown joins leases/build and closes every owned handle |
| REQ-LEASE-004: existing runtime behavior stays exact | AC-LEASE-004: actual SearchEngine/GraphSearch text/vector/corpus results, visibility/cut mismatch, restart and process interruption remain exact through the unchanged ITextProjection interface |
No raw borrowed storage view may escape its gate. leases operate on their owned generation; each query still performs current canonical candidate checks inside its authorized read cut. A source-cut mismatch builds a new generation and never borrows an old manifest as current truth. Publication closes and verifies the staged index and flushes its manifest before changing the manager's current pointer. A retired lease can finish its originally scoped work but cannot read new canonical state or make a current-authorization claim.
The lifetime-long single semaphore is replaced by a short synchronized manager transition plus independent bounded lease/build admission. No monitor or store gate is held across waits or user callbacks and no polling wait runs on a grain thread. The ITextProjection/lease public shape, faults and generated identity are unchanged. Active generation readers share only provider operations proven safe, otherwise one reader per generation and explicit saturation; an index borrowed by another lease is never mutated or closed. The maximum counts are ceilings, not permission to invent provider thread safety.
Capacity preflight must not reopen the published WAL while its index still owns that file. Inside the physical gate it captures only the manager's at most three actual generation references. An exact manager-owned, published generation with a still-open index receives a bounded live ownership/layout/size preflight without reopening payload files: root, leaf, source node and scope are verified against the retained generation and owner/manifest metadata, unknown, linked, nonregular or foreign paths are rejected, and all file, depth, generation and aggregate-byte ceilings are kept. Every tracked file is inspected without reopening, a current file over MaximumDiskBytes is rejected, and a live file length is not compared with its closed manifest length. Runtime generation references cannot be supplied by a caller, inferred from a sharing exception or replaced by a trusted-leaf list; every other generation keeps the complete checksum validation, and publication, reopen, retirement, deletion and restart keep exact lengths and checksums. The final aggregate census in RuntimeTextProjection.Release follows the same completed-operation rule as TextSettlement.Refresh: an incomplete or cancelled operation settles without consulting its exhausted caller budget while keeping all real file, byte and generation ceilings and distinct cleanup errors.
The overlap fixture owns a linked cancellation source for its admitted replacement task; an observation deadline cancels that source, resumes its actual posting barrier and joins the original task before the lease or projection is disposed. Expected generation leaves are sorted ordinally before the ordered comparison, every borrowed lease settles before its projection's shutdown even when an earlier operation fails, and the restart fixture preserves its real borrowed retired generation by the owned foreign-file failure before the failed projection shutdown closes both indexes. Frontend/new SDK/MCP syntax is N/A: unchanged public search surfaces consume this manager. Migration is N/A: no canonical record or persisted index format changes. Rollback stops the capable node and rebuilds disposable indexes.
flowchart LR
Build[One verified builder] --> Current[Publish current generation]
Current --> Retired[Previous generation retains old lease]
Retired --> Drain[Last reader settles]
Drain --> Delete[Verified owned retirement]
| Requirement | Complete operation acceptance |
|---|---|
| REQ-ONLINE-001 | AC-ONLINE-001: same-gate capture pins the actual snapshot and captures persisted principal/policy/resource, applied position, outbox upper and node/partition identity with an active canonical projection consumer retaining the necessary deltas; copied-byte seed traversal runs outside the canonical gate while genuine public update and delete commands complete |
| REQ-ONLINE-002 | AC-ONLINE-002: a distinct staged generation consumes the actual ordered retained deltas to a finite fresh upper; full independent literals prove updated terms replace original terms, deleted entities are excluded, and unchanged entities and complete metadata stay exact |
| REQ-ONLINE-003 | AC-ONLINE-003: final authority/cut/retention and complete corpus validation precede a fully flushed verified catalog publication; rejection, cancellation or exhausted bounds preserve the usable original active generation and joined cleanup |
| REQ-ONLINE-004 | AC-ONLINE-004: genuine same-root process interruption at the publication boundaries and same-owner restart select the last complete active generation, preserve exact canonical records and receipts and reject incomplete or corrupt staging; process kill is not power-loss durability |
| REQ-ONLINE-005 | AC-ONLINE-005: a real old lease keeps its exact original result and files through replacement; its final release retires only that generation, and an actual cleanup failure retains ownership and failures followed by a repaired healthy continuation |
Stages: internal capture composition and current-only derived catalog contract; capture metadata and pin; bounded off-gate seed build; ordered delta catch-up; final authority and complete corpus validation; durable selector publication; interrupted-publication recovery; leased retirement. Removing the existing RequireUpper equality is not delta catch-up, and the explicit TextIndexSelection, its single enrollment and the immutable Build/Restore upper U stay exact; implicit Search performs no canonical maintenance effect and no enrollment. Core/Search owns canonical capture/auth and consumer retention; the StorageRecovery provider owns the snapshot and physical-gate lifetime; Server/Search owns the staged build, derived catalog and retirement; Orleans/Search owns typed capabilities and the bounded parent/child calls; Query/Search consumes a validated selection without owning storage.
flowchart LR
Cut[Same-cut metadata and pin] --> Build[Off-gate staged build]
Build --> Delta[Ordered retained deltas]
Delta --> Validate[Fresh authority and complete corpus]
Validate --> Catalog[Durable derived catalog]
Catalog --> Retire[Leased retirement]
OperationKind.MaintainOnlineTextIndex=34 (appended; Batch 0 and the earlier kinds are unchanged). OnlineTextIndexMaintenanceRequest (alias keyload.search.online-text-index-maintenance-request): Id0 CommandId, Id1 Consumer (ProjectionConsumerRef), Id2 Collection, Id3 Field, Id4 ConsumerGeneration, Id5 NodeId, Id6 the partition identity; there is no caller-provided cut, trusted principal, capacity or deadline, and the persisted current administrator, consumer definition/generation/checkpoint, resource/field policy and partition identity are reloaded before every effect. OnlineTextIndexMaintenanceResult (alias keyload.search.online-text-index-maintenance-result): Id0 CommandId, Id1 Consumer, Id2 ConsumerGeneration, Id3 BaseCut and Id4 PublishedCut (TextIndexSourceCut), Id5 IndexSha256, Id6 TrackedRecords, Id7 Checkpoint (ProjectionBatchResult). BaseCut is the real initial snapshot and PublishedCut the final validated applied/outbox/owner authority; no later canonical head is claimed from an original completed result. A whole original stable-ID replay retains all fields and a changed same-ID request is rejected. OnlineTextPhase values: 0 Captured, 1 Seeded, 2 CaughtUp, 3 Validated, 4 Published, 5 RetirementMarked; GrainRequestProgress adds nullable OnlineTextPhase at Id4 (a closed operation phase, not a payload log or completion counter).
The SDK extension is MaintainOnlineTextIndexAsync returning Task<Result<OnlineTextIndexMaintenanceResult>> over the original write/commandId/cancellation path; the route is /search/text/online/maintain, the on-demand tool is keyload_search_text_online_maintain, and Q1 SDK and official MCP use CALL keyload_search_text_online_maintain(@arguments) through the existing SqlOperationRequest/catalog executor with no extra SQL grammar. Its advisory hints are readOnly=false, idempotent=true and destructive=true; guidance requires an already administrator-configured consumer, and a timeout or disconnect is an unknown write outcome. Initial discovery is not enlarged. The parent runs on the caller's connection grain with the ManagedCode CQRS stream (ICqrsStreamWriter<GrainRequestProgress,GrainOperationReply>): each bounded producer write is awaited, existing stream admission/frame limits/backpressure/cancellation and terminal failure mapping apply, and exactly one terminal result follows the actual publication and settlement. Producer, readers, child operations, pin and CTS are joined on success, cancellation and failure with primary plus cleanup errors and fatal semantics preserved; there is no one-way delivery, detached work, polling or deadline reset. The parent and its sequential children use the same connection identity, each with a fresh scoped signed identity and current persisted administrator authority, consume the existing MaximumConnectionOperations admission and use the exact original request expiry; frame accounting is operation scoped under the original MaximumTotalFrames, with each yielded child chunk admitted before the drain consumes it. The bounded ordered loop honours the configured maximum pages and request expiry/buffers; exhausted page/time/byte allowance or lost retained history is a terminal refusal that preserves the original active index, followed by a healthy continuation.
The private operation OnlineTextPublicationPhase=35 (purpose keyload-online-text-publication-verified) and the private capability read OnlineTextMaintenance (purpose keyload-online-text-capability) are signed through the GrainRequestCodec and identity restoration; generic public command and read purposes reject these identities and generic public command creation excludes 35. The internal Core factory alone issues it after validating the prepared candidate; the replicated payload is copied immutable data and the session and prepared bytes are node-local, bounded and owned.
OnlineTextPublicationPhaseCommand (alias keyload.core.search.online-text-publication-phase): Id0 Request, Id1 Result, Id2 DataEpoch, Id3 Leaf, Id4 ManifestSha256, Id5 GenerationId, Id6 ExpectedCurrentCommandId (nullable), Id7 SessionId; the command ID is the original public CommandId. StoredOutcome.OnlineTextAuthority (Id8, nullable) is OnlineTextOutcomeAuthority (alias keyload.core.search.online-text-outcome-authority): Id0 ParentFingerprint, Id1 Collection, Id2 Field, Id3 DataEpoch, Id4 the partition identity, Id5 Leaf, Id6 ManifestSha256, Id7 GenerationId; the full immutable public result stays StoredOutcome.Result. The canonical online-text-current family holds OnlineTextCurrentPublication (alias keyload.core.search.online-text-current-publication: FormatVersion 1, PrincipalId, CommandId, ParentFingerprint, Consumer, ConsumerGeneration, Authority, PublishedCut) for the exact consumer partition/collection/field/generation and takes part in the ordinal partition family inventory, snapshot, backup and key validation; no caller document stores private control data. DataEpoch is the store identity format version (TextSeedCapture.DataEpoch), a format gate rather than a physical generation counter. The Orleans capability request (OnlineTextCapabilityRequest: SessionId, Request, Kind, Page, CheckpointIntent, Acknowledged; kinds 0 ResolveOriginal, 1 Capture, 2 Seed, 3 PreparePage, 4 ApplyIntent, 5 SettleCheckpoint, 6 Validate, 7 IssuePublication, 8 ReconcileCommitted, 9 Abort) and result (OnlineTextCapabilityResult: SessionId, BaseCut, CurrentCut, ThroughSequence, TrackedRecords, IndexSha256, OriginalResult, IssuedPublication, CheckpointIntent, CurrentCommandId) are actual phase results, never caller authority or progress counters.
Order: (1) the parent reloads the fresh persisted administrator, field/resource policy, the active already-configured consumer/generation/checkpoint and the partition identity, validates expiry/cancellation/buffers and resolves the original outcome through a separately signed private capability before new work (a changed same-ID request conflicts, and a later generation retirement never erases an original completed result). (2) Capture of metadata plus the actual snapshot pin happens in the canonical gate through a callback-based disposable copied-byte pin; the charged bytes are traversed off gate while update/delete commands complete, within the bounded sessions, generations, files, bytes and leases. (3) A distinct generation is seeded, ordered retained projection pages are read to the observed finite upper, and intents are applied and the stable-ID projection acknowledgements committed through the connection grain; an empty final batch executes the empty-effects acknowledgement to obtain an actual receipt, and nothing is fabricated. (4) A fresh final validation cut validates complete records, revisions, token digest, inventory and the current policy/consumer/partition identity/retention; the Core factory rechecks the prepared witness and principal before sealing 35, and the expected applied/outbox/data epoch/read generation/policy/schema/resource/consumer/identity/current-pointer values are verified again in the ordered AtomicCommandCommit transaction (an intervening canonical commit is a refusal). (5) The original Result, typed publication authority, canonical current pointer and apply watermark are committed together. A successful persisted receipt requires every configured copy to confirm durable apply and reports its actual PersistedCopies. PublishedCut is the validated document cut before this metadata publication. A failed CAS or cancellation preserves the prior selector, and an uncertain submission stays UnknownWriteOutcome until canonical resolution. (6) Only the committed pointer and outcome authorize the local catalog publication: encode the SHA256 envelope, CreateNew pending with WriteThrough/Flush(true), validate identity/manifest/owner/files, rename in the same directory and replace the runtime pointer; old files are retained for leased readers and deleted only after the last old lease joins.
Every copy applies the signed 35 payload through its node-local Store.Commit, and a copy may have another local NodeId, journal position or read generation (for example after a snapshot installation). Replicated publication therefore validates deterministic canonical expectations and the sealed prepared witness, never the capturing node's local values, while the catalog keeps the exact captured local fence. The checkpoint intent that produced the acknowledged ProjectionBatchResult is retained in the prepared witness, its stable command fingerprint is verified against the scoped stored outcome before signing, and followers compare the replicated checkpoint outcome.
TextOnlineCatalog (alias keyload.server.text.online-catalog, root text-online, catalog.bin / catalog.pending): Id0 FormatVersion, Id1 NodeId, Id2 Scope (TextProjectionScope), Id3 the partition identity, Id4 Consumer, Id5 ConsumerGeneration, Id6 Leaf, Id7 AppliedPosition, Id8 ThroughSequence, Id9 ManifestSha256, Id10 BuildCommandId (the canonical 35 command identity). Before publication the staged index is closed and flushed, the complete manifest, record digest map and file inventory are validated, the pending catalog is read back and compared, the authenticated canonical pointer is re-read under the exclusive publication ownership (a changed pointer refuses), and the fresh authority/retention and cut are checked in the canonical gate. Failure before rename preserves the old selector; an ambiguous failure after rename is neither success nor no-effect. Startup validates the actual current catalog against the canonical pointer and outcome, manifest, owner and inventory; it never promotes pending, scans for the newest leaf or falls back. A committed 35 whose catalog is missing or pending authorizes reconstruction only from that exact recorded generation after full validation; missing or corrupt committed bytes refuse while preserving the outcome and prior ownership, and without an authoritative committed 35 the original current generation is retained and only proven uncommitted staging is reclaimed. Timeout, an unavailable required copy or a lagging local absence cannot prove that nothing was published. text-online/<generation>/staging.bin (keyload.server.text.online-staging, checksummed, CreateNew/WriteThrough/Flush(true)/readback after the owner receipt and before seed writes) holds the format, owner NodeId, canonical leaf, complete immutable request, persisted principal, the canonical OnlineTextParentIdentity fingerprint, the expected current command, the original captured source cut and the observed DataEpoch; no expiry or absent outcome authorizes its deletion and cold lookup never promotes it.
Cold reconciliation keeps the original canonical outcome, pointer, manifest and source-cut bytes. It verifies the committed authority, the original manifest checksum and the inventory, captures a fresh same-owner pin with matching incarnation/data epoch/policy/schema/resource/partition identity and compares the complete corpus with the original manifest at the current outbox horizon; only legitimate read-generation advancement and nondecreasing applied positions may rebind the disposable catalog, and a changed corpus or authority refuses reuse until a genuine new authorized maintenance or exact-image repair. A superseded completed command returns its exact retained original result without reactivating retired files. Online checkpoint child IDs use the alias keyload.orleans.search.online-text-checkpoint-identity (Id0 original CommandId, Id1 page ThroughSequence; the Guid is the first 16 bytes of SHA256 over that serialization), and every page is read through the authorized projection API, committed with empty effects and verified against the stored outcome and the configured consumer checkpoint. The retained seed pin belongs to its session, which uses the earlier of the original signed expiry and the configured QueryDeadlineSeconds linked to shutdown, with one cumulative never-renewed ReadExecutionBudget; active capabilities use ReadExecutionBudget.EnterStageCancellation(callerToken), and parent failure joins Abort while the 35 completion/reconciliation ownership survives. The original coordinator completion is retained through caller cancellation: a cancelled wait may report UnknownWriteOutcome but neither settles the task nor permits catalog or staging release, and even a terminal UnknownWriteOutcome needs authenticated canonical reconciliation before retirement or promotion.
An implicit online read uses Query's internal ICurrentTextProjection with the actual WithQueryView view, persisted principal/resource, original SearchRequest and budget; explicit selection stays first. It validates the canonical online-text-current family (exact keys, full authority; only the matching local node, persisted principal, collection/field and active administrator-configured consumer is eligible, two eligible consumers are ambiguous and refuse, and corrupt eligible authority never falls back), the checksummed catalog, the original manifest and the full canonical corpus in the same view, and transfers ownership from the inspection pin to the reader lease. A missing or mismatched catalog refuses the read (the read never writes or promotes pending). Fresh ResolveOriginal uses a separate fresh budget, joins any retained original coordinator work and repairs the disposable catalog before returning the immutable original result; a retained pending file is removed only after checksum, shape and identity validation against a fully proven current publication.
One internal TextResourceOwnership, created at the text composition boundary of the node-local host, is passed to the bootstrap (text-projections), explicit (search-indexes) and online (text-online) owners. It uses only the centrally validated TextExecutionOptions ceilings, counts shared physical entries once, admits reservations before work and holds them until actual joined disposal or retirement; every root keeps its own node-owned receipt. Prospective disk admission reserves the canonical path and target file length of every admitted file mutation before the write begins and counts only growth beyond the observed length, metadata envelope publication admits its exact encoded length before CreateNew, and open file groups (linked handles of one canonical path) count once, detach on delete, move on replace and keep their length until the real stream disposal joins. Closing handles first join every write, then capture the final length under the short gate, and close outside the gate; whole-index eviction and index disposal never run under the shared reservation lock. Seed traversal charges the same original grant for key/value examination, the retained decoded representation and the collection arrays: a VisitPrefix overload takes an admission callback invoked after each examined record and before owned copies, a refusal keeps the real attempted observation without admitting the allocation, and a capture failure whose cleanup settled completes the untouched grant while an aggregate (uncertain cleanup) failure retains it.
REQ/AC-LEASE-001..004, REQ/AC-ONLINE-001..005 and REQ/AC-CUT-001..004 apply to a public search overlapping publication. Query SearchCapturedExecution validates the request inside DatabaseEngine.WithQueryView; Server TextCapturedReadFactory acquires the same selected text reader and canonical ZoneTree snapshot under that gate, carrying only observed scalar identity (SearchReadAuthority; no signing key, live view or role cache). StorageRecovery's IScopedReadCapture charges work before copies and captures the bounded target document prefix, exact resource metadata and, for hybrid search, the vector/lineage/effect/source dependency closure (an unadmitted lookup refuses without a live-store fallback). The live gate is released before off-gate closure work, the snapshot is joined and unused reservation settled, and the same independent FTS reader is retained through posting visitation, ranking and projection. Before exposing a result SearchTerminalPrivacy reacquires the persisted principal/resource/partition identity and validates the original owner/incarnation/format/read-generation/policy/schema/resource identity: committed position advancement is allowed, while changed authority, expiry, revocation or current row privacy refuses the entire result. A revoked publisher's repair advances the policy epoch and needs a genuine new authorized 35 maintenance. The appended closed phase TextOriginalPostingRead reports a genuine posting Next while the lease is still owned; the reader, task and callback join before data is released. ConnectionReadCapabilities arms a call-local posting observation only around Query.ExecuteAsync.
Unit tests (TextOnlineWholeOperationTests, TextOnlineSeedBudgetTests, TextOnlineSeedReader) run real capture, build, query, ordered replay and publication while public update and delete complete off gate, with full independent literals (original, new, deleted, unchanged payload/reference/revision/score/order/page), a pinned old reader across replacement, real cancellation/bounds/stale CAS/spoofed public 35/authority refusals each followed by a fresh healthy operation, the full original result after later swaps and a same-owner cold continuation. TextOnlineProcessRecoveryTests kills a real RecoveryProcess at the completed 35 and at the pending-catalog flush, reopens the same root, verifies the original receipt, authority and bilingual results, rejects checksum corruption, restores the original bytes and publishes healthily again. TextOnlineClusterTests and TextCapturedClusterTests execute the complete flow through the .NET SDK, official MCP, Q1 SDK and Q1 MCP on the Aspire replicated cluster (ownership refusal, healthy publication/search, update/delete, retained public reader overlap with completion or cancellation, full original replay and a same-owner cold restart). Rollout enables operations 34/35 only on capable nodes; after a 35 is written, rollback keeps decoding support, disables new online admission and drains joined ownership before removing only verified disposable derived files. Canonical documents, journals, outcomes and consumer records stay authoritative.