docs(concurrency): add concurrency_policy.md + decompose daemon lib.rs (Phase 10g)#308
Merged
Merged
Conversation
…y rustdoc
Phase 10g closes the Phase 10 audit chain with a workspace concurrency
contract doc and per-crate rustdoc sections summarizing each crate's
runtime model.
## What lands
* `docs/architecture/code-quality/concurrency_policy.md` — 7th
companion to the existing 6 code-quality policy docs (panic,
allocation, trait, dependency, build/codegen, lint-posture).
Codifies the five concurrency dimensions:
- Task ownership (T1/T2/T3 with required four-facet annotation).
- Lock discipline (L1-L5 patterns; L6 lock-across-await
forbidden; enforced by three `await_holding_*` clippy lints at
deny).
- Channel discipline (C1-C5; C6 undocumented-unbounded forbidden).
- Timeout policy (W1-W4; W5 unbounded cross-process await
forbidden).
- Blocking-IO rule (B1-B4; B5 unbounded sync I/O on async runtime
worker forbidden).
Plus shutdown coordination, required annotation templates,
per-crate posture matrix, verification steps, anti-patterns, and
the Phase 10 audit trail cross-linking PRs #303-#307 and the local
per-dimension audit docs in `docs/dev/baseline/2026-05-19/`.
* `crates/uffs-daemon/src/lib.rs` `# Concurrency` rustdoc — runtime
model + the six named `spawn_*` constructors that form the
daemon's startup graph + cross-link to the policy doc.
* `crates/uffs-mcp/src/lib.rs` `# Concurrency` rustdoc — stdio
dispatcher / streamable-http axum server / daemon-bridge
reader-loop ownership + reload-pipeline-is-CLI-one-shot note.
* `crates/uffs-client/src/lib.rs` `# Concurrency` rustdoc — hybrid
async + sync runtime model + the deliberate 300 s async vs 60 s
env-overridable sync timeout asymmetry documented in
`phase_10_timeout_coverage_audit.md`.
* `crates/uffs-mft/src/lib.rs` `# Concurrency` rustdoc — predominantly
sync library + CLI binary; tokio only for daemon-embedded
`spawn_blocking` MFT reads.
* `CONTRIBUTING.md` `## Concurrency policy` section — quotes the
one-line rule, lists the five dimensions with their taxonomies,
and cross-links the policy doc.
## Rule-1 adherence
* Zero `#[allow(...)]` introductions.
* No suppression hacks, no skipped tests.
* Doc-only PR — no behavior change.
* `cargo clippy --workspace --all-targets -- -D warnings` — clean.
## Follow-up
A separate commit on this branch decomposes
`crates/uffs-daemon/src/lib.rs` (currently 1066 LOC, over the 800-LOC
file-size policy ceiling) into `tracing.rs`, `startup.rs`, and
`shutdown.rs` sibling modules, keeping the spawn_* cluster cohesive
per the existing rationale comment.
Refs #302 (Phase 10 umbrella).
…ling modules
`crates/uffs-daemon/src/lib.rs` was 1066 LOC (266 over the 800-LOC
file-size policy ceiling). The existing rationale comment argued
against splitting the `spawn_*` cluster because doing so would
fragment the parent-task lifetime relationships — and that argument
still holds. But the orchestrator's *setup* and *terminal* phases
have no such coupling and can move out cleanly.
## Extractions
* `crates/uffs-daemon/src/log_init.rs` (105 LOC, new) — `init_tracing`
+ `default_log_file`. No collision with the `tracing` crate
because the module is named `log_init`. `pub use
log_init::init_tracing;` re-export preserves the public API
exactly (`uffs_daemon::init_tracing` still works for the daemon
binary + the embedded `uffs daemon run` subcommand).
* `crates/uffs-daemon/src/startup.rs` (238 LOC, new) — pre-spawn
helpers:
- `install_catastrophe_panic_hook`
- `log_daemon_starting` + `emit_daemon_starting_event`
- `load_daemon_config`
- `bootstrap_lifecycle_manager`
- `gather_mft_files` + `drive_letter_matches` + `resolve_drive_list`
(windows + non-windows variants)
- `validate_data_sources`
All `pub(crate)`. `drive_letter_matches` is `pub(crate)` (was
private) because the existing regression-pin test in
`crates/uffs-daemon/src/tests.rs` exercises it directly — the
contract pin was preserved by updating the import to
`super::startup::drive_letter_matches;`.
* `crates/uffs-daemon/src/shutdown.rs` (84 LOC, new) —
`await_shutdown_then_force_exit` + `force_exit_with_watchdog`.
All `pub(crate)`.
## lib.rs after
* **689 LOC** (was 1066) — under the 800-LOC ceiling without an
exception.
* Keeps the `spawn_*` cluster cohesive:
`spawn_load_task` / `spawn_ipc_servers` /
`spawn_stats_heartbeat` / `spawn_idle_demote_controller` /
`spawn_journal_loops_for_warm_shards` /
`spawn_pressure_subscriber`.
* Keeps `DaemonConfig` (public struct) and `run_daemon` (public
orchestrator) for binary + embedded use.
* Removed the obsolete `Exception: file_size_policy allows this
file to exceed 800 LOC` rationale comment from the crate-root
rustdoc.
## file_size_exceptions.txt
Removed the now-obsolete `crates/uffs-daemon/src/lib.rs|PERMANENT:
…` entry. Local `bash scripts/ci/check_file_size_policy.sh` passes
clean.
## Rule-1 adherence
* Zero `#[allow(...)]` introductions.
* No suppression hacks, no skipped tests.
* No public API change — `uffs_daemon::{init_tracing, run_daemon,
DaemonConfig}` resolve to the same items as before via the
`pub use log_init::init_tracing;` re-export.
* `cargo fmt --all` — clean.
* `cargo clippy --workspace --all-targets -- -D warnings` — clean.
* `cargo test -p uffs-daemon --lib` — **298 passed / 0 failed**.
* `bash scripts/ci/check_file_size_policy.sh` — clean.
Refs #302 (Phase 10 umbrella).
…ed clippy entries + audit-script speedup
Closes the 4 gaps surfaced by the Phase 10 fidelity audit against the
playbook §1142-1146 pass criteria + plan §2 acceptance criteria:
## AC3 — `§0 The model at a glance` (was missing)
Adds the playbook's headline pass criterion ("concurrency model can
be explained on one page") to `concurrency_policy.md`:
* **Daemon task graph** — Mermaid diagram with 11 nodes covering
the 6 top-level `spawn_*` constructors + 2 subsystem-internal
spawns (per-shard journal loop + `RegistryPatchSink`) +
per-connection `handle_connection` + writer/notification
sub-tasks.
* **Shard-state lifecycle table** — 6 states (`Unknown`, `Cold`,
`Parked`, `Warm`, `Hot`, `Evicting`) with bloom/trie/body
presence + entry triggers; legal transitions cross-link
`ShardState::can_transition_to`. Two demote drivers explained:
idle-TTL (`spawn_idle_demote_controller`, 30 s cadence) +
memory-pressure cascade (`spawn_pressure_subscriber`, no-op on
Mac/Linux).
* **IPC-request lifecycle** — 4 numbered steps: accept →
per-connection task → reader/writer/notifier → per-RPC
timeout (search 30 s, drive-load
`IndexManager::DRIVE_LOAD_TIMEOUT`, refresh fire-and-forget).
* **Shutdown sequence** — 5 numbered steps: signal source → IPC
drain → load drain (3 s timeout) → PID + socket cleanup →
force-exit watchdog (5 s) + `process::exit(0)`.
Word count: ~520 prose words + 1 Mermaid + 1 sub-table — under the
600-word budget per plan §3 risk register.
## AC2b + AC6 — `§6 Spawn-site registry` (was missing)
Adds the workspace-tracked enumeration of all 18 prod
`tokio::spawn(` sites with the four required facets (owner /
shutdown / errors / cancel) — closes plan AC#2's "full inventory in
`concurrency_policy.md §"Spawn-site registry"`" + AC#6's "all 7
required sections".
The table is keyed by group (A = top-level, B = subsystem long-lived,
C = per-connection, D = sub-task, E = one-shot, F = runtime-cleanup
/ external) and references the source-of-truth hand-audit at
`docs/dev/baseline/2026-05-19/phase_10_task_ownership_inventory.md`
(local-only) for per-site nuance. Adding a new prod spawn site now
requires a matching row in the same PR — the audit-script `§2`
count is the gate; new spawn with no registry row fails review.
Renumbering: §6 (was Verification) → §7, §7 → §8, §8 → §9 to
preserve narrative flow (policy → posture → registry → verification
→ anti-patterns → audit trail).
## G1 — Named `await_holding_*` clippy entries
Pins all 3 lints from clippy's `suspicious` group at deny in the
workspace `Cargo.toml`:
```toml
await_holding_lock = "deny"
await_holding_refcell_ref = "deny"
await_holding_invalid_type = "deny"
```
Effectively no behavior change — they were at warn-by-default,
promoted to error via the workspace's CI `-D warnings` flag. The
named entries make the policy doc's enforcement claim literal and
survive any future tightening of the `--deny warnings` shape. An
inline comment block at the entries documents the rationale +
cross-links `concurrency_policy.md §1`.
Also fixes a leaked absolute path in the policy doc (line 36 was
`@/Users/rnio/Private/Github/UltraFastFileSearch/Cargo.toml` — now
just `Cargo.toml`).
## AC5 — Audit script `< 5 s` (was 6.2 s)
Adds `bulk_per_crate` helper to `scripts/dev/concurrency_audit.sh`
that runs **one** `rg` invocation per pattern across all of
`crates/` and buckets per-file counts into per-crate totals via
`awk` + a bash nameref (`local -n`). Replaces the prior N-crates ×
M-patterns nested loop (~126 rg invocations for §1 alone, ~1.5 s of
process-spawn overhead).
Wall-time impact: **6.2 s → 4.5 s** (verified locally). Output is
byte-identical (modulo the `Captured:` ISO timestamp).
Requires bash 4.3+ for nameref — script shebang is `#!/usr/bin/env
bash`; macOS `brew bash` ships 5.x; CI runners are Ubuntu 22.04+
with bash 5.x by default.
## Audit trail (§9) update
Updates `concurrency_policy.md §9` to mention 10g's bonus
deliverables (daemon `lib.rs` decomposition, §0 model, §6 registry,
named clippy entries) + adds 10h row pointing at the local-only
final report.
## Final report (G3, local-only)
Adds `docs/dev/baseline/2026-05-20/phase_10_final_report.md` —
mirrors `phase_9_final_report.md` pattern. ~290 LOC covering:
* §0 Executive summary
* §1 Per-dimension outcome table (5 dimensions + cross-cutting
cancellation infra + Arc<Mutex<…>> nesting depth)
* §2 Concrete change summary (10 numbered items across the 6 PRs)
* §3 Acceptance scorecard (8 of 8 ACs + 3 of 3 playbook P-criteria
green)
* §4 Test coverage audit (G2 verdict — no real gap; existing
298 unit + 4 integration + 13 journal-loop tests cover every
spawn-site cancellation path; soak/stress deferred to Phase 12
per plan §0.3)
* §5 Risk-register outcomes (6 of 6 mitigations consumed without
incident)
* §6 Cross-references
* §7 Verification at closing SHA
* §8 Decisions log (10 entries 10a-10h)
* §9 Phase 10 closure with #302 close note
Local-only (gitignored under `docs/dev/baseline/`); mirrors the
Phase 0-9 final-report cadence.
## Rule-1 adherence
* Zero `#[allow(...)]` introductions.
* No suppression hacks, no skipped tests, no lint disables — the
3 new clippy entries are *deny*, not allow.
* No public API change.
* `cargo fmt --all` — clean.
* `cargo clippy --workspace --all-targets -- -D warnings` —
clean (now also enforcing the 3 named `await_holding_*` lints).
* `cargo test -p uffs-daemon --lib` — 298 passed / 0 failed.
* `RUSTDOCFLAGS="-D warnings -D rustdoc::broken-intra-doc-links"
cargo doc --workspace --no-deps` — clean.
* `bash scripts/dev/concurrency_audit.sh` — runs in 4.5 s,
output byte-identical to pre-optimization.
Refs #302 (Phase 10 umbrella).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Phase 10g + 10h close the Phase 10 audit chain (#302) with three atomic commits:
docs(concurrency)— workspace concurrency policy doc + per-crate# Concurrencyrustdoc + CONTRIBUTING cross-link.refactor(daemon)— decomposeuffs-daemon/src/lib.rs(1066 LOC → 689 LOC) intolog_init/startup/shutdownsibling modules; remove the now-obsoletefile_size_exceptions.txtentry.docs(concurrency)— closeout:§0 The model at a glance(the playbook §1142-1146 headline pass criterion),§6 Spawn-site registry(the 18-prod-site inventory), 3 namedawait_holding_*clippy entries pinned at deny, and abulk_per_cratehelper that bringsconcurrency_audit.shunder the < 5 s acceptance bar.Commit 1 —
docs(concurrency)Creates
docs/architecture/code-quality/concurrency_policy.md— the 7th companion to the existing 6 code-quality policy docs (panic, allocation, trait, dependency, build/codegen, lint-posture).Codifies the five concurrency dimensions:
await_holding_*clippy lints at deny).Plus shutdown coordination, required annotation templates, per-crate posture matrix, verification steps, anti-patterns, and the Phase 10 audit trail cross-linking PRs #303-#307 and the local per-dimension audit docs in
docs/dev/baseline/2026-05-19/.Per-crate
# Concurrencyrustdoc sections added to:uffs-daemon/src/lib.rs— runtime model + the six namedspawn_*constructors that form the daemon's startup graph.uffs-mcp/src/lib.rs— stdio dispatcher / streamable-http axum server / daemon-bridge reader-loop + reload-pipeline-is-CLI-one-shot note.uffs-client/src/lib.rs— hybrid async + sync runtime model + the deliberate 300 s async vs 60 s env-overridable sync timeout asymmetry documented inphase_10_timeout_coverage_audit.md.uffs-mft/src/lib.rs— predominantly sync library + CLI binary; tokio only for daemon-embeddedspawn_blockingMFT reads.CONTRIBUTING.mdgets a## Concurrency policysection quoting the one-line rule, listing the five dimensions with their taxonomies, and cross-linking the policy doc.Commit 2 —
refactor(daemon)crates/uffs-daemon/src/lib.rswas 1066 LOC (266 over the 800-LOC file-size policy ceiling). The existing rationale comment argued against splitting thespawn_*cluster because doing so would fragment the parent-task lifetime relationships — and that argument still holds. But the orchestrator's setup and terminal phases have no such coupling and can move out cleanly.Three extractions:
log_init.rs(105 LOC, new) —init_tracing+default_log_file. No collision with thetracingcrate because the module is namedlog_init.pub use log_init::init_tracing;re-export preserves the public API exactly.startup.rs(238 LOC, new) — pre-spawn helpers:install_catastrophe_panic_hook,log_daemon_starting,emit_daemon_starting_event,load_daemon_config,bootstrap_lifecycle_manager,gather_mft_files,drive_letter_matches(+ windows / non-windows variants ofresolve_drive_list),validate_data_sources. Allpub(crate).drive_letter_matchesupgraded topub(crate)(was private) so the existing regression-pin test intests.rscan still exercise it directly — preserved by updating the import tosuper::startup::drive_letter_matches;.shutdown.rs(84 LOC, new) —await_shutdown_then_force_exit+force_exit_with_watchdog. Allpub(crate).lib.rsafter:spawn_*cluster cohesive.DaemonConfig(public struct) andrun_daemon(public orchestrator) for binary + embedded use.Exception: file_size_policy allows this file to exceed 800 LOCrationale comment from the crate-root rustdoc.scripts/ci/file_size_exceptions.txt— removed the now-obsoletecrates/uffs-daemon/src/lib.rs|PERMANENT: …entry.Commit 3 —
docs(concurrency)closeoutCloses the 4 gaps surfaced by the Phase 10 fidelity audit against the playbook §1142-1146 pass criteria + plan §2 acceptance criteria.
§0 The model at a glance(AC3 + playbook P3)Adds the headline pass criterion ("concurrency model can be explained on one page") to
concurrency_policy.md:spawn_*constructors + 2 subsystem-internal spawns + per-connectionhandle_connection+ writer/notification sub-tasks.Unknown/Cold/Parked/Warm/Hot/Evicting) with bloom/trie/body presence + entry triggers; cross-linksShardState::can_transition_to. Documents the two demote drivers (idle-TTL + memory-pressure cascade) and the Mac/Linux platform-no-op note.process::exit(0).Word count: ~520 prose words + 1 Mermaid + 1 sub-table — under the 600-word budget per plan §3 risk register.
§6 Spawn-site registry(AC2 + AC6)Adds the workspace-tracked enumeration of all 18 prod
tokio::spawn(sites with the four required facets (owner / shutdown / errors / cancel). The table is keyed by group (A = top-level, B = subsystem long-lived, C = per-connection, D = sub-task, E = one-shot, F = runtime-cleanup / external) and references the source-of-truth hand-audit atdocs/dev/baseline/2026-05-19/phase_10_task_ownership_inventory.md(local-only).Adding a new prod spawn site now requires a matching row in the same PR — the audit-script
§2count is the gate; new spawn with no registry row fails review.Renumbering: §6 (was Verification) → §7, §7 → §8, §8 → §9 to preserve narrative flow (policy → posture → registry → verification → anti-patterns → audit trail). No external references to the old numbers exist (verified via workspace grep).
Named
await_holding_*clippy entriesPins all 3 lints from clippy's
suspiciousgroup at deny in the workspaceCargo.toml:Effectively no behavior change — they were at warn-by-default, promoted to error via the workspace's CI
-D warningsflag. The named entries make the policy doc's enforcement claim literal and survive any future tightening of the--deny warningsshape. An inline comment block at the entries cross-linksconcurrency_policy.md §1.Also fixes a leaked absolute path in the policy doc (line 36 was
@/Users/rnio/Private/Github/UltraFastFileSearch/Cargo.toml— now justCargo.toml).concurrency_audit.sh< 5 s(AC5)Adds
bulk_per_cratehelper that runs onerginvocation per pattern across all ofcrates/and buckets per-file counts into per-crate totals viaawk+ a bash nameref (local -n). Replaces the prior N-crates × M-patterns nested loop (~126 rg invocations for §1 alone, ~1.5 s of process-spawn overhead).Wall-time impact: 6.24 s → 4.54 s (verified locally). Output is byte-identical (modulo the
Captured:ISO timestamp).Requires bash 4.3+ for nameref — script shebang is
#!/usr/bin/env bash; macOSbrew bashships 5.x; CI runners are Ubuntu 22.04+ with bash 5.x by default.Phase 10 final report (G3, local-only)
Adds
docs/dev/baseline/2026-05-20/phase_10_final_report.md— mirrorsphase_9_final_report.mdcadence (gitignored underdocs/dev/baseline/). ~290 LOC covering executive summary, per-dimension outcome table, 10-item change summary, 8-of-8 acceptance scorecard, G2 test-coverage audit (verdict: no real gap; soak/stress deferred to Phase 12 per plan §0.3), risk-register outcomes (6 of 6 mitigations consumed without incident), cross-references, verification, 10-entry decisions log, and Phase 10 closure note.G2 test-coverage audit — verdict
Audited the existing test corpus against the playbook §1134-1140 testing dimensions. No gap found. Existing coverage:
cache/journal_loop/tests/basics.rs::cancel_exits_within_convergence_deadline+ 12 other journal-loop tests usinghandle.cancel()+ bounded-timeout assertions.shutdownRPC +shutdown_daemonhelper.pressure_subscriber_drains_warm_cascade_on_low_and_no_ops_on_high(A6),idle_demote_tracing(A5),wait_for_in_flight_clear(F1).No test additions required.
Rule-1 adherence
#[allow(...)]introductions.uffs_daemon::{init_tracing, run_daemon, DaemonConfig}resolve to the same items as before via thepub use log_init::init_tracing;re-export.[...]intra-doc links) socargo doc -D rustdoc::broken-intra-doc-linksstays clean on the public surface.Verification
Local (at closing commit):
cargo fmt --all— clean.cargo clippy --workspace --all-targets -- -D warnings— clean (now also enforcing 3 namedawait_holding_*lints at deny).cargo test -p uffs-daemon --lib— 298 passed / 0 failed.cargo test -p uffs-daemon(full incl. integration + doctests) — clean (4 ignored require pre-built daemon binary; 2 doctests ignored by design).bash scripts/ci/check_file_size_policy.sh— clean.RUSTDOCFLAGS="-D warnings -D rustdoc::broken-intra-doc-links" cargo doc --workspace --no-deps— clean.bash scripts/dev/concurrency_audit.sh > /tmp/audit.md— 4.54 s (was 6.24 s pre-optimization), 321-line Markdown output.lint-fast— all 7 stages passed (file-size / typos / reuse / taplo / ...).lint-pre-push— all 17 stages passed in 132 s (includes rustdoc, doctests, tests, smoke, lint-ci-windows).Acceptance scorecard
All 8 of 8 plan §2 ACs + 3 of 3 playbook §1142-1146 P-criteria green:
concurrency_policy.md §6enumerates 18 prod sitesconcurrency_policy.md §0(~520 prose words + 1 Mermaid + 1 sub-table)concurrency_audit.sh < 5 sbulk_per_cratehelper)CONTRIBUTING.mdcross-link## Concurrency policysection (commit 1)# Concurrencyrustdocuffs-daemon+uffs-mft+uffs-mcp+uffs-clientlib roots (commit 1)Tracking
Refs #302 (Phase 10 umbrella) — closes on merge.
docs/dev/baseline/2026-05-20/phase_10_final_report.md) lives local-only per thedocs/dev/baseline/gitignore convention; SHA + close note will be linked from the umbrella issue at merge.