fsmonitor: recover status history across mixed writers - #47
Open
ttaylorr-oai wants to merge 14 commits into
Open
fsmonitor: recover status history across mixed writers#47ttaylorr-oai wants to merge 14 commits into
ttaylorr-oai wants to merge 14 commits into
Conversation
A full worktree-attribute manifest refresh probes one potential .gitattributes source per tracked directory. On the OpenAI monorepo that means approximately 237,000 candidates, almost all absent. Allocate candidate state in one contiguous block, collect indexed attribute sources during the existing index walk, and avoid sorting when candidate order is already monotonic. On Darwin, revalidate a pinned parent with fstatat() instead of reopening and closing it. The complete anchored namespace identity remains unchanged; Linux keeps its existing openat2-based validation. Cover unusual path ordering, both object formats, indexed fallback, and replacement of an anchored parent with a symlink.
Losing a filesystem-monitor boundary currently discards every cached untracked directory, even when its prior state was authenticated by a paired full worktree proof. Rebuilding that cache enumerates roughly 237,000 directories in the OpenAI monorepo. Preserve a complete, previously authenticated untracked cache only when a successful provider reset leaves configuration, attributes, and conversion semantics unchanged. Demote the cache to ordinary directory validation, then reuse the existing parallel directory-stat and ignore file revalidation before closing the new provider token. Global invalidation, changed attributes, missing proofs, dirty caches, and ambiguous directory identities retain destructive fallback. Add coverage for nested untracked files, changed ignore rules, and provider global invalidation. On the real OpenAI checkout, full recovery improves from 30.79 seconds to 8.58 seconds; subsequent status calls retain a 31.86 ms median.
A full manifest rebuild on the OpenAI checkout enumerates roughly 237,000 possible .gitattributes paths. Allocating and freeing every candidate separately adds avoidable allocator traffic to an already expensive recovery path. Store candidate names in a stable memory pool while retaining the existing anchored lookup, sorting, deduplication, and worker-lifetime guarantees. Cover a newly appearing nested attribute source under SHA-256 so pooled storage cannot turn an earlier negative lookup into stale proof.
After provider history is lost, a pathspec previously disabled threaded untracked-cache revalidation. The scoped command still needed a globally closed provider token, so it instead validated the entire cache serially before filtering its output. Permit the existing ordinary whole-tree preload for this narrow authenticated recovery state, then retain the original pathspec filtering and provider closure. Allow bounded test sweeps through sixteen recovery workers while keeping the normal six-worker default. Exercise changed files outside the requested cone, changed nested ignore rules, global invalidation, and every supported worker count.
A clean-status certificate includes repository inputs such as the active locale. When those inputs changed, validation correctly rejected the existing certificate, but ordinary status saw a safely pinned sidecar and declined to replace it. Every subsequent command repeated the slower index-reading path. Remember the specific repository-input mismatch and allow ordinary clean status to reissue an otherwise safe, single-link sidecar after complete normal validation. Preserve the optional-locks boundary and leave the physical index untouched. Cover both successful locale-change repair and the read-only case.
Unstaging an ordinary newly added file invalidated the entire fsmonitor proof despite each removed entry already passing the semantic-safety check. Worse, an installed Git reset can remove FSMN, FSCF, and FSUC altogether while changing the index relative to the last authenticated external checkpoint. The next status then rebuilt every attribute candidate and refreshed the entire index. Preserve proofs for individually safe removals and recover a missing provider boundary from a fully authenticated checkpoint only when its old token yields a safe delta. Start with every current entry dirty, restore only matching checkpoint-clean entries, and force content checks for remaining untrusted entries so externally modified hardlinks cannot become false clean. Exercise the exact add, delete, checkout, reset, status sequence for both the built-in reset and an actual foreign Git. Reject global invalidations, changed attributes, unsafe logical changes, and same-timestamp hardlink mutations.
Advance the enrolled unstable topic without changing the exact tested metadata-recovery tree. Preserve the previously reviewed source history as the first parent and the complete six-commit prototype ending at fca61fd as the second parent.
The global invalidation fixture assumes its clean priming status physically writes the paired FSUC extension. On macOS, external history can satisfy the same query without rewriting the index, so that prerequisite intermittently fails before the intended global invalidation is exercised. Pin GIT_INDEX_FILE for the priming command, matching the existing neighboring fixtures and forcing deterministic physical proof materialization.
Advance the reviewed unstable topic with a one-line test-only correction to force physical FSUC materialization during priming. Preserve the exact six-commit fca61fd production tree and existing reviewed topic history; the only added delta is the deterministic regression-test fixture.
dreynaud-oai
approved these changes
Aug 13, 2026
dreynaud-oai
left a comment
dreynaud-oai
left a comment
There was a problem hiding this comment.
Independently reviewed exact head 046008d. Verified its only delta from the previously qualified production tree is one test-only fixture line, and reproduced the formerly failing provider-reset test successfully under SHA-1 and SHA-256. Real dirty macOS and Linux monorepo checkouts show matched-build steady-state performance parity, 836 adversarial macOS checks pass, and all eight Linux fsmonitor watch-failure cases pass.
A dry run can refresh fsmonitor state while reading the index, which makes the common exit path write an otherwise unchanged index. When configured filters prevent semantic history from being preserved, that write invalidates the clean-status proof and makes the next status rebuild worktree metadata. Roll back the index lock for dry runs instead. They must not update the index or run index-change hooks.
A configured clean or process filter does not make every index change semantically unsafe. In particular, a global Git LFS configuration was invalidating the authenticated worktree proof after adding or unstaging ordinary, unfiltered files. The following status then rebuilt the entire manifest and untracked cache. Preserve history when an already-authenticated filter scope exists and the changed path selects no active clean filter. Continue rejecting actual filtered paths, changed attribute sources, unsafe index shapes, and mismatched provider tokens. Cover ordinary staging, unstaging, dry runs, nested additions, and active-filter invalidation.
dreynaud-oai
approved these changes
Aug 13, 2026
dreynaud-oai
left a comment
dreynaud-oai
left a comment
There was a problem hiding this comment.
Independently verified exact dad9e3b on the real dirty 858,001-entry Linux monorepo with matching Git client and daemon. New-file add, existing staged-file add, unstage, and dry-run preserve coherent FSMN/FSUC/FSCF and reduce subsequent status from ~19 s to 0.91-1.25 s. Separate live-daemon SHA-1/SHA-256 gates verify unfiltered staging and dry-run preserve history while actual filtered paths fail closed; attribute, text-conversion, and clean-filter safety scenarios also pass.
The og wrapper enables bulk index preloading through command-scoped configuration, while direct Git invocations may not. Hashing those transient acceleration settings into the authenticated worktree proof made the two clients disagree about repository configuration. When an unused clean filter was configured, alternating clients repeatedly discarded the full metadata manifest and rebuilt it. Exclude only command-scoped core.preloadIndex and core.preloadIndexBulk from the proof digest. Continue hashing persistent preload settings, semantic conversion settings, tracked-file policy, and filters. Cover both hash algorithms, every persistent scope, alternating wrapper and direct invocations, and fail-closed filemode changes.
dreynaud-oai
approved these changes
Aug 13, 2026
dreynaud-oai
left a comment
dreynaud-oai
left a comment
There was a problem hiding this comment.
Independently approved exact f92dd57. Real dirty Linux monorepo (858,001 paths, actual system LFS filter): 21 plain/og/preload transitions, 0.95-0.99 s steady status, zero manifest scans and coherent proofs; add/re-add/commit dry-run/unstage remain 0.9-1.23 s. Real dirty macOS monorepo (1,161,336 paths, existing .658 daemon preserved): 21 transitions, 0.25-0.28 s steady status. Active clean-filter fail-closed staging passes 28 scenarios each for SHA-1 and SHA-256. Production source blob independently matched c224fe3876961513c813f18187e3e046fe37ec49; only exact command-scoped acceleration keys are excluded.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ship the six follow-up fsmonitor/status commits ending at
fca61fdceec3017dff2f5d4b945f6c154290dfddin the nextcodex-unstablerelease. The reviewed source is a fast-forward merge of the existing
enrolled topic and the exact tested prototype; its tree is byte-identical
to
fca61fdcee, and no protected workflow or controller files change.the index without
FSMN,FSCF, orFSUC. The reportedgit add x; rm x; git checkout -- x; git reset; git statussequenceremains bounded and reports
? xcorrectly.scoped status requests, without serially reopening the entire worktree.
reissue clean proofs when repository fingerprint inputs change.
mutations, and unauthenticated history.
Validation:
t7527: 156/156 with SHA-1 and SHA-256.t7519: 77/77 with SHA-1 and SHA-256.t7530: 47/47 with SHA-1 and SHA-256.t7063: 64/64 with SHA-1 and SHA-256.t7529: 36/36 with SHA-1 and SHA-256.provider-reset scoped recovery improved from 13.25 s to 5.21 s.