odb: scan all sources' packfiles before loose objects - #975
Open
tyrielv wants to merge 1 commit into
Open
Conversation
The object database refactor that introduced per-source object stores (cb506a8 "odb: introduce \"files\" source" and the surrounding series, first released in v2.54.0) changed how do_oid_object_info_extended() searches for an object. It now iterates the sources and, within each source, consults that source's packfiles and then its loose object store before moving on to the next source. Before that series the search consulted every packfile -- across the primary object directory and all alternates -- before it looked at any loose object. The refactor reversed that for the multi-source case: for an object that lives in an alternate's packfile, the primary source's loose object store is now consulted first. That loose lookup is a filesystem stat(), and because callers such as cache_tree_fully_valid() pass ODB_HAS_OBJECT_RECHECK_PACKED (which clears OBJECT_INFO_QUICK) the cached-loose-index fast path is skipped and a real stat() runs for every such object. In a repository that keeps its objects in an alternate -- the common arrangement for VFS for Git and Scalar enlistments, where a shared object cache is mounted as an alternate -- this is a steep penalty. cache_tree_fully_valid() walks the whole cache tree and calls odb_has_object() for every node; on an enlistment with a ~2.4M-entry index that is ~380k objects, each incurring a wasted stat() on the primary loose store. A same-commit branch switch spent ~32s in cache_tree_fully_valid() (two calls of ~16s), observed in the field as a ~2x rise in median checkout duration after the client carrying the refactor rolled out. Restore the previous ordering without undoing the per-source encapsulation: when there is more than one source, scan the packfiles of every source first (OBJECT_INFO_SKIP_LOOSE) and only then consult each source's loose store (OBJECT_INFO_SKIP_PACKED). The single-source case is unchanged, so repositories without alternates keep the existing path. With the fix the same branch switch spends ~2s in cache_tree_fully_valid(), the ~380k wasted stat()s are gone, and performance matches versions predating the refactor. Assisted-by: Claude Opus 4.8 Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
tyrielv
marked this pull request as ready for review
August 7, 2026 22:18
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Since v2.55,
git checkout(and any branch-changing operation) isdramatically slower on large VFS for Git / Scalar enlistments. A
same-commit branch switch on a ~2.4M-entry index went from under a
second to ~36s. Fixes #974.
Root cause
The upstream per-source object database refactor (first released in
v2.54.0) changed object lookup from "scan all packfiles, then all loose
objects" to "per source: packed then loose." For an object that lives in
an alternate's packfile -- the normal arrangement for VFS for Git and
Scalar enlistments, where a shared object cache is mounted as an
alternate -- the primary source's loose object store is now consulted
first. That loose lookup is a filesystem
stat(), and because callerssuch as
cache_tree_fully_valid()passODB_HAS_OBJECT_RECHECK_PACKED(which clears
OBJECT_INFO_QUICK) the cached-loose-index fast path isskipped, so a real
stat()runs for every object.cache_tree_fully_valid()walks the whole cache tree and callsodb_has_object()for every node -- ~380k objects on this index -- eachincurring a wasted
stat(). Full analysis, the instrumented breakdown(
loose_lstats: 380,944,odb_misses: 0), and the alternativesconsidered are in #974.
Fix
Preserve the refactor's per-source encapsulation but restore the old
ordering: when there is more than one source, scan the packfiles of
every source first, then consult each source's loose store. Single-source
repositories are unaffected.
stat()scache_tree_fully_valid()Test
t5615adds a regression test: an object stored as a packed delta inan alternate and loose in the main object store.
%(deltabase)provesthe read resolves to the alternate's packfile (nonzero base) rather than
the loose copy (zero oid). It fails without this change and passes with
it.
t5613,t1006, andt0410remain green.Notes
This is a targeted mitigation for the microsoft/git fork so the
regression can be addressed quickly. The change keeps the per-source
encapsulation intact and only alters the search order when alternates are
present. A broader upstream discussion of the ordering may be worthwhile
separately.