includeIgnored silently bypassed on older git (<2.34): git ls-files -s --recurse-submodules fails, falls back to filesystem walk that ignores the opt-in

[newshowardz777]

Summary

On systems with older git (verified failing on git 2.25.1 / Ubuntu 20.04; working on git 2.34.1+ / Ubuntu 22.04/24.04/26.04), codegraph index silently drops files that should be pulled in via codegraph.json 's includeIgnored . The user gets no error — just a smaller index. Same project, same codegraph.json , same codegraph version (1.5.0) indexes correctly on newer git.

Environment

• codegraph: @colbymchenry/codegraph 1.5.0
• OS: Ubuntu 20.04 LTS
• git: 2.25.1 (fails) — works on 2.34.1+ (Ubuntu 22.04/24.04/26.04)
• Node: 20.x
• engines in package.json only constrains Node ( >=20.0.0 <25.0.0 ); no minimum git version is declared.

Reproduction

Minimal project ( t1/ ):

On Ubuntu 20.04 / git 2.25.1:

On Ubuntu 22.04 / git 2.34.1 (same repo, same codegraph 1.5.0):

Root cause

Both call sites run git ls-files -z -s --recurse-submodules . On older git, --recurse-submodules only supports -c/--cached , not -s/--stage :
Call site 1 — collectGitFiles() (the indexing path, fatal )
No try/catch here → execFileSync throws (exit 128) → exception escapes → getGitVisibleFiles() catches it and return null → scanDirectory() falls back to scanDirectoryWalk() (the non-git filesystem walk). scanDirectoryWalk() only parses .gitignore and does not implement includeIgnored — that opt-in lives exclusively in the git path ( findIgnoredEmbeddedRepos() , called from collectGitFiles() ), which never runs. dir_b is gitignored → skipped → only a.c is indexed.

Net effect: on older git the entire includeIgnored mechanism is silently disabled, and because the failure is swallowed into a "non-git project" fallback, the user gets no diagnostic — just missing files.
Call site 2 — discoverEmbeddedRepoRoots() (the watcher scope, non-fatal but inconsistent )
This one is already wrapped in try/catch, so on older git the gitlink discovery pass is silently skipped (unexpanded gitlinks → not found). Non-fatal, but it means the watcher scope diverges from the indexer scope on older git: the indexer (once site 1 is fixed) recurses into gitlinks via collectGitFiles() , while the watcher's discoverEmbeddedRepoRoots() misses them.

Impact

Any project relying on includeIgnored to index embedded/gitignored repos is silently mis-indexed on older git. The super-repo-of-clones layout (the very use case includeIgnored exists for, per issues #622/#699/#514) is affected — and in particular embedded repos tracked as gitlinks (mode 160000, without .gitmodules ) are missed by the watcher on old git even after site 1 is fixed. No warning or error is surfaced in either case.

Suggested fix

Wrap both call sites in a try/catch and fall back to git ls-files -z -s (drop --recurse-submodules ) when the combined form is unsupported.

Site 1 — collectGitFiles() :

Site 2 — discoverEmbeddedRepoRoots() :
Why the -s fallback is safe

• -s keeps the \t format, so the existing 160000 gitlink detection still works.
• Without --recurse-submodules , active submodules surface as 160000 gitlinks and are recursed into by the existing gitlink pass at the end of collectGitFiles() ( classifyGitDir(...) === 'embedded' covers submodule checkouts). So active submodules are still indexed.
• Embedded independent repos (the dir_b case — untracked, not a gitlink) are still discovered by the untracked trailing-slash pass and findIgnoredEmbeddedRepos() , so includeIgnored works again. Verification
  Applied to the packaged lib/dist/extraction/index.js on Ubuntu 20.04 / git 2.25.1:

$ codegraph index .
◆  Indexed 2 files        # a.c + dir_b/b.c  ✓
$ codegraph files
├── dir_b
│   └── b.c (c, 2 symbols)
└── a.c (c, 2 symbols)

Behavior on newer git is unchanged (the try branch still runs the original command).

Alternatives considered

• Declare a minimum git version (e.g. ≥2.34) and error out clearly on older git. This avoids the silent-misindex UX but excludes Ubuntu 20.04 LTS users entirely; the graceful fallback above is strictly better for compatibility.
• Fall back to git ls-files -z -c --recurse-submodules (which old git supports). Rejected: -c omits mode bits, breaking 160000 gitlink detection, so the gitlink pass would stop finding unexpanded submodules. The -s fallback keeps mode bits.

[maxmilian]

Confirmed on main (44e1812), and the diagnosis holds exactly as written — collectGitFiles' git ls-files -z -s --recurse-submodules at src/extraction/index.ts:954 has no try/catch, the throw escapes to getGitVisibleFiles' catch { return null } at :1076, and scanDirectory then falls through to scanDirectoryWalk. Every one of the 39 includeIgnored references sits above line 1030, so the walk has no notion of the opt-in at all.

I'd like to take this one if nobody objects — you did the root-cause work and the patch shape here, so I'll credit you as co-author on the PR. If you'd rather send it yourself, say so and it's yours.

Two corrections to the report, both of which make it worse rather than better.

The version boundary is 2.36, not 2.34. I checked builtin/ls-files.c in git.git across tags rather than trusting the release notes:

tag	condition guarding the die
v2.34.1 (:720)	recurse_submodules && (show_stage || show_deleted || …) → dies
v2.35.0 (:728)	recurse_submodules && (show_stage || show_deleted || …) → dies
v2.36.0 (:728)	show_stage removed from the list → passes

-s was only permitted alongside --recurse-submodules in 2.36; the 2.35 docs still say "Currently there is only support for the --cached mode." The die() is also unconditional — it doesn't check whether the repo actually contains submodules, so every invocation fails on those versions regardless of the repo.

That widens the blast radius well past Ubuntu 20.04: Ubuntu 22.04 LTS ships git 2.34.1, Debian 11 ships 2.30.2, and RHEL/CentOS 8 is older still. Your note that it worked on 2.34.1 doesn't match the source — I'd guess that box had a backport or a PPA build.

And it isn't only includeIgnored that's lost. When getGitVisibleFiles returns null, the whole git-visible path goes with it — submodule expansion, gitlink recursion, and the codegraph.json include allowlist (collectIncludedFilesForRoot is inside the same try). On those distros CodeGraph silently indexes a different file set than the configuration asks for, with no error and no warning, and the only symptom is files quietly missing from query results.

I reproduced it without needing an old git, using a PATH shim that exits 128 with the real fatal: ls-files --recurse-submodules unsupported mode whenever it sees -s together with --recurse-submodules. Fixture is yours: a root repo, dir_b/ in .gitignore, codegraph.json with {"includeIgnored": ["dir_b/"]}, and dir_b carrying its own .git.

real git 2.55   → scanDirectory() = ["a.c", "dir_b/b.c"]
shim (< 2.36)   → scanDirectory() = ["a.c"]          ← silently dropped

Your suggested fix works: wrapping the call and retrying as git ls-files -z -s without --recurse-submodules restores both entries under the shim. Dropping the flag rather than -s is the right way round — the mode bits are what gitlink detection reads, so keeping -s preserves that, and submodule contents are already reached through the embedded-repo recursion.

The shim is what makes this testable in CI on any git version, so the PR will carry it as a regression test rather than leaving the path uncovered.

One thing I'm not claiming, in case it saves someone a search: #1567 (nested .gitignore, node_modules walked into, 63× file count) looks like the same fallback from the outside, but scanDirectoryWalk does apply buildDefaultIgnore — which covers node_modules — so I can't call it the same root cause. @milos85vasic, if you're still seeing that, git --version on the affected machine would settle whether it's related.

@colbymchenry — also worth noting that #1568 touches this exact catch block to add a logDebug. It doesn't fix the fallback, but it would make it visible, so the two are complementary rather than conflicting.

[newshowardz777]

@maxmilian
Sounds good, thank you. Please take it — I won’t send a separate PR

[maxmilian]

@newshowardz777 Thanks — PR is already up: #1604. It landed the same day as my comment above, so it predates your reply; sorry for not linking it sooner.

It takes the fallback approach for the -s --recurse-submodules failure, with a regression test that installs a PATH shim so the pre-2.36 die is reproducible on any git version, and the CHANGELOG entry credits you by name. It's rebased on current main as of today and the full suite is green.