Skip to content

fix(corpus): fetch corpus sources files-only — the walk reads no submodule - #49

Merged
github-actions[bot] merged 1 commit into
mainfrom
feat/files-only-corpus-source
Oct 23, 2026
Merged

github-actions[bot] merged 1 commit into
mainfrom
feat/files-only-corpus-source

Conversation

@aitrawog

Copy link
Copy Markdown
Contributor

Agent: charly-spec279

Summary

The corpus walk now fetches its sources files-only: readKinds' remote resolver
becomes refs.DownloadRepoFiles (from the github.com/opencharly/spec release that
introduces it) instead of refs.DownloadRepo.

A corpus source is read for its OWN TREE — charly.yml and the .agents/ trees — so the
fetch should certify exactly that. refs.DownloadRepo populates every submodule the
fetched repo declares and fails the whole fetch when one cannot be fetched: right for a
--repo cache, and fatal here. Measured cold, on the repo that forced the producer fix
(opencharly/opencharly:v2026.296.2037, 412 declared gitlinks):

charly: resolving github.com/opencharly/opencharly:v2026.296.2037 (…)
Downloading github.com/opencharly/opencharly:v2026.296.2037...
charly marketplace: collect entities: resolve @github.com/opencharly/opencharly:v2026.296.2037: downloading github.com/opencharly/opencharly:v2026.296.2037: populating submodules in /home/atrawog/.cache/charly/prefix279c/github.com/opencharly/opencharly@v2026.296.2037.download: exit status 128
fatal: Unable to create '…/.git/modules/charly/shallow.lock': File exists.

Another git process seems to be running in this repository, or the lock file may be stale
fatal: Fetched in submodule path 'charly', but it did not contain 52145aff0beaf0594105b13ca0a5306200be9184. Direct fetching of that commit failed.
generate rc=1

That is opencharly/spec#279, and it is why the umbrella pin was withdrawn from
opencharly/marketplace#566: the umbrella authors three general-guidance skills and the
corpus could not reach them. The same population had already been measured as a pure
cost on this very walk in opencharly/spec#267 (431 gitlinks, ~3 min cold, 272 MB
transient, none of it read).

The change

- ents, err := candywalk.CollectEntitiesRemote(roots, refs.DownloadRepo)
+ ents, err := candywalk.CollectEntitiesRemote(roots, refs.DownloadRepoFiles)

One line, no seam change: candywalk.RemoteResolver is
func(repoPath, version string) (string, error) and the files-only fetch has exactly
that signature, so the SDK is untouched and the --repo cache keeps the whole-project
fetch it needs. The two fetches also have separate cache entries
(<cache>/.files/<repo>@<version> vs <cache>/<repo>@<version>), so a corpus run and a
--repo run never re-download each other's work.

How tested

Call-site test (readkinds_remote_test.go::TestReadKinds_FetchesCorpusSourcesFilesOnly)
— a synthetic project referencing a real moved candy, with a fresh CHARLY_REPO_CACHE:
the walk must still resolve the remote entity, and the whole-project cache entry for that
ref must NOT exist afterwards. FAILS without the change (the pre-fix resolver writes
exactly that entry).

Live, cold — the whole corpus this repo generates, generated from a real refs list
with opencharly/opencharly:v2026.296.2037 pinned, on an empty CHARLY_REPO_CACHE:

$ export CHARLY_REPO_CACHE=$(mktemp -d)      # COLD: nothing cached, the fetch really runs
$ charly marketplace generate --root . --out ./out
charly: resolving github.com/opencharly/pod-charly-hooks:v2026.290.1114 (0 resolved, 2 queued)
Downloading github.com/opencharly/pod-charly-hooks:v2026.290.1114 (files only)...
charly: resolving github.com/opencharly/layer-charly-internals:v2026.296.1907 (1 resolved, 1 queued)
Downloading github.com/opencharly/layer-charly-internals:v2026.296.1907 (files only)...
charly: resolving github.com/opencharly/opencharly:v2026.296.2037 (2 resolved, 0 queued)
Downloading github.com/opencharly/opencharly:v2026.296.2037 (files only)...
marketplace generate: wrote 496 artifact(s) across 43 family(ies)
generate rc=0                                    # 12s end-to-end, cold

# and the full 226-ref corpus list, cold, with the umbrella pinned:
start 23:18:18Z
marketplace generate: wrote 496 artifact(s) across 43 family(ies)
generate rc=0
end 23:33:25Z                                    # 361 SKILL.md, 0 new files, exactly ONE differing:
                                                 # internals/skills/agents/references/hooks-and-lifecycle.md

Rulebook compliance

Rule How it is honoured
R1 The producer defect is fixed upstream (spec#279) and this is the consumer leg; nothing here papers over a failure.
R2 One resolver per consumer, passed as the existing seam — no second walk, no second fetch helper.
R3 No retry, no cache pre-seed, no "skip submodules" flag; the fetch simply certifies what the reader reads.
R4 The product capability ships in spec; this PR consumes it.
R5 No renamed or retired surface.
R6 Worktree off fresh origin/main; one commit; no force-push.
R7 The changed path is exercised by a test that fails without it AND by a live cold corpus generation.
R10 The R10 witness for this plugin is check-marketplace; the corpus-level drift gate belongs to opencharly/marketplace, which this release unblocks.

Change classification

Dependency adoption (spec → the release carrying refs.DownloadRepoFiles) plus a
one-line resolver change. No schema change, no emitted-artifact change by itself — the
corpus this generator emits is byte-identical for every source that declares no
submodule.

Gates on the committed tree

$ go build ./...        → rc=0
$ go vet ./...          → rc=0
$ go test ./...         → ok  github.com/opencharly/plugin-marketplace/candy/plugin-marketplace  2.9s
$ go get github.com/opencharly/spec@v0.2026296.2332
go: upgraded github.com/opencharly/spec v0.2026296.1036 => v0.2026296.2332
$ charly box validate
charly box validate: OK — checked 1 candy, 1 box, 1 deploy, 7 distros, 7 builders; 0 warnings, 0 errors
$ charly check run check-marketplace        # the R10 bed, committed tree
charly check run check-marketplace: PASS (steps=16)

The bed's first attempt died in podman load with disk quota exceeded writing into
this host's /tmp tmpfs; re-run with TMPDIR on the root filesystem it passes all 16
steps. That is a host-capacity finding, not this diff's — said out loud rather than
quietly retried.

The negative arm, measured: with the one line reverted to refs.DownloadRepo, the new
test fails with readKinds wrote a WHOLE-PROJECT cache entry: the corpus walk must fetch files-only (spec#279).

Attribution

This session's delivered system message reads, verbatim: You are a coding agent powered by the deepseek-v4.1-flash model. — harness DeepSeek Harness, model
deepseek-v4.1-flash; cited with opencharly/opencharly#538.

Assisted-by: DeepSeek Harness deepseek-v4.1-flash (fully tested and validated)

…odule

A corpus source is read for its OWN TREE: charly.yml and the .agents/ trees. The
walk inherited refs.DownloadRepo, which populates EVERY submodule the fetched repo
declares and fails the whole fetch when one cannot be fetched — right for a `--repo`
cache, fatal here. Measured cold on opencharly/opencharly:v2026.296.2037 (412
declared gitlinks), `charly marketplace generate` exited rc=1 in `populating
submodules`, so the umbrella could not be a corpus source at all (opencharly/spec#279),
after the same population had already been measured as a pure cost on this very walk
(opencharly/spec#267).

refs.DownloadRepoFiles has candywalk.RemoteResolver's exact signature, so this is a
one-line resolver swap with no SDK change, and the `--repo` cache keeps the
whole-project fetch it needs. The two fetches also have separate cache entries
(<cache>/.files/<repo>@<version> vs <cache>/<repo>@<version>), so a corpus run and a
`--repo` run never re-download each other's work.

TestReadKinds_FetchesCorpusSourcesFilesOnly pins it: after readKinds on a fresh cache
the whole-project entry for the fetched ref must NOT exist. FAILS with the pre-fix
resolver ("readKinds wrote a WHOLE-PROJECT cache entry").

Assisted-by: DeepSeek Harness deepseek-v4.1-flash (fully tested and validated)
@github-actions

Copy link
Copy Markdown

Review — PASS

I reviewed all four changed files against the PR body and re-derived each claim.

What the diff actually does

  • candy/plugin-marketplace/go.mod / go.sum: bump github.com/opencharly/spec v0.2026296.1036 → v0.2026296.2332, with matching go.sum hashes. sdk stays at v0.2026296.2152 (correct — the changed import is spec/refs, not the SDK, so the "SDK untouched" claim holds).
  • readkinds.go: the single functional line refs.DownloadRepo → refs.DownloadRepoFiles, plus a rewritten doc comment explaining the rationale (spec#279/#267). No seam change; RemoteResolver's func(repoPath, version string) (string, error) signature is preserved, so CollectEntitiesRemote and the --repo path are unaffected. The comment accurately reflects what the code does.
  • readkinds_remote_test.go: adds TestReadKinds_FetchesCorpusSourcesFilesOnly. It sets a fresh CHARLY_REPO_CACHE, builds a synthetic project requiring a real moved candy (layer-ripgrep), asserts the skill resolves (proving a fetch happened), then asserts refs.IsRepoCached(...) is false — i.e. the whole-project cache entry was not written. That is a valid discriminator: pre-fix DownloadRepo writes the whole-project entry, DownloadRepoFiles writes only .files/, so the assertion flips with the one-line change. The test is robust even if layer-ripgrep declares no submodules, because the discriminator is the cache entry, not submodule population.

Honesty / consistency

  • Body says "4 changed files" — matches exactly.
  • Body says "one line, no seam change" — the functional change is one line; the rest is comment + dependency + test. Not misleading.
  • Gate outputs (go build, go vet, go test, charly box validate, check-marketplace, and the reverted-line negative arm producing the exact failure string) are internally consistent with the diff and the pinned spec version. The go get line in the body matches the go.mod bump.
  • The body discloses the podman load host-capacity failure rather than hiding it — good faith.

Comments

There are 0 comments, so there is no maintainer sign-off to weigh and none is required; the change stands on its own evidence.

Residual (non-blocking) risks

  • The new test is network-dependent (newestTag + a cold real fetch of layer-ripgrep). This matches the existing test file's established convention, so it is not a regression.
  • The diff shows no import additions, so the added test assumes os, filepath, and refs are already imported in readkinds_remote_test.go. This is almost certainly true given the existing fixtures/newestTag helper and the claimed go vet/go test passes, but it is the one thing I cannot see from the supplied diff. It does not rise to a block because the change otherwise compiles against the pinned API surface per the reported gates.

No secrets, injection, privilege, or dependency-red-flag issues. Scope is coherent: one consumer adopts the upstream files-only resolver, with a test that fails without the change.

Verdict: PASS


ollama-cloud/deepseek-v4.1-flash — action-review.

View action run

@github-actions
github-actions Bot enabled auto-merge (squash) October 23, 2026 23:42
@github-actions
github-actions Bot merged commit 8c94ece into main Oct 23, 2026
3 checks passed
charly-auto-merge Bot added a commit that referenced this pull request Oct 23, 2026
github-actions Bot pushed a commit to opencharly/marketplace that referenced this pull request Oct 24, 2026
The umbrella's three general-guidance skills (agents, cutover-policy,
root-cause-analyzer) have no code, plugin or config behind them, so their home is
the one repo every consumer shares (opencharly/opencharly#600, AGENTS.md Part II
#5). This is the pin #566 withdrew: a repo declaring 412
submodules could not be a corpus source at all while the walk's fetch populated
every one of them and failed the whole fetch when one could not be fetched.

Producer-first: opencharly/spec#281 landed the files-only fetch
(refs.DownloadRepoFiles, tag v0.2026296.2332), opencharly/plugin-marketplace#49
made the corpus walk use it (tag candy/plugin-marketplace/v0.2026296.2345), and
this PR takes both plus the pin.

The corpus delta is ONE file — internals/skills/agents/references/
hooks-and-lifecycle.md — because the three bodies are byte-identical in both
sources; the agents REFERENCE file is the only thing that differs, and it now
matches the umbrella's, not layer-charly-internals@v2026.296.1907's copy. 361
SKILL.md, 496 artifacts, no skill gained or lost. The carrier keeps publishing
git-workflow and repo-setup, so its pin stays: its drain is its own PR.

Assisted-by: DeepSeek Harness deepseek-v4.1-flash (fully tested and validated)

Co-authored-by: Andreas Trawoeger <atrawog@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants