Skip to content

Track shadow knowledge revisits in Markdown - #42

Merged
Xingdi (Eric) Yuan (xingdi-eric-yuan) merged 12 commits into
mainfrom
feature/shadow-citation-ranking
Sep 28, 2026
Merged

Xingdi (Eric) Yuan (xingdi-eric-yuan) merged 12 commits into
mainfrom
feature/shadow-citation-ranking

Conversation

@xingdi-eric-yuan

@xingdi-eric-yuan Xingdi (Eric) Yuan (xingdi-eric-yuan) commented Sep 23, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Add a visible citation_score to shadow discoveries and preferences so agents can recognize knowledge that previous tasks have revisited. The score is an approximate, agent-reported frequency hint stored directly in Markdown.

- Rejects expired tokens before opening a session.
  _(verified, source: exploration, citation_score: 3)_

New entries start at zero; omitted scores also mean zero.

Recording a citation

After consulting a discovery, the agent records it once per task using its file, symbol, and exact claim text:

python .github/skills/shadow-frog/shadow-cite.py .shadow/src/auth.py.md --symbol Auth.login --text "Rejects expired tokens before opening a session."

Use .claude/skills/ for Claude Code. Supply the claim body in --text and repeat that argument to update several entries in the same file/section atomically. For preferences and cross-cutting discoveries, supply the file and claim text alone.

Features

  • Visible metadata: a nonnegative citation_score follows the source and optional label fields in each entry.
  • Safe increments: shadow-cite.py identifies exact claims, serializes counter updates with per-file locking, and publishes them atomically. Ambiguous matches, invalid scores, busy writers, and detected edit races produce corrective feedback.
  • Viewer support: discovery views display scores and use them as a secondary ranking signal after relevance, source trust, and verification status.
  • Authoring and reconciliation: new Dream discoveries start at zero. Reconciliation and Meditate retain the larger score when combining duplicate knowledge, avoiding double-counted shared ancestry.
  • Consistent guidance and examples: the skills describe once-per-task citation reporting, and the live demo shows the metadata field.

Citation reporting is explicit: each successful helper invocation increments the selected entries, so the agent or coordinator handles once-per-task deduplication. Subagents can report consulted claims to a single writer. Scores travel with the Markdown through Git; Dream reconciliation imports score updates supplied on matching manifest discoveries.

Track surfaced discoveries with deterministic identities and an atomic local SQLite ledger shared across worktrees. Add bounded symbol-aware retrieval, stable pagination, per-entry expansion and retry-safe counting while preserving canonical knowledge formats and hook fail-open behavior.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Bind cursor requests by digest rather than storing raw query text in local usage metadata.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Reuse fully synchronized rollback journals and retry only SQLite lock contention after rolling back each whole write attempt. Keep hook timeouts and the six-process exact-count workload unchanged; cover blocked commits and non-lock failures.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@xingdi-eric-yuan Xingdi (Eric) Yuan (xingdi-eric-yuan) changed the title Add single-score citation ranking and bounded shadow retrieval [WIP] Add single-score citation ranking and bounded shadow retrieval Sep 24, 2026
Keep canonical file/symbol identities, literal text, and merged labels intact. Share hook budgets across previews, bind long reads to their content revision and event, and bound local retry/pagination metadata while preserving fail-open accounting.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Preserve reconciliation path containment alongside citation identity, bounded retrieval, and logical-read accounting. Retain both feature entries in the changelog.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep full commit synchronization while allowing readers and score writers to progress independently. Retain bounded retries and exact-count stress coverage, configure automatic checkpoints, and give the expanded cross-platform regression job time to finish.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Move shared knowledge/citation helpers into the core skill, expose bounded reads by known file or symbol, and keep the Viewer user-facing. Restore direct-navigation guidance, route hooks through the core reader, and include the optional reference in pinned tooling.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use visible citation_score metadata and an exact-claim increment helper with per-file locking and atomic publication. Preserve native navigation and user-facing views, retain counters during knowledge merges, and remove database-backed retrieval and pagination machinery.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@xingdi-eric-yuan Xingdi (Eric) Yuan (xingdi-eric-yuan) changed the title [WIP] Add single-score citation ranking and bounded shadow retrieval [WIP] Track shadow knowledge revisits in Markdown Sep 24, 2026
@xingdi-eric-yuan Xingdi (Eric) Yuan (xingdi-eric-yuan) changed the title [WIP] Track shadow knowledge revisits in Markdown Track shadow knowledge revisits in Markdown Sep 24, 2026
Exclude fenced examples and unrelated headings from exact-entry matching, merge cross-cutting scores at the actual discovery metadata, and clarify that existing shadows require no score reset.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@xingdi-eric-yuan
Xingdi (Eric) Yuan (xingdi-eric-yuan) merged commit 33d4994 into main Sep 28, 2026
6 checks passed
@xingdi-eric-yuan
Xingdi (Eric) Yuan (xingdi-eric-yuan) deleted the feature/shadow-citation-ranking branch September 28, 2026 13:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant