feat(generate): add generate llms command for per-project llms.txt#7
Merged
Conversation
Add a `generate llms` command that produces an llms.txt file for each
documentation project, supporting a progressive-disclosure setup where a
master llms.txt links to each project's own llms.txt.
For each project's current version (and non-versioned projects), it
enumerates pages, extracts the page title and meta :description:, resolves
the production URL (with .md appended), and writes content/<project>/llms.txt.
After writing, it prints a per-project character-count summary (with and
without descriptions) flagging files over the 50k llms.txt guideline.
Details:
- Root landing pages use the <root>/index.md markdown form (no <root>.md).
- Snooty {+name+} substitutions in titles and descriptions are resolved
from the project's snooty.toml [constants].
- Pages without a description omit the trailing ": description".
- includes/ and code-examples/ dirs and the deprecated app-services and
realm projects are excluded.
- --no-descriptions flag omits descriptions from written files.
Add internal/rst meta-description and page-title parsers, snooty constants
parsing + ResolveSubstitutions, tests, and README documentation.
Document the generate llms command under a new 0.4.0 release (which also folds in the previously-unreleased resolve url command) and bump the version constant in main.go. Also fix the 0.3.0 release date (2026-01-07, was incorrectly 2025-01-07).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a new
generate llmscommand that produces anllms.txtfile for each documentation project, supporting a progressive-disclosure setup: a masterllms.txtacts as a sitemap linking to each project's ownllms.txt.For each project's current version (and non-versioned projects), the command:
meta:description:.md<output-dir>/<project>/llms.txtin the format:- [Page Title](https://www.mongodb.com/docs/manual/core/document.md): Description.llms.txtguidelineBehavior details
<root>/index.mdform (the<root>.mdform 404s in production;<root>/index.mdis the real markdown).{+name+}references in titles and descriptions are resolved from the project'ssnooty.toml[constants].: description.includes/andcode-examples/directories; deprecatedapp-servicesandrealmprojects; non-project dirs (404,docs-platform,meta,table-of-contents).--no-descriptionsflag omits descriptions from written files (useful for oversized projects / iterating on docs).Flags
--output-dir,--for-project,--no-descriptions,--base-urlFindings from a full run (46 projects)
6 projects exceed 50k with descriptions (
atlas,atlas-cli,cloud-manager,manual,mongocli,ops-manager); none of them drop under 50k by removing descriptions alone, so they'll need per-section splitting. The other 40 fit comfortably with descriptions.Changes
commands/generate/{generate.go,llms/*},internal/rst/{meta_parser.go,page_title.go}main.go(register command),internal/snooty/snooty.go(constants +ResolveSubstitutions),README.mdTest plan
go build ./...✅go test ./...✅ (all packages green).mdURLs return 200 live; confirmed 0 remaining{+...+}refs across all generated files