Fix sitemap: include the /packages page, exclude the exported 404 page - #123
Merged
guanzhousongmicrosoft merged 1 commit intoJul 30, 2026
Merged
Conversation
Verified against the first production sitemap (268 URLs): - /packages/ was missing. The top-level 'packages' exclusion was meant for the package-repository metadata the deploy workflow adds, but the Next.js /packages download page also exports to out/packages/ - and the workflow only ever adds release-info.json there (not a page), and only after the sitemap step has already run. Stop excluding it. - /404/ was listed. With trailingSlash the export emits out/404/ index.html alongside out/404.html; the directory form needs an explicit exclusion. Re-tested against a mock out/ tree that now models the real export shape (404/ directory, packages/ page plus release-info.json).
guanzhousongmicrosoft
approved these changes
Jul 30, 2026
GuanzhouSong
added a commit
to GuanzhouSong/documentdb.github.io
that referenced
this pull request
Aug 3, 2026
The exclusion list was growing one framework route at a time. documentdb#123 added 404 after it appeared in the sitemap; this branch added _not-found for the same reason; and the versioned-docs work in progress adds a third hand-written skip for the archived version directories. Each is the same rule discovered again: do not advertise a page that tells crawlers not to index it. That property is readable from the page itself, so collectPages now reads each index.html and skips the ones carrying a noindex robots meta. The name list keeps only the directories that hold no pages at all - _next, deb, rpm, images - where it is a traversal concern rather than an indexing decision. The tag is matched in either attribute order and tolerates content lists such as "noindex,nofollow", since Next.js and hand-written metadata do not agree on either. The final log line now reports how many pages were skipped, so a filter that starts matching too much is visible in the build output instead of silently shrinking the sitemap. Verified against a synthetic export carrying an indexable root, /docs, /docs/versions, /packages and /samples, plus three noindex pages written in three different tag forms and an excluded _next directory: five URLs emitted, three skipped, and out/404.html left alone as the file it is. Note that the archived-version case is now covered by this rule, so the docs/versions skip on the versioned-docs branch can go when it lands.
guanzhousongmicrosoft
pushed a commit
that referenced
this pull request
Aug 3, 2026
* SEO cleanup: sitemap, 404 metadata, coming-soon heading Three small fixes found while validating the live site. Sitemap advertised /_not-found/. collectPages() emits a URL for any directory holding an index.html and filters only top-level names in an explicit deny-list. With trailingSlash the export writes out/_not-found/index.html, and _not-found was not in the list. #123 fixed exactly this class for 404 but only added that one name. The page also serves noindex, so the sitemap was submitting a URL that tells crawlers not to index it. Added _not-found to the exclusion set. 404 page emitted two conflicting robots tags. not-found.tsx exported no metadata, so it inherited the root layout's, which hardcodes index/follow in metadataService - and Next.js separately injects noindex for the not-found route. It also carried the homepage title byte for byte. Added a metadata export with a distinct title and index: false. /docs/architecture had no heading element at all - not just no h1, zero h1-h6. The coming-soon layout renders the graphic plus placeholder prose, and the markdown carries no heading, while the sidebar section label is deliberately a <p> because "the article's h1 comes from the markdown content". That contract is unmet for coming-soon pages, so the h1 now comes from the frontmatter title, guarded in case a future coming-soon page does start with a markdown heading. The docs index also showed the Architecture card with no marker, styled identically to the six finished ones. Marked it the same way the Kubernetes Operator card marks preview status, via a title suffix. Fixes #129 * Filter the sitemap on noindex rather than on directory name The exclusion list was growing one framework route at a time. #123 added 404 after it appeared in the sitemap; this branch added _not-found for the same reason; and the versioned-docs work in progress adds a third hand-written skip for the archived version directories. Each is the same rule discovered again: do not advertise a page that tells crawlers not to index it. That property is readable from the page itself, so collectPages now reads each index.html and skips the ones carrying a noindex robots meta. The name list keeps only the directories that hold no pages at all - _next, deb, rpm, images - where it is a traversal concern rather than an indexing decision. The tag is matched in either attribute order and tolerates content lists such as "noindex,nofollow", since Next.js and hand-written metadata do not agree on either. The final log line now reports how many pages were skipped, so a filter that starts matching too much is visible in the build output instead of silently shrinking the sitemap. Verified against a synthetic export carrying an indexable root, /docs, /docs/versions, /packages and /samples, plus three noindex pages written in three different tag forms and an excluded _next directory: five URLs emitted, three skipped, and out/404.html left alone as the file it is. Note that the archived-version case is now covered by this rule, so the docs/versions skip on the versioned-docs branch can go when it lands.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Post-deploy review of the first production sitemap (268 URLs at https://documentdb.io/sitemap.xml) found two defects in the generator, both introduced in #120:
/packages/was missing — the top-levelpackagesexclusion was meant for the package-repository metadata the deploy workflow adds toout/, but the Next.js /packages download page also exports toout/packages/index.html. The workflow only ever addsrelease-info.jsonthere (not a page), and only after the sitemap step has run, so the exclusion protected nothing and cost the sitemap one of the site's highest-intent pages./404/was listed — withtrailingSlash: truethe export emitsout/404/index.htmlalongsideout/404.html; the original mock test modeled only the file form, so the directory slipped through.Validation
Re-tested against a mock
out/tree that now models the real export shape (404/directory,packages/page withrelease-info.jsonbeside it,deb/repo dir with its own index.html): output contains/packages/and/samples/, and excludes/404/and/deb/.🤖 Generated with Claude Code
https://claude.ai/code/session_01XGMeNSmhAgmqzkdc7cQQgf