Add agent-readiness improvements (Link headers, markdown negotiation, Content-Signal, agent-skills, WebMCP)#807
Open
IEvangelist wants to merge 6 commits into
Open
Add agent-readiness improvements (Link headers, markdown negotiation, Content-Signal, agent-skills, WebMCP)#807IEvangelist wants to merge 6 commits into
IEvangelist wants to merge 6 commits into
Conversation
… robots Content-Signal, agent-skills, WebMCP) Bring aspire.dev up to spec for the checks at https://isitagentready.com: * RFC 8288 Link headers on HTML responses - LinkHeaderMiddleware advertises </llms.txt>; rel="llms", </.well-known/agent-skills/index.json>; rel="agent-skills", </sitemap-index.xml>; rel="sitemap", and a per-page rel="alternate" type="text/markdown" link when a .md companion exists. - Header attached via Response.OnStarting on 2xx text/html responses only; redirects, JSON, static assets, and well-known JSON are skipped. * Cloudflare-style "Markdown for Agents" content negotiation - MarkdownNegotiationMiddleware handles Accept: text/markdown by streaming the .md companion (emitted by starlight-page-actions) directly via IFileProvider.SendFileAsync. No path rewrite, so no interaction with UseRouting / MapStaticAssets endpoint selection. - Cache-Control: private, max-age=0, must-revalidate ensures Front Door does NOT cache, avoiding Vary: Accept cache-key explosion. - 406 when markdown preferred but no companion AND no HTML acceptable. - HEAD parity, Vary: Accept on negotiated responses, infrastructure paths (.well-known, _astro, healthz, install., pagefind) bypass negotiation. * Both new middlewares run BEFORE UseDefaultFiles + UseRouting (UseDefaultFiles rewrites /foo/ -> /foo/index.html, breaking companion mapping; MapStaticAssets registers endpoints during UseRouting, so post-routing path rewrites do not re-trigger endpoint selection). * robots.txt declares Content-Signal: ai-train=yes, search=yes, ai-input=yes inside the User-agent: * group (per draft-romm-aipref-contentsignals). * /.well-known/agent-skills/index.json (Agent Skills Discovery RFC v0.2.0) with a getting-started-with-aspire SKILL.md and a digest field of the form sha256:<lowerhex>. compute-skill-digests.mjs recomputes / verifies on every build; pnpm lint runs verify-skill-digests in --check mode. .gitattributes pins LF for the agent-skills artifacts so digests are byte-stable across Windows / Linux checkouts. * WebMCP integration on the Astro side - src/scripts/webmcp.ts feature-detects navigator.modelContext.registerTool and registers a single search-aspire-docs tool with a JSON Schema input. - Backed by src/scripts/search/* (SearchProvider abstraction with Pagefind today and a Typesense stub for the upcoming migration). The WebMCP tool surface is engine-agnostic so the Pagefind -> Typesense swap is a one-line change in src/scripts/search/index.ts. - Hooked into Head.astro via a single import line. * New host-level tests in tests/StaticHost.Tests/ - In-process TestServer with a temp wwwroot fixture (no frontend build required; PrivateAssets="all" on the frontend.esproj reference prevents the dist/ directory from leaking into test compilations). - 51 tests covering markdown negotiation (incl. HEAD, 406, fallback, Vary behavior, infrastructure-path skip), Link header content + skip rules, AcceptHeaderParser q-value handling, and the well-known artifacts. * New Playwright spec tests/e2e/webmcp.spec.ts asserts that the homepage registers exactly one WebMCP tool (search-aspire-docs) when the runtime exposes navigator.modelContext, and that the absence of the API is non-fatal. Out of scope (intentionally not advertised, would mislead agents): * /.well-known/openid-configuration / oauth-authorization-server (no protected APIs). * /.well-known/oauth-protected-resource (no protected resource). * /.well-known/mcp/server-card.json (aspire.dev does not host an MCP server). * /.well-known/api-catalog (RFC 9727 requires real API endpoints; aspire.dev exposes documentation, an LLM corpus, a sitemap, and an RSS feed - none of which are APIs in the RFC's sense). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ill conventions The first cut was inaccurate — it told agents to `mkdir my-aspire-app && cd my-aspire-app && aspire new` and then non-interactively pick a template. In reality `aspire new` is fully interactive and creates its own project folder, so the mkdir+cd pattern is wrong and the fabricated template flags would mislead agents. Rewrite the skill to align with the conventions used by the official skill at github.com/microsoft/aspire/tree/main/.agents/skills/aspire while keeping this one short and focused on getting started: * Frontmatter description is now a long when-to-use / when-not-to-use sentence in the same shape as the official skill. * Body: install, `aspire new` (interactive, no fabricated flags), `aspire start` (called out as the agent-friendly path vs. `aspire run` which blocks the terminal), and a concise list of authoritative references. * Explicit pointer to the official `aspire` skill for the operate-an-existing- app workflow, so an agent that has both available picks the right one. * Updated index.json description to match the new framing. * compute-skill-digests.mjs refreshed the sha256 to reflect the new bytes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Aspire is a polyglot stack — the AppHost can be authored in C# or TypeScript today, with additional languages (Java, Go, Python, Rust, …) on the roadmap. Calling it "the .NET cloud-native stack" was both inaccurate and misleading to agents who would then assume C#-only tooling and dismiss the TypeScript AppHost path. Re-runs compute-skill-digests.mjs to update the index.json digest to match the corrected SKILL.md bytes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Concerns flagged by the user: 1. LinkHeaderMiddleware.ShouldSkip duplicated the infrastructure path list maintained on MarkdownPathMapper.IsInfrastructurePath. Fixed by routing the path-skip check through the helper so both middlewares stay in lock-step. 2. Not every page on aspire.dev produces a `.md` companion (DocFX-rendered /reference/api/**, the search route, Lunaria stats, redirects, the 404 page). The previous mapper accepted any `.md` that happened to exist on disk, so a stray markdown file with no real HTML page would have been advertised via the `Link: rel="alternate"; type="text/markdown"` header and served by the negotiation middleware on `Accept: text/markdown`. Fixed by requiring BOTH the `.md` AND the corresponding HTML page to exist before declaring a companion. Adds new xUnit cases pinning the stray-md scenario for both middlewares. Additional cleanup along the way: * AcceptHeaderParser.PrefersMarkdown: removed a dead `htmlQ` assignment and hoisted `HighestExplicitQuality` to a private static method so markdown and html lookups go through the same helper. * Extracted shared sample HTML/Markdown bodies and seed helpers used by LinkHeaderTests + MarkdownNegotiationTests into SamplePages so the on-disk Starlight layout is described in one place. Tests: 57 passing (was 51), 0 warnings, 0 errors. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds “agent-readiness” features to the StaticHost and frontend to align aspire.dev with common agent discovery/interaction checks (Link headers, markdown negotiation, Content-Signal, agent-skills artifacts + digesting, and WebMCP tool registration), plus a dedicated StaticHost test project and Playwright coverage.
Changes:
- Introduces
UseAgentReadiness()middleware composition (markdown content negotiation + RFC 8288 Link headers) and wires it into the StaticHost pipeline beforeUseDefaultFiles/UseRouting. - Adds
.well-known/agent-skillsdiscovery artifacts (plus digest computation + LF enforcement) and arobots.txtContent-Signal directive. - Adds frontend WebMCP registration (
search-aspire-docs) backed by a pluggable search provider layer, with Playwright e2e tests.
Reviewed changes
Copilot reviewed 33 out of 33 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/StaticHost.Tests/WellKnownArtifactTests.cs | Verifies robots.txt Content-Signal and agent-skills discovery/digests + LF-only enforcement. |
| tests/StaticHost.Tests/StaticHost.Tests.csproj | New StaticHost-focused test project wiring. |
| tests/StaticHost.Tests/SamplePages.cs | Shared HTML/MD fixtures for middleware tests. |
| tests/StaticHost.Tests/MarkdownPathMapperTests.cs | Unit coverage for request-path → markdown-companion mapping rules. |
| tests/StaticHost.Tests/MarkdownNegotiationTests.cs | Host-level tests for Accept negotiation (GET/HEAD/406/Vary/Cache-Control). |
| tests/StaticHost.Tests/LinkHeaderTests.cs | Host-level tests for Link header attach/skip rules. |
| tests/StaticHost.Tests/GlobalUsings.cs | Global usings for the new test project. |
| tests/StaticHost.Tests/AgentReadinessTestServer.cs | In-proc TestServer harness mirroring production middleware ordering. |
| tests/StaticHost.Tests/AcceptHeaderParserTests.cs | Direct q-value parsing/predicate tests. |
| src/statichost/StaticHost/StaticHost.csproj | Makes frontend ESProj reference non-transitive via PrivateAssets=all. |
| src/statichost/StaticHost/Properties/AssemblyInfo.cs | InternalsVisibleTo for StaticHost.Tests. |
| src/statichost/StaticHost/Program.cs | Wires app.UseAgentReadiness() before default files/routing. |
| src/statichost/StaticHost/GlobalUsings.cs | Adds global using for AgentReadiness namespace. |
| src/statichost/StaticHost/AgentReadiness/MarkdownPathMapper.cs | Central path mapping + infrastructure skip rules. |
| src/statichost/StaticHost/AgentReadiness/MarkdownNegotiationMiddleware.cs | Serves .md companions based on Accept preferences + caching semantics. |
| src/statichost/StaticHost/AgentReadiness/LinkHeaderMiddleware.cs | Emits discovery Link headers on successful HTML responses. |
| src/statichost/StaticHost/AgentReadiness/AgentReadinessExtensions.cs | Composition root extension and enforced middleware ordering. |
| src/statichost/StaticHost/AgentReadiness/AcceptHeaderParser.cs | Minimal Accept parser focused on html/markdown preference logic. |
| src/frontend/tsconfig.json | Adds @scripts/* path mapping. |
| src/frontend/tests/e2e/webmcp.spec.ts | Playwright coverage for WebMCP tool registration + non-fatal absence. |
| src/frontend/src/scripts/webmcp.ts | Registers search-aspire-docs tool when navigator.modelContext is present. |
| src/frontend/src/scripts/search/typesense-provider.ts | Stub Typesense provider for future migration. |
| src/frontend/src/scripts/search/SearchProvider.ts | Shared provider interface + response/result shapes. |
| src/frontend/src/scripts/search/pagefind-provider.ts | Pagefind-backed provider (dynamic import) with graceful unavailability. |
| src/frontend/src/scripts/search/index.ts | Provider selection via PUBLIC_SEARCH_PROVIDER. |
| src/frontend/src/components/starlight/Head.astro | Imports WebMCP registration script site-wide. |
| src/frontend/scripts/compute-skill-digests.mjs | Recomputes/validates agent-skills SHA-256 digests from raw bytes. |
| src/frontend/public/robots.txt | Adds Content-Signal directive under User-agent: *. |
| src/frontend/public/.well-known/agent-skills/index.json | New agent-skills discovery index (v0.2.0). |
| src/frontend/public/.well-known/agent-skills/getting-started-with-aspire/SKILL.md | New skill document for “getting started” guidance. |
| src/frontend/package.json | Adds digest compute/verify scripts; runs them in dev/build/lint. |
| Aspire.Dev.slnx | Adds StaticHost.Tests to solution. |
| .gitattributes | Forces LF in agent-skills artifacts to keep digests byte-stable. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
* MarkdownNegotiationMiddleware: drop the stale reference to the `HasMarkdownCompanion` API (which never existed in this PR's final form); point readers at `MarkdownPathMapper.TryGetMarkdownCompanion` instead so the comment matches the live code. * compute-skill-digests.mjs: harden `resolvePublicPath` against `..` traversal. Switches from `path.join` to `path.resolve` and asserts the result stays under `publicRoot` so a malicious or malformed `url` in index.json (e.g. `/.well-known/agent-skills/../../../../etc/passwd`) cannot read bytes outside the published public/ tree in dev/CI. * WellKnownArtifactTests: rename `Agent_skills_files_are_LF_only` to `AgentSkills_files_are_LF_only` to match the surrounding `AgentSkills_*` PascalCase prefix. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
eerhardt
reviewed
May 11, 2026
| @@ -0,0 +1,3 @@ | |||
| using System.Runtime.CompilerServices; | |||
|
|
|||
| [assembly: InternalsVisibleTo("StaticHost.Tests")] | |||
Member
There was a problem hiding this comment.
(nit) you did this the other way in #758 - using the .csproj.
|
|
||
| private static bool ShouldSkip(HttpRequest request) | ||
| { | ||
| if (!HttpMethods.IsGet(request.Method) && !HttpMethods.IsHead(request.Method)) |
Member
There was a problem hiding this comment.
(nit) the logic is inverted here from MarkdownNegotiationMiddleware.
Two follow-ups from @eerhardt's review on PR #807: * Move InternalsVisibleTo into StaticHost.csproj using Include="$(AssemblyName).Tests" to match the convention established in PR #758 and the existing src/tools/*.csproj files. Delete the now-unnecessary Properties/AssemblyInfo.cs. * Mirror MarkdownNegotiationMiddleware's positive ShouldHandle pattern in LinkHeaderMiddleware. The two were inverted: one returned "should I handle?" while the other returned "should I skip?". Both now use the same shape — guard with !ShouldHandle, then check MarkdownPathMapper.IsInfrastructurePath separately. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implementation of the checks at https://isitagentready.com. See commit message for full details. Verified: all 51 new + 16 existing C# tests pass, frontend lint clean, full solution build clean. Tests cover middleware ordering, markdown negotiation (incl. HEAD/406/Vary), Link header skip rules, AcceptHeaderParser q-values, agent-skills schema + digest verification, robots Content-Signal, and WebMCP tool registration via Playwright init-script stub. Out of scope (intentionally): OAuth/OIDC discovery, OAuth Protected Resource, MCP Server Card, and api-catalog (RFC 9727 requires real API endpoints; aspire.dev has none).