feat(fetch): add opt-in host allowlisting (--allowed-hosts)

[g-yixuan]

Description

Adds an opt-in host allowlist to the fetch server, addressing the SSRF surface tracked in #2317 without changing default behavior.

Fixes #2317.

• New --allowed-hosts HOST [HOST ...] CLI flag. When unset, behavior is exactly as before (all hosts allowed), so deployments that intentionally fetch localhost/internal endpoints are unaffected (per the discussion on the issue).
• Entry forms: exact host, case-insensitive (example.com); wildcard (*.example.com) matching the bare domain and any subdomain; literal IPs.
• Enforcement covers the initial request, the robots.txt pre-check, and every redirect hop: redirects are now followed manually (bounded at 20, same as httpx's default) and each hop is re-validated before connecting, so a 302 from an allowed host can no longer bounce a fetch to a disallowed host.
• Denials fail closed with a clear error that does not echo the configured allowlist.

Publishing Your Server

N/A — change to an existing server.

Server Details

• Server: fetch (Python, mcp-server-fetch)
• Changes to: tools/prompts request handling (URL validation + redirect handling), CLI args, README

Motivation and Context

#2317 asks for host allowlisting. The issue discussion raised valid concerns about breaking users who intentionally fetch internal endpoints, so this PR deliberately does not change the default posture — the default-deny/IP-range part of the issue is separately covered by #4497, and this PR composes cleanly with it. A previous allowlisting attempt (#2568) went stale and was closed with an invitation to resubmit on a current base; this is that fresh take, scoped to avoid the default-behavior controversy.

On the TOCTOU/socket-level point in the issue: with a hostname allowlist, the validated hostname is exactly what gets connected (SNI/Host), so DNS check-time/connect-time skew does not weaken the control; the realistic bypass was redirect hops, which this closes.

How Has This Been Tested?

• uv run pytest — 64 passed (existing suite untouched and green, plus new coverage: exact/wildcard/IP matching, case/trailing-dot normalization, userinfo bypass attempt, per-hop redirect enforcement parametrized over 301/302/303/307/308, redirect-loop limit, empty and malformed Location headers (matching httpx's prior behavior), relative Location resolution, robots.txt path enforcement, per-request timeout semantics, and unset-flag backwards compatibility).
• uv run pyright — 0 errors; uv run ruff check . — clean.
• Exercised end-to-end over the MCP protocol: stdio client against the real server binary started with --allowed-hosts 127.0.0.1, plus a real local HTTP server — allowlisted host fetched, http://169.254.169.254/latest/meta-data blocked, redirect to a non-allowlisted host blocked, malformed Location handled with a clean error.
• URL validation uses httpx.URL — the same parser that performs the connection — so the validated host is always the host that gets connected.

Breaking Changes

None — default behavior is unchanged when --allowed-hosts is not set.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation update

Checklist

• I have read the MCP Protocol Documentation
• My changes follows MCP security best practices
• I have updated the server's README accordingly
• I have tested this with an LLM client
• My code follows the repository's style guidelines
• New and existing tests pass locally
• I have added appropriate error handling
• I have documented all environment variables and configuration options

Additional context

• Wildcard semantics are documented in the README: *.example.com matches example.com itself and any subdomain.
• The README also states an intentional limitation: the allowlist matches hostnames, not resolved IPs, which is what makes deliberately allowlisting internal hosts possible.

[g-yixuan]

Thanks for the look — both points are already covered:

• The robots.txt pre-check goes through the same gate: it uses the same _get_following_redirects helper, so the robots fetch and every one of its redirect hops are validated against the allowlist and count against the same 20-redirect bound (see test_robots_redirect_to_denied_host_blocked).
• Wildcard semantics are explicit: *.example.com matches the bare apex example.com as well as any subdomain — documented in the README and pinned by test_wildcard_matches_bare_domain.

[tiagovilasboas]

AppSec review — fetch --allowed-hosts (#4770)

Solid allowlist slice of #2317: httpx.URL parsing, per-hop redirect revalidation (incl. robots), wildcard/exact/IP tests, and fail-closed denials that do not echo the list. Hostname-only matching (DNS→internal for an allowlisted name) is documented and intentional.

Verdict: Request changes (comment form — no formal review permission on this repo).

Blocking

1. Empty/whitespace Location: Location: "" / spaces → urljoin keeps current URL and burns up to 20 hops; missing Location on 3xx returns the redirect body. Please fail closed on empty/missing Location when status is redirect (same ask as on fix(fetch): harden URL fetching against SSRF #4773) — update the tests that currently pin httpx’s loop/as-is behavior.

2. Compose with fix(fetch): harden URL fetching against SSRF #4773 via one redirect helper: This PR’s _get_following_redirects (allowlist, cap 20) and fix(fetch): harden URL fetching against SSRF #4773’s _get_with_validation (scheme + private-IP denylist, cap 5) both replace follow_redirects=True. Land a shared per-hop helper (allowlist ∪ IP/scheme checks, one hop budget, one Location policy) or the merge will drop a control.

3. Don’t Fixes #2317 wholesale: Issue also wanted default private-IP blocking and socket-level TOCTOU. This correctly scopes to opt-in allowlisting only — use Partial/Add support for host allowlisting to Fetch server #2317 and leave default-deny to fix(fetch): block SSRF to internal/metadata IPs by default #4497/fix(fetch): harden URL fetching against SSRF #4773.

Non-blocking

• IDN: httpx returns Unicode hosts; punycode allowlist entries silently never match — normalize or document.
• Error code INTERNAL_ERROR vs INVALID_PARAMS for denials — minor.

Happy to re-review once the three blockers are addressed.