Skip to content

agentmesh-one infers its public router address from X-Forwarded-* but the audit remediation stopped trusting those headers for the OIDC issuer: one rule for both #752

Closed

Description

@aojea

Two parts of the same binary treat forwarded headers differently.

Issuer. Until e67f5eab (2026-10-06, "security: remediate audit findings") the control plane built its OIDC issuer URL from the request and trusted X-Forwarded-Proto for the scheme. That commit put the header behind Options.TrustForwardedHeaders, default false, and nothing sets it in either binary. Behind a TLS-terminating proxy the issuer is therefore http://<host> unless the operator sets --external-url (agentmesh-one) or --sts-issuer-url (agentmesh-control-plane). This is the audited state: a deployment behind a proxy states its public URL.

Router address. #735 (ce32a935, 2026-10-17) made agentmesh-one infer the public wss:// multiaddr it advertises on /info, /enroll, /enroll/status and /register from Host / X-Forwarded-Host and X-Forwarded-Proto, unconditionally, when --external-url is not set (wrapPublicHTTPHandler, inferredRequestMultiaddr). That re-introduced for the router address the trust the audit removed for the issuer, without a review against that finding.

What a forged header buys an attacker. For the issuer: a client that reaches the plain listener directly chooses the scheme of the issuer in its own discovery response and in the iss of the JWT minted for its request; a relying party pinning the issuer sees a value that varies per request. For the router address: a client choosing X-Forwarded-Host gets a /info answer that names a router address of its choosing — only in its own response, but /enroll responses are what a node persists as its mesh configuration. In both cases the exposure needs direct access to the plain listener; behind Cloud Run the front door sets the headers itself.

Options.

  1. One rule: a hosted deployment states its public URL. Revert the inference half of standalone: put the inferred public router address on the /info router entries #735 (keep the /info entry fix), and have agentmesh-one refuse to start on a non-loopback listener without --external-url or --tunnel, with a message that names the flag. Zero config stays zero trust: the Cloud Run guide gains one flag.
  2. One rule, opt-in: a --trust-forwarded-headers flag on agentmesh-one that sets both the router inference and TrustForwardedHeaders, off by default, help text naming the risk.
  3. Leave as is and document the asymmetry.

I would take 1: it matches the audit's decision and the two guides already pass --external-url. #749 (guide for federated Google credentials) was written for that state; an earlier revision of it set TrustForwardedHeaders in agentmesh-one and was withdrawn for this discussion.

Activity

  1. aojea commented on Oct 18, 2026

    @aojea
    CollaboratorAuthor

    Analysis and proposal, after reading what a node does with an advertised address.

    An address is a reachability hint, not a trust anchor. The multiaddr a node receives from /enroll or /info ends in /p2p/<PeerID>, and in agentmesh-one that PeerID is the server's own; a forged header changes host, port or scheme, never the PeerID. The node then (1) runs the libp2p security handshake, which fails on peer id mismatch (ErrFatalAuth), and (2) at /mesh/auth requires a router Biscuit signed by a control plane key it already trusts, with the expected router as subject (VerifyBiscuitAndGetKey). A wrong address makes a node dial the wrong place; it cannot make the wrong place be the router. The issuer is parallel: iss ends up in a discovery document and in JWTs signed by the control plane's key; a relying party pins the issuer URL, so a forged iss makes the forger's own tokens useless and nothing else. The inference wrapper rewrites only the response to the request that carried the header; nothing is stored. So the concrete effect of a forged forwarded header is self-inflicted unavailability, plus one amplification path if an operator puts a caching proxy in front of an authenticated API. Zero trust and inference do not conflict: the mesh verifies identity on every connection and call, and does not trust the address it learned a peer at.

    Why inference exists. One reason: a PaaS hostname was not known before the first deploy, so the Cloud Run guide would need two passes. That reason is weaker than it looked. Cloud Run URLs are deterministic, https://<service>-<project number>.<region>.run.app, known before the first deploy; Fly.io and Render hostnames are deterministic too. --tunnel learns its URL from the provider. A hub bound to a VM address advertises its interfaces as before #735.

    Proposal: always state the public URL; remove inference.

    • Revert the inference half of standalone: put the inferred public router address on the /info router entries #735 (wrapPublicHTTPHandler, inferredRequestMultiaddr, prependInferredRouterAddr, the response recorder); keep the /info entry fix, which is correct when --external-url is set.
    • Delete Options.TrustForwardedHeaders (unused since e67f5eab).
    • When --external-url is absent and a request arrives with X-Forwarded-Proto or X-Forwarded-Host, log a warning once and count it: the advertised router address and OIDC issuer will not be reachable from outside.
    • Cloud Run guide: the one-command deploy computes the URL (gcloud projects describe … projectNumber) and passes --external-url.
    • agentmesh-control-plane keeps --sts-issuer-url; no inference there today.

    Result: nothing the hub advertises or signs derives from a request header; one advertised address and one issuer for every client; the lease and /info agree; less code. Cost: one flag whose value is computable before the deploy.

  2. added a commit that references this issue on Oct 18, 2026
    758f986
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions