From e170ab540d457986dcebddc20f79b70bcbb198d4 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 23 Sep 2026 16:05:58 +0000 Subject: [PATCH 1/2] Assert the /metrics 404 against Caddy on the host, not through the public name Production's public name now resolves to a fronting edge that answers every path outside the MCP and OAuth ones with a 308 to internetcomputer.org. The release deploy's last check, exactly a 404 from https://$DOMAIN/metrics, can therefore no longer pass there (run 35884611192 failed on it after the deploy itself had gone through), and it no longer measured this host's configuration anyway. The assertion now asks Caddy on the host, over SSH with $DOMAIN pinned to loopback so its site block answers. The request through the public name stays, but only a 200 there fails the deploy: that alone proves the exposition public. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01Xd7VT72Qt16qiAJynu9EKj --- deploy/native/README.md | 7 +++++-- deploy/native/deploy.sh | 33 ++++++++++++++++++++------------- 2 files changed, 25 insertions(+), 15 deletions(-) diff --git a/deploy/native/README.md b/deploy/native/README.md index 5f11f48..f80a171 100644 --- a/deploy/native/README.md +++ b/deploy/native/README.md @@ -192,8 +192,11 @@ driven by the dashboard's Statuspage pusher, which keeps running on every host. Both hosts set `MCP_SERVE_METRICS=1` (hardcoded in the unit), so the app serves the Prometheus exposition at `/metrics` on its own port for a scraper to reach on -the host's private address. Caddy answers `/metrics` on the public origin with a -404, which is what keeps it off the internet. +the host's private address. Caddy answers `/metrics` with a 404, which is what keeps +it off the internet. `deploy.sh` asserts that 404 against Caddy on the host itself, +since a public name behind a fronting edge (production's is) answers with the edge's +response rather than Caddy's; it separately fails the deploy if the public name ever +returns 200 for it. The mechanics live in [`deploy-native.yml`](../../.github/workflows/deploy-native.yml), a reusable workflow both call. It first runs the status dashboard's unit tests (a diff --git a/deploy/native/deploy.sh b/deploy/native/deploy.sh index d41b6a4..c8452ec 100755 --- a/deploy/native/deploy.sh +++ b/deploy/native/deploy.sh @@ -274,27 +274,34 @@ echo ">> external check:" curl -sS --max-time 20 -o /dev/null -w "https://$DOMAIN/ -> HTTP %{http_code} (TLS verify %{ssl_verify_result})\n" "https://$DOMAIN/" || true curl -sS --max-time 20 -o /dev/null -w "https://$DOMAIN/status/ -> HTTP %{http_code}\n" "https://$DOMAIN/status/" || true -# ...and /metrics must NOT be reachable on the public origin. Caddy answers it -# with a 404; losing that one block would publish the exposition, so assert it on -# every deploy. The assertion is "exactly the configured 404": any other code — -# a proxied 500, a `000` from an unreachable or stalled origin — has not disproved -# exposure, so it retries (Caddy may be reloading) and then fails hard. +# /metrics must NOT be reachable on the public origin: the Caddyfile answers it +# with a 404, and losing that one block would publish the exposition. Ask Caddy +# itself, from the host, with $DOMAIN pinned to loopback so its site block +# answers: production's public name now resolves to a fronting edge that +# redirects every non-MCP path, so the public answer says nothing about this +# host. -k because the question is the route, not the certificate. hidden="" for attempt in 1 2 3 4 5; do - code="$(curl -sS --max-time 20 -o /dev/null -w '%{http_code}' "https://$DOMAIN/metrics" || echo 000)" - if [ "$code" = 200 ]; then - echo "FATAL: https://$DOMAIN/metrics answered 200 — the Prometheus exposition is public" >&2 - exit 1 - fi + code="$($SSH "curl -ksS --max-time 10 -o /dev/null -w '%{http_code}' --resolve '$DOMAIN:443:127.0.0.1' 'https://$DOMAIN/metrics'" || echo 000)" if [ "$code" = 404 ]; then hidden=1 - echo "https://$DOMAIN/metrics -> HTTP 404 (not published, as intended)" + echo "caddy on the host: https://$DOMAIN/metrics -> HTTP 404 (not published, as intended)" break fi - echo "https://$DOMAIN/metrics -> HTTP $code, expected 404 (attempt $attempt)" >&2 + echo "caddy on the host: https://$DOMAIN/metrics -> HTTP $code, expected 404 (attempt $attempt)" >&2 sleep 3 done if [ -z "$hidden" ]; then - echo "FATAL: https://$DOMAIN/metrics never answered the configured 404, so its exposure is unproven" >&2 + echo "FATAL: Caddy never answered the configured 404 for /metrics, so its exposure is unproven" >&2 + exit 1 +fi + +# Through the public name too: a 200 means the exposition IS public, whatever +# sits in front, and fails the deploy; anything else (Caddy's 404, an edge's +# redirect) is reported, not judged. +code="$(curl -sS --max-time 20 -o /dev/null -w '%{http_code}' "https://$DOMAIN/metrics" || echo 000)" +if [ "$code" = 200 ]; then + echo "FATAL: https://$DOMAIN/metrics answered 200 — the Prometheus exposition is public" >&2 exit 1 fi +echo "https://$DOMAIN/metrics -> HTTP $code" From b6801c3c02e747e6bffab3540d286148cf600096 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 23 Sep 2026 16:29:11 +0000 Subject: [PATCH 2/2] Judge the public /metrics probe by its body, not its status A 200 through the public name is not proof the exposition is public (an edge may answer any unknown path with a page of its own), and a redirect left unfollowed could hide one that lands on it. The probe now follows redirects and fails the deploy only when the body carries imcp2_build_info, the marker the on-host checks already use; otherwise it reports the final status and URL. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01Xd7VT72Qt16qiAJynu9EKj --- deploy/native/README.md | 4 ++-- deploy/native/deploy.sh | 16 +++++++++------- 2 files changed, 11 insertions(+), 9 deletions(-) diff --git a/deploy/native/README.md b/deploy/native/README.md index f80a171..bd2c7b1 100644 --- a/deploy/native/README.md +++ b/deploy/native/README.md @@ -195,8 +195,8 @@ the Prometheus exposition at `/metrics` on its own port for a scraper to reach o the host's private address. Caddy answers `/metrics` with a 404, which is what keeps it off the internet. `deploy.sh` asserts that 404 against Caddy on the host itself, since a public name behind a fronting edge (production's is) answers with the edge's -response rather than Caddy's; it separately fails the deploy if the public name ever -returns 200 for it. +response rather than Caddy's; it separately fails the deploy if the exposition itself +ever shows up through the public name, redirects followed. The mechanics live in [`deploy-native.yml`](../../.github/workflows/deploy-native.yml), a reusable workflow both call. It first runs the status dashboard's unit tests (a diff --git a/deploy/native/deploy.sh b/deploy/native/deploy.sh index c8452ec..56e91cd 100755 --- a/deploy/native/deploy.sh +++ b/deploy/native/deploy.sh @@ -296,12 +296,14 @@ if [ -z "$hidden" ]; then exit 1 fi -# Through the public name too: a 200 means the exposition IS public, whatever -# sits in front, and fails the deploy; anything else (Caddy's 404, an edge's -# redirect) is reported, not judged. -code="$(curl -sS --max-time 20 -o /dev/null -w '%{http_code}' "https://$DOMAIN/metrics" || echo 000)" -if [ "$code" = 200 ]; then - echo "FATAL: https://$DOMAIN/metrics answered 200 — the Prometheus exposition is public" >&2 +# Through the public name too, redirects followed: the exposition's own marker +# in the body is what proves it public, whatever sits in front. A status alone +# proves nothing either way (an edge's 200 page, a redirect elsewhere). +public="$(mktemp)" +summary="$(curl -sSL --max-redirs 5 --max-time 20 -o "$public" -w '%{http_code} at %{url_effective}' "https://$DOMAIN/metrics")" || summary="${summary:-000} (request failed)" +if grep -q imcp2_build_info "$public"; then + echo "FATAL: https://$DOMAIN/metrics serves the Prometheus exposition publicly ($summary)" >&2 exit 1 fi -echo "https://$DOMAIN/metrics -> HTTP $code" +rm -f "$public" +echo "https://$DOMAIN/metrics -> $summary; no exposition in the body"