Bot detection: challenge fails on path-based resources (/crowdsec-internal not routed)

[mazzma12]

Summary

With CrowdSec bot detection enabled (#1, upstream fosrl#3674), the challenge cannot complete on resources that use a path match.

The challenge page loads fpscanner.js / pow-worker.js and posts its result under /crowdsec-internal/challenge/* on the same host. For path-based resources Pangolin only generates Host(...) && PathPrefix(<path>) routers (server/lib/traefik/rule.ts appendPathMatch, server/lib/traefik/getTraefikConfig.ts), so these requests match no router and Traefik returns 404 before the bouncer middleware runs. Traefik middlewares, including entrypoint ones, only run once a router has matched, so a router covering /crowdsec-internal on the host is required in any fix.

Host-only resources are not affected. This is not a CrowdSec bug: the plugin's bot-detection example documents that routers must also match /crowdsec-internal, and CrowdSec's challenge protocol says these paths must be terminated by the remediation component and never reach the origin.

Reproduction

• CrowdSec 1.8.1 (crowdsecurity/crowdsec:latest), installer collections + crowdsecurity/appsec-bot-challenge, appsec_configs: [crowdsecurity/appsec-default, crowdsecurity/appsec-bot-*]
• Traefik v3.7, crowdsec-bouncer-traefik-plugin v1.8.0-alpha, Pangolin's crowdsec middleware block on the entrypoint
• Path-based resource (Host(`pathapp.e2e.localhost`) && PathPrefix(`/foo`)) → whoami

Opening /foo in Chrome:

Evidence	Result
Page	"Security Check · Verification failed · Please try again"
Traefik access log	GET /foo 200 via the resource router; GET /crowdsec-internal/challenge/fpscanner.js (or POST .../submit when the script is cached) 404, no router
cscli metrics show bot-detection	Requested 1+, Submitted 0

Reproduced both with a hand-written router and with routers generated by a Pangolin dev server (OSS build).

Fix options

Catch-all router (first attempt, not proposed). A host-less PathPrefix(`/crowdsec-internal`) router in the installer's dynamic_config.yml works, but it matches every Host (dashboard, unknown hosts, raw IP), lives outside Pangolin's routing and has no TLS domains. Upstream review on fosrl#3674 pushed back on catch-alls.

Option B: per-domain router generated by Pangolin (proposed, prototyped).

• New config key traefik.bot_detection (default false); the installer appends traefik: bot_detection: true to config.yml when bot detection is opted in.
• When enabled, both Traefik config generators (OSS and private) emit one router per resource domain:

crowdsec-internal-<domain>-router:
  rule: "Host(`<domain>`) && PathPrefix(`/crowdsec-internal`)"   # HostRegexp for wildcard resources
  entryPoints: [<same as the resource>]
  service: noop@internal
  priority: 3000   # above every priority Pangolin generates
  tls: <same as the resource, when ssl>

• Scoped to domains Pangolin serves, reuses the resource's TLS settings, and never forwards to the origin (noop@internal; AppSec answers every challenge request).

Tested end to end with the Pangolin dev server (resources created through the API), Traefik v3.7 + badger + crowdsec plugin v1.8.0-alpha, CrowdSec 1.8.1:

	bot_detection: true	bot_detection: false (control)
Generated routers	crowdsec-internal-* present for the HTTP and HTTPS resources, all enabled in Traefik	absent
Path-based resource in Chrome	"Verification successful", backend reached with the __crowdsec_challenge cookie	"Verification failed"
Bot detection metrics	Requested 1, Submitted 1, Solved 1, Rejected 0	Requested 1, Submitted 0

Option C: extend each path-based rule with || PathPrefix(/crowdsec-internal) in rule.ts. Smallest diff, but the router's service is the resource backend, so any /crowdsec-internal request the bouncer lets through (e.g. crowdsecAppsecFailureBlock: false, AppSec outage) reaches the origin, which the challenge protocol forbids. Not recommended.

Option D (longer term): per-resource bot protection. Attach the crowdsec middleware only to resources that opt in instead of the whole entrypoint, and generate the option B router for their domains. Also addresses non-browser clients (mobile apps, API tools) being challenged on every resource.

Still to verify

• The challenge in a browser over the websecure entrypoint with a real certificate (the router and TLS settings were verified in Traefik's API only)
• Traefik writes no access-log entry for requests that end on noop@internal
• A resource with a user-set priority above 3000 and a host-only rule would outrank the crowdsec-internal router (the bouncer still answers first)

Testing note: browsers driven over the DevTools protocol (Playwright, Puppeteer, an attached debugger) are rejected by design (score_reasons: cdp=100); use a plain browser session.