Fix telemetry provenance, add secrets/exploit traps, budget slow responses

Cargo.toml

@@ -8,6 +8,7 @@ description = "Self-hosted exploit-path honeypot with sticky deception — traps
 [dependencies]
 axum = "0.8"
 tokio = { version = "1", features = ["full"] }
+tokio-stream = "0.1"
 sqlx = { version = "0.8", default-features = false, features = ["runtime-tokio", "tls-rustls", "postgres", "chrono", "json"] }
 governor = "0.8"
 serde_json = "1"

README.md

@@ -27,11 +27,24 @@ Bots race to complete WordPress's setup wizard on fresh installs — whoever fin
 
 `GET /.env` returns a realistic `.env` file containing a per-IP planted DB password (`fk` + 12 chars, deterministic from IP hash). The password is inserted into `granted_credentials`. If an attacker reads the `.env` and later submits that password at any login form, the submission is captured and matchable to the original probe — correlating the attacker across vectors.
 
+The planted pair is recorded in `honeypot_event.planted_user` / `planted_pass` — **never** in `submitted_user` / `submitted_pass`. Those two columns mean "the attacker sent us this", exclusively. (They did not always: the trap originally wrote plants into the `submitted_*` columns, which made the honeypot's own output indistinguishable from attacker input and inflated every credential metric roughly fourfold. Migration `004` separates the historical rows.)
+
 ## Attacker engagement
 
 Beyond passive capture, RustyPot actively wastes attacker resources:
 
 - **Tarpit escalation** — after each fake-success grant, the tarpit delay for failed attempts increases: 30s → 60s → 120s → 240s (capped below Cloud Run's timeout). The attacker's throughput drops progressively.
+- **Slow-response budget** — every delaying trap reserves a slot before it
+  holds a request open (`SLOW_RESPONSE_BUDGET`, default 64). Cloud Run gives
+  this service `containerConcurrency` 80 across `maxScale` 3 — 240 request
+  slots — and a held response occupies one for its full duration. Without a cap,
+  enough parallel tarpits would fill the pool and the honeypot would stop being
+  able to record new probes: tarpitting itself out of existence. When the budget
+  is spent the trap answers immediately instead, and logs a delay of 0, so
+  `response_delay_ms` always reflects time actually spent rather than time
+  intended. A fast response is unremarkable to an attacker; a request the
+  platform kills at its 300 s timeout is a 504 that identifies the trap.
+- **Recon tarpit** — the `.env` and `.git` families get their own, much shorter ladder (`RECON_TARPIT_ESCALATION`, default `0,2,5,10,20` s), stepped by how many recon paths the IP has swept (`RECON_TARPIT_STEP`, default 10). Secrets harvesting is the most common objective observed, and it used to be the one family that cost the attacker nothing. The ladder is deliberately short: sweepers hit hundreds of paths, and a 30–240 s hold per path would pin every instance and starve the credential traps. A one-off probe sits on rung 0; a 300-path enumerator climbs.
 - **Canary links** — every link in the fake admin dashboard carries a per-IP tracking token (`?fk=...`). When a bot clicks any link, the token is logged, mapping their post-exploitation path sequence.
 - **Git loop** — `/.git/config` returns a realistic git config. `/.git/objects/` returns HTML directory listings. Each object page links to 3 more subdirectories, each with 10 objects — an infinite chain for HTML-following scanners. Pack files return 8KB with valid `PACK` headers.
 - **Cookie bombing** — the first fake-success response sets 20 cookies of 400 bytes each (~9KB). The attacker's HTTP client echoes all cookies on every subsequent request, cutting effective throughput.
@@ -52,17 +65,27 @@ Beyond passive capture, RustyPot actively wastes attacker resources:
 | `/.env*` (any variant: `.env.dev`, `.envrc`, `.env_copy`, ...) and `/{subdir}/.env*` | any | Fake `.env` with per-IP planted credential — matches any path segment containing `.env` |
 | **Active traps** | | |
 | `/.git/*` | any | Infinite git-object chain (config → HEAD → refs → objects → loop) |
+| `/wp-admin/admin-ajax.php` | any | Plugin exploit surface: parses the account a privilege-escalation creates (recorded `origin='ajax'`, grants instantly at login), answers injection attempts with a fabricated `wp_users` dump |
+| `/wp-content/plugins/*/*.php` | any | Plugin entry points — the exploit the fingerprint bait advertises now lands somewhere instead of 404ing |
 | `/wp-admin/*` | any | Fake dashboard with canary links. POST: capture body |
 | `/admin/*` `/administrator/*` | any | Drupal/Django/Joomla post-login capture |
+| `/wp-json/batch/v1` | any | Real batch semantics — one row per bundled sub-request, so a 50-attempt amplification batch reads as 50 attempts |
 | `/wp-json/*` | any | GET: 200 `[]`. POST: capture body, return 201 |
+| `/.aws/credentials` `/.git-credentials` `/.gitconfig` `/.gitlab-ci.yml` `/.github/workflows/*` `/.npmrc` `/.docker/config.json` | any | Secret-file honeytokens with a per-IP planted credential (`origin='secret'`) |
+| `/phpinfo.php` | any | Full fake `phpinfo()` (~27 KB) with planted credentials in the environment block |
+| `/actuator` `/actuator/env` `/actuator/health` `/actuator/mappings` `/actuator/configprops` | any | Spring Boot Actuator with planted datasource credentials |
+| `/actuator/heapdump` | any | Valid HPROF header, then a slow trickle — the one endpoint attackers expect to be huge and slow. Holds a budget slot for the stream, released on hang-up; served whole and fast when the budget is spent |
+| `/phpmyadmin/*` `/pma/*` `/dbadmin/*` `/adminer.php` (15 spellings) | any | phpMyAdmin login form + credential capture |
+| `/wp-includes/js/*` `/wp-includes/css/*` `/wp-admin/css/*` | any | Core JS/CSS. A real WordPress always serves these; 404ing them identified the install as fake |
 | **Passive 404 + log** | | |
 | `/.svn/*` `/.hg/*` | any | VCS exposure |
-| `/.aws/*` `/.ssh/*` | any | Cloud key / SSH key probes |
-| `/actuator/*` `/_ignition/*` | any | Spring Boot / Laravel debug endpoints |
+| `/.ssh/*` | any | SSH key probes |
+| `/_ignition/*` | any | Laravel debug endpoint |
 | `/solr/*` `/server-status` `/server-info` | any | Service exposure |
 | `/composer.json` `/package.json` | GET | Dependency file probes |
-| `/phpinfo.php` `/shell.php` `/c99.php` `/r57.php` `/webshell.php` `/index.php` | any | PHP shell probes |
-| `/phpmyadmin/*` `/phpMyAdmin/*` `/pma/*` `/dbadmin/*` `/mysql/*` `/sqlmanager/*` `/adminer.php` | any | DB admin variants |
+| `/shell.php` `/c99.php` `/r57.php` `/webshell.php` | any | PHP shell probes |
+| `/index.php` | any | PHP probe; also serves an injected XML-RPC canary post back to the IP that injected it |
+| `/mysql/*` `/sqlmanager/*` | any | DB admin variants |
 | **Fingerprint bait** | | |
 | `/wp-includes/version.php` | any | Raw core `version.php` naming an outdated `$wp_version` |
 | `/readme.html` | any | Core readme naming the same version |
@@ -71,6 +94,81 @@ Beyond passive capture, RustyPot actively wastes attacker resources:
 | **Catch-all** | | |
 | anything else the edge routes here | any | Logged, then 404 — including method mismatches (`GET /xmlrpc.php`) |
 
+## Secret-file honeytokens
+
+Beyond `.env`, every file an attacker reads specifically to extract a
+credential is a honeytoken vector on the same model: `.aws/credentials`,
+`.aws/config`, `.git-credentials`, `.gitconfig`, `.gitlab-ci.yml`,
+`.github/workflows/*.yml`, `.npmrc`, `.docker/config.json`. Each carries a
+deterministic per-IP secret recorded with `origin='secret'`.
+
+`.aws/credentials` is the highest-intel member. Set `AWS_CANARY_ACCESS_KEY_ID`
+and `AWS_CANARY_SECRET_ACCESS_KEY` to a **real** AWS canary token (a
+permissionless IAM user with a CloudTrail alarm) and you learn the attacker's
+IP at the moment they *use* the key — the only signal here that survives their
+infrastructure rotation. A genuine canary is necessarily one fixed credential,
+so per-IP attribution comes from the `honeypot_event` row that recorded serving
+it, matched on time. Unset, the file carries a per-IP fake with the right shape.
+
+`/phpinfo.php` and `/actuator/env` plant the same way: both are pages whose
+whole purpose is dumping the process environment, so credentials in them look
+like a misconfiguration rather than bait.
+
+## Impersonated crawlers
+
+User-agents branded as ChatGPT-User, PerplexityBot, Amazonbot, GPTBot and
+friends show up requesting `.env`, `.aws/credentials` and login forms. No
+legitimate crawler does that; the branding is chosen because sites commonly
+allowlist those crawlers. Requests matching a crawler user-agent **on a path no
+crawler would request** get a distinct honeytoken prefix (`fk` → `fkx`), so a
+credential surfacing later carries "this actor impersonates AI crawlers" as a
+tooling fingerprint without needing a join.
+
+## Wire-level disguise
+
+Every HTML trap response is dressed as PHP-served WordPress by middleware —
+`X-Powered-By`, WordPress's fixed 1984 `Expires`, the no-cache pair, the
+`wordpress_test_cookie` on `wp-login.php`, and the `Link: rel="https://api.w.org/"`
+REST advertisement. `cloudflare-worker.js` strips `x-cloud-trace-context`
+on the way back, which the container cannot do itself and which otherwise
+reaches the client on every response. (`server` needs no rewrite: Cloudflare
+already replaces the origin's value with `server: cloudflare`.) Applied
+centrally so a new trap cannot forget it.
+
+`PUBLIC_HOSTNAME` supplies the host for the REST `Link` header. It is never
+derived from the request's `Host`: behind the edge that is the origin's own
+address, so echoing it would publish the backend URL and identify the stack.
+Unset, the header is simply omitted.
+
+## Content-injection canary
+
+`metaWeblog.newPost` probes inject a unique token as the post title and body,
+then search the web for it — if it appears, the site accepts unauthenticated
+publishing and joins a spam farm. RustyPot reports success with a post id and
+serves the token back at `/index.php?p=<id>`, which earns the follow-up visit.
+
+**The injected content is served only to the IP that injected it, only with
+`X-Robots-Tag: noindex`, and always HTML-escaped.** Attacker-supplied content
+reachable by anyone else, or indexable, would turn this service into a spam
+relay for whatever they inject next. The store is bounded at 512 posts.
+
+## Reading the data
+
+Query **`honeypot_event_live`**, not `honeypot_event`. The raw table also holds:
+
+- **Synthetic rows** (`synthetic = TRUE`) — written by the out-of-band backfill
+  importers, not captured by this service. They carry no POST body, no real
+  headers, and in the `drop-recovery` case an *inferred* source IP. Aggregating
+  them with live captures fabricates attacker behaviour that was never observed.
+- **Cloudflare-origin rows** — requests whose client is Cloudflare itself
+  (`cf-connecting-ip` is a Cloudflare address), not an attacker proxied through
+  it. `is_cloudflare_origin(source_ip)` tests this against the `cloudflare_ranges`
+  table; refresh that table from https://www.cloudflare.com/ips/ when the
+  published prefixes change.
+
+The view excludes both. `honeypot_event` remains the place to answer questions
+*about* capture coverage — which is what the provenance panel does.
+
 Every request that reaches the service is recorded, including ones it answers
 with 404 or 503 — the paths RustyPot does *not* yet trap are the feed for
 deciding which trap to build next, so they must not be dropped silently.

cloudflare-worker.js

@@ -23,25 +23,69 @@ function isHoneypotPath(pathname) {
   return HONEYPOT_PREFIXES.test(pathname) || ENV_ANYWHERE.test(pathname);
 }
 
+/**
+ * Cloudflare exposes the visitor's country on `request.cf`, NOT as a request
+ * header — `cf-ipcountry` only reaches an origin if the "Add visitor location
+ * headers" Managed Transform is enabled. It was not, so RustyPot's
+ * `cf_ipcountry` column was NULL on every row ever recorded and every geo panel
+ * was structurally empty rather than merely sparse. Forward it explicitly.
+ */
+function withGeoHeaders(request) {
+  const headers = new Headers(request.headers);
+  const cf = request.cf;
+  if (cf?.country) headers.set("cf-ipcountry", cf.country);
+  if (cf?.asn) headers.set("cf-asn", String(cf.asn));
+  if (cf?.asOrganization) headers.set("cf-as-org", cf.asOrganization);
+  return headers;
+}
+
+/**
+ * Strip the hosting platform's fingerprints from honeypot responses.
+ *
+ * Cloud Run stamps `x-cloud-trace-context` on every response and it survives
+ * all the way to the client — a Google Cloud tell on what is meant to look like
+ * a PHP host. The container cannot remove it; the header is added downstream of
+ * it, so the strip has to happen here.
+ *
+ * `server` is deliberately NOT rewritten. Cloudflare already replaces whatever
+ * the origin sent with `server: cloudflare`, which is what every
+ * Cloudflare-fronted site returns and therefore reveals nothing. Setting it to
+ * a fake nginx here would be overwritten anyway, and a value that disagreed
+ * with the rest of the CF response set would be more conspicuous than the
+ * default.
+ */
+function disguiseOrigin(upstream) {
+  const headers = new Headers(upstream.headers);
+  headers.delete("x-cloud-trace-context");
+  headers.delete("alt-svc");
+  return new Response(upstream.body, {
+    status: upstream.status,
+    statusText: upstream.statusText,
+    headers,
+  });
+}
+
 export default {
   async fetch(request, env) {
     const url = new URL(request.url);
+    const hasBody = request.method !== "GET" && request.method !== "HEAD";
 
     if (isHoneypotPath(url.pathname)) {
       const target = new URL(url.pathname + url.search, env.HONEYPOT_BACKEND);
-      return fetch(target, {
+      const upstream = await fetch(target, {
         method: request.method,
-        headers: request.headers,
-        body: request.method !== "GET" && request.method !== "HEAD" ? request.body : undefined,
+        headers: withGeoHeaders(request),
+        body: hasBody ? request.body : undefined,
         redirect: "manual",
       });
+      return disguiseOrigin(upstream);
     }
 
     const appTarget = new URL(url.pathname + url.search, env.APP_BACKEND);
     return fetch(appTarget, {
       method: request.method,
       headers: request.headers,
-      body: request.method !== "GET" && request.method !== "HEAD" ? request.body : undefined,
+      body: hasBody ? request.body : undefined,
       redirect: "manual",
     });
   },

migrations/004_event_provenance.sql

@@ -0,0 +1,94 @@
+-- Event provenance + plant/submission separation.
+--
+-- Three problems this fixes, all of which made the dashboard unreadable:
+--
+-- 1. `synthetic` — rows imported by out-of-band backfill scripts (they tagged
+--    themselves with a `source` key in request_headers: 'gap-recovery',
+--    'drop-recovery') were indistinguishable from live captures. They carry no
+--    POST body, no real headers, and in the 'drop-recovery' case an *inferred*
+--    source_ip. Aggregating them with real captures inflates every panel and
+--    fabricates attacker behaviour that was never observed.
+--
+-- 2. `planted_user` / `planted_pass` — the `.env` honeytoken wrote the
+--    credential IT PLANTED into `submitted_user`/`submitted_pass`, the columns
+--    that are supposed to mean "the attacker sent us this". Any
+--    "credentials captured" panel counted our own plants; the real number was
+--    roughly a quarter of what was displayed. Plants now have their own
+--    columns, and `submitted_*` means attacker-supplied, exclusively.
+--
+-- 3. `cloudflare_ranges` + `honeypot_event_live` — traffic originating from
+--    Cloudflare's own infrastructure (cf-connecting-ip is itself a Cloudflare
+--    address) is not an attacker. It was the single largest "attacker" path in
+--    the dashboard. The view filters it, and Grafana should query the view.
+
+ALTER TABLE honeypot_event
+    ADD COLUMN IF NOT EXISTS synthetic    BOOLEAN NOT NULL DEFAULT FALSE,
+    ADD COLUMN IF NOT EXISTS planted_user TEXT,
+    ADD COLUMN IF NOT EXISTS planted_pass TEXT;
+
+-- Backfill 1: tag every row an importer wrote.
+UPDATE honeypot_event
+   SET synthetic = TRUE
+ WHERE request_headers ? 'source'
+   AND NOT synthetic;
+
+-- Backfill 2: move honeytoken plants out of the submitted_* columns. A plant is
+-- an env-trap row whose password carries the honeytoken prefix; the attacker
+-- never typed it, we generated and served it.
+UPDATE honeypot_event
+   SET planted_user   = submitted_user,
+       planted_pass   = submitted_pass,
+       submitted_user = NULL,
+       submitted_pass = NULL
+ WHERE submitted_pass IS NOT NULL
+   AND submitted_pass ~ '^fk[A-Za-z0-9]'
+   AND path ~ '\.env'
+   AND planted_pass IS NULL;
+
+-- Published Cloudflare edge prefixes (https://www.cloudflare.com/ips/).
+-- Seeded rather than hardcoded in a WHERE clause so the list can be refreshed
+-- without a code change when Cloudflare adds a range.
+CREATE TABLE IF NOT EXISTS cloudflare_ranges (prefix INET PRIMARY KEY);
+
+INSERT INTO cloudflare_ranges (prefix) VALUES
+    ('173.245.48.0/20'), ('103.21.244.0/22'), ('103.22.200.0/22'),
+    ('103.31.4.0/22'),   ('141.101.64.0/18'), ('108.162.192.0/18'),
+    ('190.93.240.0/20'), ('188.114.96.0/20'), ('197.234.240.0/22'),
+    ('198.41.128.0/17'), ('162.158.0.0/15'),  ('104.16.0.0/13'),
+    ('104.24.0.0/14'),   ('172.64.0.0/13'),   ('131.0.72.0/22'),
+    ('2400:cb00::/32'),  ('2606:4700::/32'),  ('2803:f800::/32'),
+    ('2405:b500::/32'),  ('2405:8100::/32'),  ('2a06:98c0::/29'),
+    ('2c0f:f248::/32')
+ON CONFLICT (prefix) DO NOTHING;
+
+-- source_ip is TEXT on purpose (we log whatever the proxy sent, including
+-- malformed values). Cast defensively so one bad row can't error a dashboard.
+CREATE OR REPLACE FUNCTION try_inet(txt TEXT) RETURNS INET AS $$
+BEGIN
+    RETURN txt::INET;
+EXCEPTION WHEN others THEN
+    RETURN NULL;
+END;
+$$ LANGUAGE plpgsql IMMUTABLE RETURNS NULL ON NULL INPUT;
+
+-- TRUE when the client is Cloudflare itself rather than an attacker proxied
+-- through it. Note this tests source_ip, which extract_source_ip() sets from
+-- cf-connecting-ip — so a real attacker behind Cloudflare is NOT matched here.
+CREATE OR REPLACE FUNCTION is_cloudflare_origin(txt TEXT) RETURNS BOOLEAN AS $$
+    SELECT EXISTS (
+        SELECT 1 FROM cloudflare_ranges r WHERE try_inet(txt) <<= r.prefix
+    );
+$$ LANGUAGE sql STABLE;
+
+-- The view Grafana should point at: genuine, externally-originated captures.
+CREATE OR REPLACE VIEW honeypot_event_live AS
+    SELECT * FROM honeypot_event
+     WHERE NOT synthetic
+       AND NOT is_cloudflare_origin(source_ip);
+
+CREATE INDEX IF NOT EXISTS honeypot_event_synthetic_idx
+    ON honeypot_event (ts DESC) WHERE NOT synthetic;
+
+-- Replaces honeypot_event_has_creds_idx's intent: attacker-submitted only.
+CREATE INDEX IF NOT EXISTS honeypot_event_submitted_idx
+    ON honeypot_event (ts DESC) WHERE submitted_user IS NOT NULL AND NOT synthetic;