Skip to content

Fix telemetry provenance, add secrets/exploit traps, budget slow responses - #1

Merged
ShaneMain merged 2 commits into
mainfrom
traps/telemetry-fixes-and-new-traps
Aug 29, 2026
Merged

Fix telemetry provenance, add secrets/exploit traps, budget slow responses#1
ShaneMain merged 2 commits into
mainfrom
traps/telemetry-fixes-and-new-traps

Conversation

@ShaneMain

Copy link
Copy Markdown
Owner

Why

Six weeks of captures showed the dashboard was mostly measuring things that were not attacker behaviour, and that the busiest attacker objective had no trap at all.

Telemetry correctness (migration 004)

before after
"credentials captured" 730 175
events counted as attacker traffic 3828 1465
cf_ipcountry populated 0 rows forwarded from request.cf
  • synthetic flags backfilled rows — 56% of the table, no POST bodies, inferred IPs in the drop-recovery case.
  • planted_user/planted_pass separate honeytokens we serve from credentials attackers send. The .env trap wrote plants into submitted_*.
  • honeypot_event_live excludes Cloudflare-origin traffic, which was the top "attacker" path (a 2-hour cadence from CF's own ranges).
  • The xmlrpc parser captured post titles as usernames on publishing calls.

New traps

Secrets harvesting was ~46% of real traffic and answered 404. Now trapped: .aws/credentials (optional real AWS canary token), .git-credentials, .gitconfig, .gitlab-ci.yml, .github/workflows, .npmrc, .docker/config.json, fake phpinfo(), Spring Boot Actuator + slow heapdump, admin-ajax.php and plugin PHP endpoints, wp-json/batch/v1, phpMyAdmin across 15 spellings, and an XML-RPC content-injection canary.

The canary is served only to the injecting IP, noindex and escaped — attacker content reachable by anyone else would make this a spam relay.

Credibility

Responses now carry PHP/WordPress headers via middleware and the Worker strips server: Google Frontend. Core JS/CSS is served; 404ing jquery.js identified the install as fake.

Resource safety

  • Shared slow-response budget bounds concurrent held responses; over budget, traps answer immediately and log a delay of 0 so response_delay_ms stays truthful.
  • Body limits were inert — set on empty Routers before routes were added, so axum never wrapped them. A 100 KB POST to /wp-login.php returned 200; now 413.
  • Batch amplification capped at 25. One 141 KB request wrote 2000 rows, 8 MB of duplicated body text, and held a pool connection for 38s. Now 25 rows, 4 KB, 0.48s.
  • The unbudgeted heapdump path buffered the full dump at peak concurrency — the fallback was the memory-hungry branch on a 256 MiB container.
  • admin-ajax only registers instant-grant pairs for real privilege escalation; any pair used to qualify, which was a two-request oracle for identifying the honeypot.
  • Planted AWS keys and phpass hashes no longer share a constant tail across every IP.

Verification

fmt --check clean, clippy --all-targets -D warnings clean, 117 tests (from 56). All four migrations applied to a throwaway Postgres and every trap exercised end-to-end, including the full kit chain: escalation → instant-grant verification login → 302.

https://claude.ai/code/session_01MSq1R1A6xQ9dk1CzBVf6Xk

…w responses

Analysis of six weeks of captures found the dashboard was mostly measuring
things that were not attacker behaviour, and that the busiest attacker
objective had no trap at all. This addresses both, plus the correctness issues
the new code surfaced.

Telemetry (migration 004):
- `synthetic` flags rows written by the backfill importers. They were 56% of
  the table, carry no POST body, and in the drop-recovery case an inferred
  source IP; aggregating them with live captures fabricated behaviour that was
  never observed.
- `planted_user`/`planted_pass` separate honeytokens WE serve from credentials
  an attacker sends. The `.env` trap wrote its plants into `submitted_*`,
  inflating every credential metric roughly fourfold (730 -> 175).
- `cloudflare_ranges` + `honeypot_event_live` exclude traffic originating from
  Cloudflare's own infrastructure, which was the single largest "attacker" path
  in the dashboard.
- The Worker forwards `cf-ipcountry` from `request.cf`: it is not a request
  header by default, so the column was NULL on every row ever recorded.
- The xmlrpc parser took the last two <string> values, which on publishing
  calls (metaWeblog.newPost) captured the post title and body instead of the
  credentials.

Traps for what attackers actually do (secrets harvesting was ~46% of real
traffic and answered 404):
- Secret-file honeytokens: .aws/credentials (with an optional real AWS canary
  token), .git-credentials, .gitconfig, .gitlab-ci.yml, .github/workflows,
  .npmrc, .docker/config.json.
- Fake phpinfo() and Spring Boot Actuator, both with planted credentials in the
  environment they exist to dump; /actuator/heapdump streams slowly.
- admin-ajax.php and plugin PHP endpoints: parses the backdoor account a
  privilege-escalation creates, answers injection with a fabricated wp_users
  dump. Closes the observed drop-off where scanners read the version bait and
  left because the advertised plugin endpoint 404'd.
- wp-json/batch/v1 with real batch semantics, so amplification batches arrive
  instead of stopping at the capability probe.
- phpMyAdmin login across the 15 probed spellings.
- XML-RPC content-injection canary, served back only to the injecting IP, with
  noindex and escaped: attacker content reachable by anyone else would make
  this a spam relay.

Credibility:
- Responses carry PHP/WordPress headers (X-Powered-By, the 1984 Expires,
  wordpress_test_cookie, the REST Link) via middleware, and the Worker strips
  `server: Google Frontend`. A bot reading headers could previously tell this
  was not PHP before submitting anything.
- Core JS/CSS is served; 404ing jquery.js identified the install as fake.

Resource safety:
- A shared slow-response budget bounds how many responses may be held open at
  once. Cloud Run gives 80 concurrent requests across 3 instances; unbounded
  tarpits would fill the pool and stop the service recording new probes. Over
  budget, traps answer immediately and log a delay of 0, so response_delay_ms
  reflects time actually spent.
- Body limits now apply: they were set on empty Routers before routes were
  added, so axum never wrapped them and everything used the 2 MiB default.
- Batch captures are capped at 25. One 141 KB request previously wrote 2000
  rows and 8 MB of duplicated body text and held a pool connection 38 seconds.
- The unbudgeted heapdump path buffered the full dump; it ran at peak
  concurrency, making the fallback the memory-hungry branch on a 256 MiB
  container.
- admin-ajax only registers an instant-grant pair for real privilege
  escalation. Any pair used to qualify, which was a two-request oracle for
  identifying the honeypot and a way to skip stuffer churn on common passwords.
- Planted AWS keys and phpass hashes derive entropy across their full length;
  they previously shared a constant tail across every IP.

Claude-Session: https://claude.ai/code/session_01MSq1R1A6xQ9dk1CzBVf6Xk
The Link header was built from the request's Host. Behind the edge Worker the
origin sees the Cloud Run hostname, so every HTML response through
fillerkiller.app carried:

  link: <https://fillerkiller-honeypot-….run.app/wp-json/>; rel="https://api.w.org/"

That is worse than the fingerprint it was added to fix. It identifies the stack
as Cloud Run rather than PHP, and it hands attackers the backend address, which
reaches the honeypot directly with no edge rate limiting and no Worker routing.

The hostname now comes from PUBLIC_HOSTNAME, validated as a bare hostname. When
unset the header is omitted — harmless, where leaking the origin is not.

Also corrects the Worker: `server` is not rewritten, because Cloudflare already
replaces the origin's value with `server: cloudflare` before anything reaches a
client. Verified against the live edge — `server: Google Frontend` never left
the origin, so that half of the disguise was solving a problem that did not
exist. `x-cloud-trace-context` does survive to the client and is still stripped.

Claude-Session: https://claude.ai/code/session_01MSq1R1A6xQ9dk1CzBVf6Xk
@ShaneMain
ShaneMain merged commit 5c64fa1 into main Aug 29, 2026
1 check passed
@ShaneMain
ShaneMain deleted the traps/telemetry-fixes-and-new-traps branch August 29, 2026 14:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant