Fix telemetry provenance, add secrets/exploit traps, budget slow responses - #1
Merged
Merged
Conversation
…w responses Analysis of six weeks of captures found the dashboard was mostly measuring things that were not attacker behaviour, and that the busiest attacker objective had no trap at all. This addresses both, plus the correctness issues the new code surfaced. Telemetry (migration 004): - `synthetic` flags rows written by the backfill importers. They were 56% of the table, carry no POST body, and in the drop-recovery case an inferred source IP; aggregating them with live captures fabricated behaviour that was never observed. - `planted_user`/`planted_pass` separate honeytokens WE serve from credentials an attacker sends. The `.env` trap wrote its plants into `submitted_*`, inflating every credential metric roughly fourfold (730 -> 175). - `cloudflare_ranges` + `honeypot_event_live` exclude traffic originating from Cloudflare's own infrastructure, which was the single largest "attacker" path in the dashboard. - The Worker forwards `cf-ipcountry` from `request.cf`: it is not a request header by default, so the column was NULL on every row ever recorded. - The xmlrpc parser took the last two <string> values, which on publishing calls (metaWeblog.newPost) captured the post title and body instead of the credentials. Traps for what attackers actually do (secrets harvesting was ~46% of real traffic and answered 404): - Secret-file honeytokens: .aws/credentials (with an optional real AWS canary token), .git-credentials, .gitconfig, .gitlab-ci.yml, .github/workflows, .npmrc, .docker/config.json. - Fake phpinfo() and Spring Boot Actuator, both with planted credentials in the environment they exist to dump; /actuator/heapdump streams slowly. - admin-ajax.php and plugin PHP endpoints: parses the backdoor account a privilege-escalation creates, answers injection with a fabricated wp_users dump. Closes the observed drop-off where scanners read the version bait and left because the advertised plugin endpoint 404'd. - wp-json/batch/v1 with real batch semantics, so amplification batches arrive instead of stopping at the capability probe. - phpMyAdmin login across the 15 probed spellings. - XML-RPC content-injection canary, served back only to the injecting IP, with noindex and escaped: attacker content reachable by anyone else would make this a spam relay. Credibility: - Responses carry PHP/WordPress headers (X-Powered-By, the 1984 Expires, wordpress_test_cookie, the REST Link) via middleware, and the Worker strips `server: Google Frontend`. A bot reading headers could previously tell this was not PHP before submitting anything. - Core JS/CSS is served; 404ing jquery.js identified the install as fake. Resource safety: - A shared slow-response budget bounds how many responses may be held open at once. Cloud Run gives 80 concurrent requests across 3 instances; unbounded tarpits would fill the pool and stop the service recording new probes. Over budget, traps answer immediately and log a delay of 0, so response_delay_ms reflects time actually spent. - Body limits now apply: they were set on empty Routers before routes were added, so axum never wrapped them and everything used the 2 MiB default. - Batch captures are capped at 25. One 141 KB request previously wrote 2000 rows and 8 MB of duplicated body text and held a pool connection 38 seconds. - The unbudgeted heapdump path buffered the full dump; it ran at peak concurrency, making the fallback the memory-hungry branch on a 256 MiB container. - admin-ajax only registers an instant-grant pair for real privilege escalation. Any pair used to qualify, which was a two-request oracle for identifying the honeypot and a way to skip stuffer churn on common passwords. - Planted AWS keys and phpass hashes derive entropy across their full length; they previously shared a constant tail across every IP. Claude-Session: https://claude.ai/code/session_01MSq1R1A6xQ9dk1CzBVf6Xk
The Link header was built from the request's Host. Behind the edge Worker the origin sees the Cloud Run hostname, so every HTML response through fillerkiller.app carried: link: <https://fillerkiller-honeypot-….run.app/wp-json/>; rel="https://api.w.org/" That is worse than the fingerprint it was added to fix. It identifies the stack as Cloud Run rather than PHP, and it hands attackers the backend address, which reaches the honeypot directly with no edge rate limiting and no Worker routing. The hostname now comes from PUBLIC_HOSTNAME, validated as a bare hostname. When unset the header is omitted — harmless, where leaking the origin is not. Also corrects the Worker: `server` is not rewritten, because Cloudflare already replaces the origin's value with `server: cloudflare` before anything reaches a client. Verified against the live edge — `server: Google Frontend` never left the origin, so that half of the disguise was solving a problem that did not exist. `x-cloud-trace-context` does survive to the client and is still stripped. Claude-Session: https://claude.ai/code/session_01MSq1R1A6xQ9dk1CzBVf6Xk
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Six weeks of captures showed the dashboard was mostly measuring things that were not attacker behaviour, and that the busiest attacker objective had no trap at all.
Telemetry correctness (migration 004)
cf_ipcountrypopulatedrequest.cfsyntheticflags backfilled rows — 56% of the table, no POST bodies, inferred IPs in the drop-recovery case.planted_user/planted_passseparate honeytokens we serve from credentials attackers send. The.envtrap wrote plants intosubmitted_*.honeypot_event_liveexcludes Cloudflare-origin traffic, which was the top "attacker" path (a 2-hour cadence from CF's own ranges).New traps
Secrets harvesting was ~46% of real traffic and answered 404. Now trapped:
.aws/credentials(optional real AWS canary token),.git-credentials,.gitconfig,.gitlab-ci.yml,.github/workflows,.npmrc,.docker/config.json, fakephpinfo(), Spring Boot Actuator + slow heapdump,admin-ajax.phpand plugin PHP endpoints,wp-json/batch/v1, phpMyAdmin across 15 spellings, and an XML-RPC content-injection canary.The canary is served only to the injecting IP, noindex and escaped — attacker content reachable by anyone else would make this a spam relay.
Credibility
Responses now carry PHP/WordPress headers via middleware and the Worker strips
server: Google Frontend. Core JS/CSS is served; 404ingjquery.jsidentified the install as fake.Resource safety
response_delay_msstays truthful./wp-login.phpreturned 200; now 413.admin-ajaxonly registers instant-grant pairs for real privilege escalation; any pair used to qualify, which was a two-request oracle for identifying the honeypot.Verification
fmt --checkclean,clippy --all-targets -D warningsclean, 117 tests (from 56). All four migrations applied to a throwaway Postgres and every trap exercised end-to-end, including the full kit chain: escalation → instant-grant verification login → 302.https://claude.ai/code/session_01MSq1R1A6xQ9dk1CzBVf6Xk