From 3432b9e82824c425712622d18100c483c8385bc3 Mon Sep 17 00:00:00 2001 From: Shantanu Mane Date: Tue, 21 Jul 2026 20:21:25 +0530 Subject: [PATCH 01/11] feat(dns): add NVS-backed ad-block with suffix matching (edge DNS phase 2) - Add DnsBlocklist with label-boundary suffix matching and atomic block counter - Seed blocklist from baked-in defaults on first boot, persist to NVS - Sinkhole blocked A queries to 0.0.0.0, NXDOMAIN for other qtypes - Insert blocklist check between local table and cache in resolution order - Add read-only GET /api/blocklist endpoint - Add phase 2 design spec - Replace default single-app partition table with a custom one (4MB app, 64KB nvs) to fix near-full app partition --- ...26-07-21-edge-dns-phase2-adblock-design.md | 133 +++++++++++++++ main/CMakeLists.txt | 3 +- main/dns_blocklist.cpp | 151 ++++++++++++++++++ main/dns_blocklist.h | 55 +++++++ main/dns_blocklist_defaults.h | 31 ++++ main/dns_server.cpp | 39 ++++- main/http_server.cpp | 47 ++++++ partitions.csv | 10 ++ sdkconfig | 6 +- sdkconfig.defaults | 5 + 10 files changed, 473 insertions(+), 7 deletions(-) create mode 100644 docs/superpowers/specs/2026-07-21-edge-dns-phase2-adblock-design.md create mode 100644 main/dns_blocklist.cpp create mode 100644 main/dns_blocklist.h create mode 100644 main/dns_blocklist_defaults.h create mode 100644 partitions.csv diff --git a/docs/superpowers/specs/2026-07-21-edge-dns-phase2-adblock-design.md b/docs/superpowers/specs/2026-07-21-edge-dns-phase2-adblock-design.md new file mode 100644 index 0000000..b50d9ba --- /dev/null +++ b/docs/superpowers/specs/2026-07-21-edge-dns-phase2-adblock-design.md @@ -0,0 +1,133 @@ +--- +tags: [decision, feature, arch] +status: active +supersedes: "" +related: [] +complexity: medium +--- + +# Edge DNS — Phase 2: Ad-block + +## What + +Adds a domain blocklist to the resolver built in Phase 1. A query for a blocked +domain (or any subdomain of one) is sinkholed — `0.0.0.0` for A queries, NXDOMAIN +for everything else — instead of reaching the cache or upstream forwarder. The +blocklist is seeded from a small baked-in default set on first boot, persisted to +NVS, and exposed read-only over HTTP alongside a running block counter. + +This is Phase 2 of the "edge DNS" vision scoped in `ARCHITECTURE.md`. Phase 3 +(metrics) is expected to read the block counter this phase introduces. + +## Why + +- **The problem:** the Phase-1 resolver forwards everything it doesn't recognize + locally, including ad/tracker domains — there is no way to stop resolving a name + the operator doesn't want resolved. +- **Where it sits in the resolution order:** local table → **blocklist** → cache → + forward. It must run before the cache/forwarder (a cached or freshly-forwarded + answer for a blocked name would defeat the block), and after the local table (the + static table's existing "always wins" invariant from Phase 1 must not regress — + an operator-configured local record should never be silently sinkholed). +- **Suffix matching, not exact-match:** real ad/tracker infrastructure lives on + subdomains (`ads.doubleclick.net`, `pixel.doubleclick.net`, …). Exact-match-only, + while textually what `ARCHITECTURE.md` first sketched, would miss the common + case and provide little real blocking value. Suffix matching on label boundaries + (not raw string suffix, to avoid `notdoubleclick.net` false-matching + `doubleclick.net`) is the standard hosts-list/Pi-hole semantic and is what this + phase implements instead. +- **NVS storage, read-only HTTP:** runtime persistence (survives reboot) without + yet building a write API. This keeps the phase's blast radius small — the + blocklist is populated once at boot (from NVS if present, else from the baked-in + defaults, which are then persisted) and never mutated afterward. That in turn + means the Phase-1 "no shared mutable state across tasks, no mutex" property + extends cleanly: both the DNS task and the HTTP task only *read* the set after + boot. The one piece of state that does change at runtime — the block counter — + is a single `std::atomic`, not a structure needing a lock. A write/edit HTTP + endpoint is deferred; adding one later is a small, additive change (mutate the + set, call the existing `save_to_nvs()`), not a redesign. + +## How (key parts) + +### Resolution order +1. Local static table (`find_dns_record`, unchanged) — still authoritative, still + first. +2. **Blocklist match** (new) — sinkhole and return before touching the cache. +3. Cache hit → serve (unchanged). +4. Miss → forward to upstream (unchanged). + +### Matching: suffix walk over an unordered_set +`DnsBlocklist::is_blocked(qname_lower)` tests the full lowercased name, then +repeatedly strips the leftmost label and retests, stopping at the root: +`a.b.doubleclick.net` → `b.doubleclick.net` → `doubleclick.net` → `net`. Each +step is an O(1) `unordered_set` lookup; stripping on label (`.`) boundaries rather +than raw string suffix gives correct semantics for free (`notdoubleclick.net` never +tests as `doubleclick.net`). Blocking a domain implicitly blocks every subdomain of +it — the standard ad-block-list convention. + +### Storage: NVS blob, seeded from a baked-in default list +- `main/dns_blocklist_defaults.h` — header-only `constexpr` array of ~15–30 + curated ad/tracker domains, in the same style as `dns_records.h`. +- `DnsBlocklist::load_from_nvs()` — opens NVS namespace `blocklist`, reads a single + newline-separated blob under key `list`. If the key doesn't exist (first boot), + populates the in-memory set from the defaults header and calls + `save_to_nvs()` to persist it, so every subsequent boot loads from NVS instead of + the compiled-in list. A single blob (rather than one NVS key per domain) sidesteps + NVS's 15-character key-name limit and keeps the read/write path to one call each. +- `DnsBlocklist::save_to_nvs()` — serializes the current set back to the same blob + key. Exists now so the load-time seeding path has something to call; a future + HTTP mutate endpoint calls the same function. + +### Counter and cross-task read access +`std::atomic blocks_total_` increments on every sinkholed query +(`record_block()`, relaxed ordering — an approximate counter for operational +visibility, not a consistency-critical value). `blocks_total()`, `size()`, and a +`const std::unordered_set& domains()` accessor are read by the new +`GET /api/blocklist` HTTP handler. This is a deliberate, narrow deviation from +Phase 1's "state lives inside the DNS task" pattern — justified because the set +itself is immutable after boot (only ever read from both tasks) and the counter +is atomic, so no mutex or queue is introduced. + +### HTTP: status-only +`GET /api/blocklist` mirrors the existing read-only `GET /api/records` handler +(`http_server.cpp`) — same cJSON pattern, no new response infrastructure: +```json +{ "count": 23, "blocked_total": 104, "domains": ["doubleclick.net", ...] } +``` + +### Wiring +`blocklist().load_from_nvs()` is called inside `dns_server_start()`, before the +DNS task is created — NVS itself is already initialized earlier by +`wifi_connect()` (`nvs_flash_init()` in `wifi_connect.cpp`), so by the time either +the DNS task or the HTTP server (started after `dns_server_start()` in +`main.cpp`) can touch the blocklist, it's already loaded. + +## Gotchas + +- **Load-before-serve ordering matters twice over:** the blocklist must be loaded + before the DNS task starts (or the first queries would race an empty set) *and* + before `http_server_start()` (or an early `GET /api/blocklist` could observe zero + domains). Both are satisfied by loading synchronously inside `dns_server_start()` + ahead of `xTaskCreate`, since `main.cpp` calls `dns_server_start()` before + `http_server_start()`. +- **Suffix matching must walk label boundaries, not raw string suffixes** — a naive + `ends_with(qname, blocked)` check would wrongly block `notdoubleclick.net` for a + blocklist entry of `doubleclick.net`. The label-strip-and-retest approach avoids + this by construction. +- **Local table precedence must not regress** (same invariant Phase 1 called out + for cache/forwarder) — the blocklist check sits after `find_dns_record`, so an + operator's own static record for a name always wins even if that name happens to + also appear on the blocklist. +- **NVS blob size:** a single blob under one key is simplest, but every save + rewrites the whole list; fine at the curated-list sizes this phase targets + (dozens of entries), would need reconsidering if a future phase imports large + (10k+) third-party lists. + +## Open threads + +- HTTP write/mutate endpoint (`POST`/`DELETE /api/blocklist`) — deferred; the NVS + `save_to_nvs()` path exists so this is additive, not a redesign. +- Large third-party list import (SPIFFS/LittleFS-backed, per `ARCHITECTURE.md`'s + future-scoping) — a different storage tier than NVS, out of scope here. +- Phase 3 (`/metrics`) is expected to expose `blocks_total()` alongside cache/ + forwarder counters once that phase starts. diff --git a/main/CMakeLists.txt b/main/CMakeLists.txt index 17da7a8..c4ba8e6 100644 --- a/main/CMakeLists.txt +++ b/main/CMakeLists.txt @@ -1,5 +1,6 @@ idf_component_register(SRCS "main.cpp" "wifi_connect.cpp" "dns_server.cpp" "dns_wire.cpp" - "dns_cache.cpp" "dns_forwarder.cpp" "http_server.cpp" + "dns_cache.cpp" "dns_forwarder.cpp" "dns_blocklist.cpp" + "http_server.cpp" INCLUDE_DIRS "." REQUIRES esp_wifi esp_netif esp_event nvs_flash lwip esp_http_server json esp_psram esp_timer) diff --git a/main/dns_blocklist.cpp b/main/dns_blocklist.cpp new file mode 100644 index 0000000..5d7c8eb --- /dev/null +++ b/main/dns_blocklist.cpp @@ -0,0 +1,151 @@ +#include "dns_blocklist.h" + +#include + +#include "dns_blocklist_defaults.h" +#include "esp_log.h" +#include "nvs.h" + +namespace { + +constexpr const char *TAG = "dns_blocklist"; +constexpr const char *NVS_NAMESPACE = "blocklist"; +constexpr const char *NVS_KEY_LIST = "list"; + +// Serializes a domain set to a single newline-separated blob — one NVS +// value rather than one key per domain, since NVS key names are capped at +// 15 characters and a domain can exceed that. +std::string serialize(const std::unordered_set &domains) +{ + std::ostringstream out; + for (const auto &domain : domains) { + out << domain << '\n'; + } + return out.str(); +} + +std::unordered_set deserialize(const std::string &blob) +{ + std::unordered_set domains; + std::istringstream in(blob); + std::string line; + while (std::getline(in, line)) { + if (!line.empty()) { + domains.insert(line); + } + } + return domains; +} + +} // namespace + +esp_err_t DnsBlocklist::load_from_nvs() +{ + nvs_handle_t handle; + esp_err_t err = nvs_open(NVS_NAMESPACE, NVS_READWRITE, &handle); + if (err != ESP_OK) { + ESP_LOGE(TAG, "nvs_open() failed: %s", esp_err_to_name(err)); + return err; + } + + size_t required_size = 0; + err = nvs_get_str(handle, NVS_KEY_LIST, nullptr, &required_size); + if (err == ESP_ERR_NVS_NOT_FOUND) { + // First boot: seed from the baked-in defaults and persist them, so + // every later boot loads from NVS instead of this compiled-in list. + domains_.clear(); + for (const char *domain : DNS_BLOCKLIST_DEFAULTS) { + domains_.insert(domain); + } + nvs_close(handle); + ESP_LOGI(TAG, "no NVS blocklist found, seeding %u default domain(s)", + static_cast(domains_.size())); + return save_to_nvs(); + } + if (err != ESP_OK) { + ESP_LOGE(TAG, "nvs_get_str() size query failed: %s", esp_err_to_name(err)); + nvs_close(handle); + return err; + } + + std::string blob(required_size, '\0'); + err = nvs_get_str(handle, NVS_KEY_LIST, blob.data(), &required_size); + nvs_close(handle); + if (err != ESP_OK) { + ESP_LOGE(TAG, "nvs_get_str() read failed: %s", esp_err_to_name(err)); + return err; + } + // required_size includes the NUL terminator; drop the trailing NUL(s) + // std::string's own storage would otherwise retain. + if (!blob.empty() && blob.back() == '\0') { + blob.resize(blob.size() - 1); + } + + domains_ = deserialize(blob); + ESP_LOGI(TAG, "loaded %u domain(s) from NVS", static_cast(domains_.size())); + return ESP_OK; +} + +esp_err_t DnsBlocklist::save_to_nvs() const +{ + nvs_handle_t handle; + esp_err_t err = nvs_open(NVS_NAMESPACE, NVS_READWRITE, &handle); + if (err != ESP_OK) { + ESP_LOGE(TAG, "nvs_open() failed: %s", esp_err_to_name(err)); + return err; + } + + std::string blob = serialize(domains_); + err = nvs_set_str(handle, NVS_KEY_LIST, blob.c_str()); + if (err == ESP_OK) { + err = nvs_commit(handle); + } + nvs_close(handle); + if (err != ESP_OK) { + ESP_LOGE(TAG, "failed to persist blocklist: %s", esp_err_to_name(err)); + } + return err; +} + +bool DnsBlocklist::is_blocked(const std::string &qname_lower) const +{ + size_t start = 0; + while (start < qname_lower.size()) { + std::string suffix = qname_lower.substr(start); + if (domains_.count(suffix) > 0) { + return true; + } + size_t dot = qname_lower.find('.', start); + if (dot == std::string::npos) { + break; + } + start = dot + 1; + } + return false; +} + +void DnsBlocklist::record_block() +{ + blocks_total_.fetch_add(1, std::memory_order_relaxed); +} + +uint32_t DnsBlocklist::blocks_total() const +{ + return blocks_total_.load(std::memory_order_relaxed); +} + +size_t DnsBlocklist::size() const +{ + return domains_.size(); +} + +const std::unordered_set &DnsBlocklist::domains() const +{ + return domains_; +} + +DnsBlocklist &blocklist() +{ + static DnsBlocklist instance; + return instance; +} diff --git a/main/dns_blocklist.h b/main/dns_blocklist.h new file mode 100644 index 0000000..4fea2d7 --- /dev/null +++ b/main/dns_blocklist.h @@ -0,0 +1,55 @@ +#pragma once + +#include +#include +#include +#include + +#include "esp_err.h" + +// Domain blocklist consulted after the local static table and before the +// TTL cache/forwarder (see dns_server.cpp and docs/superpowers/specs/ +// 2026-07-21-edge-dns-phase2-adblock-design.md). Unlike DnsCache, this is +// deliberately readable from both the DNS task and the HTTP task: the +// domain set is populated once at boot (load_from_nvs()) and never mutated +// afterward, so concurrent reads need no synchronization. The only value +// that changes at runtime is blocks_total_, which is a plain atomic counter +// rather than a structure needing a lock. +class DnsBlocklist { +public: + // Loads the domain set from NVS (namespace "blocklist", key "list": a + // single newline-separated blob). If the key doesn't exist yet (first + // boot), seeds the in-memory set from dns_blocklist_defaults.h and + // persists it via save_to_nvs(), so every later boot loads from NVS + // instead of the compiled-in list. Must be called before the DNS task + // starts and before the HTTP server starts (see dns_server_start()). + esp_err_t load_from_nvs(); + + // Serializes the current set back to the same NVS blob. Called by + // load_from_nvs() on first-boot seeding; also the hook a future HTTP + // mutate endpoint would call after editing the set. + esp_err_t save_to_nvs() const; + + // Suffix match on label boundaries: tests qname_lower, then repeatedly + // strips the leftmost label and retests (so blocking "doubleclick.net" + // also blocks "ads.doubleclick.net", but never "notdoubleclick.net"). + // qname_lower must already be lowercased (see lowercase_ascii in + // dns_cache.h) — this function does no case normalization itself. + bool is_blocked(const std::string &qname_lower) const; + + // Increments the block counter. Called once per sinkholed query. + void record_block(); + + uint32_t blocks_total() const; + size_t size() const; + const std::unordered_set &domains() const; + +private: + std::unordered_set domains_; + std::atomic blocks_total_{0}; +}; + +// Process-wide instance: the DNS task calls is_blocked()/record_block(), +// the HTTP task's /api/blocklist handler calls domains()/blocks_total(). +// Both only read domains_ after boot (see class comment above). +DnsBlocklist &blocklist(); diff --git a/main/dns_blocklist_defaults.h b/main/dns_blocklist_defaults.h new file mode 100644 index 0000000..a25d1d6 --- /dev/null +++ b/main/dns_blocklist_defaults.h @@ -0,0 +1,31 @@ +#pragma once + +#include + +// Curated default ad/tracker domains, used to seed the blocklist on first +// boot (see dns_blocklist.h). Suffix matching means each entry also blocks +// its subdomains (e.g. "doubleclick.net" covers "ads.doubleclick.net") — +// see DnsBlocklist::is_blocked. Editable by reflashing; once seeded, the +// live list lives in NVS and this array is no longer consulted. +constexpr std::array DNS_BLOCKLIST_DEFAULTS = {{ + "doubleclick.net", + "googlesyndication.com", + "googleadservices.com", + "google-analytics.com", + "googletagmanager.com", + "googletagservices.com", + "adservice.google.com", + "ads.yahoo.com", + "advertising.com", + "scorecardresearch.com", + "adnxs.com", + "outbrain.com", + "taboola.com", + "criteo.com", + "quantserve.com", + "moatads.com", + "rubiconproject.com", + "pubmatic.com", + "adsrvr.org", + "amazon-adsystem.com", +}}; diff --git a/main/dns_server.cpp b/main/dns_server.cpp index daf4357..997e15f 100644 --- a/main/dns_server.cpp +++ b/main/dns_server.cpp @@ -8,6 +8,7 @@ #include #include +#include "dns_blocklist.h" #include "dns_cache.h" #include "dns_forwarder.h" #include "dns_records.h" @@ -116,9 +117,11 @@ void handle_upstream_reply(int listen_sock, DnsForwarder &forwarder, DnsCache &c reply->client.client_addr_len, reply->qname_lower.c_str()); } -// Handles one query that arrived on the listen socket: local table, -// then cache, then forwarding — see the design doc for why local -// records take precedence over anything cached/forwarded. +// Handles one query that arrived on the listen socket: local table, then +// blocklist, then cache, then forwarding — see the design doc for why +// local records take precedence over anything cached/forwarded, and why +// the blocklist runs before the cache/forwarder (a cached or freshly +// forwarded answer for a blocked name would otherwise defeat the block). void handle_client_query(int listen_sock, DnsForwarder &forwarder, DnsCache &cache) { std::array rx_buffer; @@ -195,6 +198,27 @@ void handle_client_query(int listen_sock, DnsForwarder &forwarder, DnsCache &cac } std::string qname_lower = lowercase_ascii(*qname); + + if (blocklist().is_blocked(qname_lower)) { + blocklist().record_block(); + std::optional resp_len; + if (qtype == DNS_TYPE_A) { + constexpr std::array SINKHOLE_IP = {0, 0, 0, 0}; + resp_len = build_a_record_response(header->id, header->flags, question_section, + question_section_len, SINKHOLE_IP, tx_buffer, + sizeof(tx_buffer)); + } else { + resp_len = build_nxdomain_response(header->id, header->flags, question_section, + question_section_len, tx_buffer, sizeof(tx_buffer)); + } + if (resp_len) { + ESP_LOGI(TAG, "blocked '%s'", qname->c_str()); + send_response(listen_sock, tx_buffer, *resp_len, source_addr, socklen, + qname->c_str()); + } + return; + } + const dns_cache_entry_t *cached = cache.lookup(qname_lower, qtype, now_ms()); if (cached != nullptr) { auto resp_len = build_relayed_response(header->id, header->flags, question_section, @@ -308,5 +332,14 @@ void dns_server_task(void *) void dns_server_start() { + // Loaded synchronously, before the task (and before http_server_start(), + // called after this returns — see main.cpp) — so neither the first + // query nor the first /api/blocklist request can race an empty list. + esp_err_t blocklist_err = blocklist().load_from_nvs(); + if (blocklist_err != ESP_OK) { + ESP_LOGW(TAG, "blocklist failed to load (%s); starting with an empty list", + esp_err_to_name(blocklist_err)); + } + xTaskCreate(dns_server_task, "dns_server", 6144, nullptr, 5, nullptr); } diff --git a/main/http_server.cpp b/main/http_server.cpp index a1c9801..8fb3a5a 100644 --- a/main/http_server.cpp +++ b/main/http_server.cpp @@ -6,6 +6,7 @@ #include #include "cJSON.h" +#include "dns_blocklist.h" #include "dns_records.h" #include "esp_http_server.h" #include "esp_log.h" @@ -108,6 +109,51 @@ constexpr httpd_uri_t RECORDS_URI = { .user_ctx = nullptr, }; +esp_err_t blocklist_get_handler(httpd_req_t *req) +{ + cJSON *root = cJSON_CreateObject(); + if (root == nullptr) { + ESP_LOGE(TAG, "Failed to allocate JSON object"); + return httpd_resp_send_500(req); + } + + const DnsBlocklist &bl = blocklist(); + cJSON_AddNumberToObject(root, "count", static_cast(bl.size())); + cJSON_AddNumberToObject(root, "blocked_total", static_cast(bl.blocks_total())); + + cJSON *domains = cJSON_CreateArray(); + if (domains == nullptr) { + ESP_LOGE(TAG, "Failed to allocate JSON array"); + cJSON_Delete(root); + return httpd_resp_send_500(req); + } + for (const auto &domain : bl.domains()) { + cJSON_AddItemToArray(domains, cJSON_CreateString(domain.c_str())); + } + cJSON_AddItemToObject(root, "domains", domains); + + char *json_str = cJSON_PrintUnformatted(root); + if (json_str == nullptr) { + ESP_LOGE(TAG, "Failed to serialize JSON"); + cJSON_Delete(root); + return httpd_resp_send_500(req); + } + + httpd_resp_set_type(req, "application/json"); + esp_err_t ret = httpd_resp_send(req, json_str, HTTPD_RESP_USE_STRLEN); + + cJSON_free(json_str); + cJSON_Delete(root); + return ret; +} + +constexpr httpd_uri_t BLOCKLIST_URI = { + .uri = "/api/blocklist", + .method = HTTP_GET, + .handler = blocklist_get_handler, + .user_ctx = nullptr, +}; + } // namespace void http_server_start() @@ -118,6 +164,7 @@ void http_server_start() ESP_ERROR_CHECK(httpd_start(&server, &config)); ESP_ERROR_CHECK(httpd_register_uri_handler(server, &ROOT_URI)); ESP_ERROR_CHECK(httpd_register_uri_handler(server, &RECORDS_URI)); + ESP_ERROR_CHECK(httpd_register_uri_handler(server, &BLOCKLIST_URI)); ESP_LOGI(TAG, "HTTP server listening on port %d", config.server_port); } diff --git a/partitions.csv b/partitions.csv new file mode 100644 index 0000000..6dd35b4 --- /dev/null +++ b/partitions.csv @@ -0,0 +1,10 @@ +# Name, Type, SubType, Offset, Size, Flags +# Note: if you have increased the bootloader size, make sure to update the offsets to avoid overlap +# +# Custom table (16MB flash): default IDF single-app table (24K nvs, 1M app) +# was sized for small flash chips and left the app partition at 98% full +# after Phase 2 (ad-block). Sized generously here so Phases 3/4 don't repeat +# this — see docs/superpowers/specs/2026-07-21-edge-dns-phase2-adblock-design.md. +nvs, data, nvs, 0x9000, 0x10000, +phy_init, data, phy, , 0x1000, +factory, app, factory, , 0x400000, diff --git a/sdkconfig b/sdkconfig index a83be81..65c80d7 100644 --- a/sdkconfig +++ b/sdkconfig @@ -555,13 +555,13 @@ CONFIG_ESPTOOLPY_MONITOR_BAUD=115200 # # Partition Table # -CONFIG_PARTITION_TABLE_SINGLE_APP=y +# CONFIG_PARTITION_TABLE_SINGLE_APP is not set # CONFIG_PARTITION_TABLE_SINGLE_APP_LARGE is not set # CONFIG_PARTITION_TABLE_TWO_OTA is not set # CONFIG_PARTITION_TABLE_TWO_OTA_LARGE is not set -# CONFIG_PARTITION_TABLE_CUSTOM is not set +CONFIG_PARTITION_TABLE_CUSTOM=y CONFIG_PARTITION_TABLE_CUSTOM_FILENAME="partitions.csv" -CONFIG_PARTITION_TABLE_FILENAME="partitions_singleapp.csv" +CONFIG_PARTITION_TABLE_FILENAME="partitions.csv" CONFIG_PARTITION_TABLE_OFFSET=0x8000 CONFIG_PARTITION_TABLE_MD5=y # end of Partition Table diff --git a/sdkconfig.defaults b/sdkconfig.defaults index 32101a1..dbaa85a 100644 --- a/sdkconfig.defaults +++ b/sdkconfig.defaults @@ -1,6 +1,11 @@ CONFIG_ESPTOOLPY_FLASHSIZE_16MB=y CONFIG_HTTPD_MAX_REQ_HDR_LEN=1024 +# Custom partitions.csv: the default single-app table (24K nvs, 1M app) left +# the app partition at 98% full after Phase 2 — see partitions.csv. +CONFIG_PARTITION_TABLE_CUSTOM=y +CONFIG_PARTITION_TABLE_CUSTOM_FILENAME="partitions.csv" + # PSRAM: assumes the common ESP32-S3 16MB-flash module pairing, WROOM-1 # N16R8 (8MB octal PSRAM). If the actual module uses quad PSRAM instead, # CONFIG_SPIRAM_IGNORE_NOTFOUND below keeps boot from hard-failing — From db9fc4e54f0126ff20390d54253b89ca900cb146 Mon Sep 17 00:00:00 2001 From: Shantanu Mane Date: Tue, 21 Jul 2026 22:21:11 +0530 Subject: [PATCH 02/11] feat(dns): add Prometheus metrics and mDNS responder (edge DNS phases 3-4) - Add DnsMetrics singleton with atomic query/cache/forward/SERVFAIL counters - Add upstream-latency histogram sourced from DnsForwarder's in-flight slots - Expose GET /metrics in Prometheus plaintext format - Add mDNS responder advertising the device as edge-dns.local (A + AAAA) - Advertise an _http._tcp service pointing at the existing dashboard - Bring up IPv6 link-local addressing on the STA netif for AAAA support - Add espressif/mdns as a managed component dependency - Ignore managed_components/ (regenerated by the component manager) - Document both phases in ARCHITECTURE.md, README.md, and design specs --- .gitignore | 1 + ARCHITECTURE.md | 43 ++++-- README.md | 12 +- dependencies.lock | 20 +++ ...26-07-21-edge-dns-phase3-metrics-design.md | 141 ++++++++++++++++++ .../2026-07-21-edge-dns-phase4-mdns-design.md | 117 +++++++++++++++ main/CMakeLists.txt | 4 +- main/dns_forwarder.cpp | 16 ++ main/dns_forwarder.h | 6 + main/dns_metrics.cpp | 92 ++++++++++++ main/dns_metrics.h | 95 ++++++++++++ main/dns_server.cpp | 23 ++- main/http_server.cpp | 109 ++++++++++++++ main/idf_component.yml | 2 + main/main.cpp | 2 + main/mdns_responder.cpp | 22 +++ main/mdns_responder.h | 8 + main/wifi_connect.cpp | 14 +- sdkconfig | 41 +++++ 19 files changed, 745 insertions(+), 23 deletions(-) create mode 100644 dependencies.lock create mode 100644 docs/superpowers/specs/2026-07-21-edge-dns-phase3-metrics-design.md create mode 100644 docs/superpowers/specs/2026-07-21-edge-dns-phase4-mdns-design.md create mode 100644 main/dns_metrics.cpp create mode 100644 main/dns_metrics.h create mode 100644 main/idf_component.yml create mode 100644 main/mdns_responder.cpp create mode 100644 main/mdns_responder.h diff --git a/.gitignore b/.gitignore index ed29c10..9ecc0a9 100644 --- a/.gitignore +++ b/.gitignore @@ -1,6 +1,7 @@ main/wifi_credentials.h build/ +managed_components/ cmake-build-debug/ cmake-build-debug-esp-idf/ .idea/ diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 6e54fb0..8ae524b 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -2,9 +2,9 @@ ## What this is -`mini_dns` is ESP32-S3 firmware moving from proof-of-concept toward a marketable "edge DNS" appliance: a single device that connects to Wi-Fi, serves a small hardcoded set of hostnames authoritatively, forwards everything else to an upstream recursive resolver with TTL caching, and serves an HTTP page + JSON API showing the static record table. It was built incrementally — Wi-Fi → raw UDP → DNS parsing → single-record response → multi-record + NXDOMAIN → HTTP server → JSON API → page wired to the API → **forwarding resolver + cache (Phase 1 of the edge-DNS roadmap)** — each step flashed and confirmed on real hardware before moving on. +`mini_dns` is ESP32-S3 firmware moving from proof-of-concept toward a marketable "edge DNS" appliance: a single device that connects to Wi-Fi, serves a small hardcoded set of hostnames authoritatively, forwards everything else to an upstream recursive resolver with TTL caching, sinkholes ad/tracker domains, exposes Prometheus metrics, advertises itself via mDNS, and serves an HTTP page + JSON API showing the static record table. It was built incrementally — Wi-Fi → raw UDP → DNS parsing → single-record response → multi-record + NXDOMAIN → HTTP server → JSON API → page wired to the API → **forwarding resolver + cache (Phase 1)** → **NVS-backed ad-block (Phase 2)** → **Prometheus `/metrics` (Phase 3)** → **mDNS responder (Phase 4)** — each step flashed and confirmed on real hardware before moving on. -**Records are still compile-time constants, read-only after boot** — there is no persistence, no provisioning UI, no OTA, no runtime record editing, no auth. Ad-block, metrics/OTEL export, and mDNS are planned as later phases (see Future scoping), not built yet. If you're extending this, read the Non-Goals section before adding anything that smells like a "real" feature. +**Records are still compile-time constants, read-only after boot** — there is no persistence, no provisioning UI, no OTA, no runtime record editing, no auth. If you're extending this, read the Non-Goals section before adding anything that smells like a "real" feature. ## Target hardware / toolchain @@ -25,31 +25,38 @@ sequenceDiagram wifi_connect-->>app_main: returns once IP obtained app_main->>dns_task: dns_server_start() [xTaskCreate, returns immediately] app_main->>httpd: http_server_start() [httpd_start, returns immediately] + app_main->>mdns: mdns_responder_start() [mdns_init, returns immediately] app_main--)app_main: app_main() returns, its task exits - Note over dns_task,httpd: Both keep running independently.
No shared mutable state between them. + Note over dns_task,httpd: All three keep running independently.
No shared mutable state between them. ``` -Three independent runtime pieces exist after boot, with **no synchronization between them**: +Four independent runtime pieces exist after boot, with **no synchronization between them**: -1. **Wi-Fi event handling** (`wifi_connect.cpp`) — registered event handlers keep running for the life of the device (auto-reconnect on disconnect), but `wifi_connect()` itself only blocks `app_main` during initial bring-up. +1. **Wi-Fi event handling** (`wifi_connect.cpp`) — registered event handlers keep running for the life of the device (auto-reconnect on disconnect, plus IPv6 link-local bring-up as of Phase 4), but `wifi_connect()` itself only blocks `app_main` during initial IPv4 bring-up. 2. **DNS task** (`dns_server.cpp`) — a task we create explicitly via `xTaskCreate`, because raw BSD sockets have no framework driving an accept/receive loop for us. 3. **HTTP server task(s)** (`http_server.cpp`) — `esp_http_server` owns and drives its own task(s) internally; we only configure and start it. +4. **mDNS responder task** (`mdns_responder.cpp`, Phase 4) — the `mdns` component owns and drives its own internal task, the same shape as `esp_http_server`; we only configure and start it. It reads whatever A/AAAA addresses the STA netif currently holds and shares no mutable state with the other three pieces. `DNS_RECORDS` is still `constexpr`, shared read-only across tasks with no locking needed. **As of the forwarding resolver (Phase 1), the DNS task also owns a cache and an in-flight query table** (`DnsCache`, `DnsForwarder`) — these *are* mutable, but they're only ever touched from within the single DNS task's `select()` loop, so the "no shared mutable state across tasks" property still holds; it's just no longer literally "no mutable state," only "no mutable state shared *between* tasks." +Two pieces of state *do* deliberately cross the DNS task / HTTP task boundary, and both do it without a mutex: `DnsBlocklist`'s domain set (Phase 2) is populated once at boot and never mutated again, so concurrent reads need no synchronization; and `DnsMetrics` (Phase 3) is a set of `std::atomic` counters/gauges — the DNS task increments/sets them inline, the HTTP task's `/metrics` handler reads a `snapshot()`. Both are narrow, deliberate exceptions to "no shared mutable state," justified the same way: either the shared value is immutable after boot, or it's a single atomic word. + ## Component map | File | Responsibility | |---|---| -| `main/main.cpp` | Boot sequence: logs detected PSRAM size → `wifi_connect()` → `dns_server_start()` → `http_server_start()`. | -| `main/wifi_connect.h/.cpp` | Blocking Wi-Fi station bring-up with hardcoded credentials. Retries forever on disconnect (2s backoff), no give-up state. | +| `main/main.cpp` | Boot sequence: logs detected PSRAM size → `wifi_connect()` → `dns_server_start()` → `http_server_start()` → `mdns_responder_start()`. | +| `main/wifi_connect.h/.cpp` | Blocking Wi-Fi station bring-up with hardcoded credentials. Retries forever on disconnect (2s backoff), no give-up state. As of Phase 4, also triggers IPv6 link-local address creation on `WIFI_EVENT_STA_CONNECTED` (non-blocking, doesn't gate boot readiness) so mDNS has an address to advertise as AAAA. | | `main/wifi_credentials.h` | `WIFI_SSID`/`WIFI_PASSWORD` constants. **Gitignored** — does not exist on a fresh checkout, must be recreated. | | `main/dns_records.h` | The hardcoded hostname→IPv4 table (`DNS_RECORDS`, a `constexpr std::array`). Pure data, no logic, no dependencies beyond ``/`` — intentionally kept dependency-free. | | `main/dns_wire.h/.cpp` | Pure DNS wire-format functions: header/question parsing, name skipping, answer-section TTL scanning, and response building (A-record, NXDOMAIN, and a generic "relay a captured answer section" builder used by both cache hits and forwarded replies). No I/O, no FreeRTOS/lwIP dependency — the natural home for host-side unit tests. | | `main/dns_cache.h/.cpp` | TTL cache (`DnsCache`) keyed by (lowercased qname, qtype), storing captured answer-section bytes + a clamped TTL. Single-owner (the DNS task); no mutex. | | `main/dns_forwarder.h/.cpp` | Upstream UDP client socket (`DnsForwarder`) plus the in-flight query table that correlates upstream replies back to the client that asked, using a slot-index + generation-counter transaction ID scheme. Single-owner; no mutex. | -| `main/dns_server.h/.cpp` | UDP/53 listener in its own FreeRTOS task, running a `select()` loop over the listen socket and the forwarder's upstream socket. Resolves each query via local table → cache → forward, in that order, and reaps timed-out forwarded queries into SERVFAIL. | -| `main/http_server.h/.cpp` | `esp_http_server` bring-up with two routes: `GET /` (static HTML page with an inline `fetch()` script) and `GET /api/records` (JSON array via cJSON). Unchanged by Phase 1 — still only reflects the static table, not cache/forwarding activity. | +| `main/dns_server.h/.cpp` | UDP/53 listener in its own FreeRTOS task, running a `select()` loop over the listen socket and the forwarder's upstream socket. Resolves each query via local table → blocklist → cache → forward, in that order, and reaps timed-out forwarded queries into SERVFAIL. Also the sole writer of `DnsMetrics` (Phase 3). | +| `main/dns_blocklist.h/.cpp` | NVS-backed ad-block domain set (`DnsBlocklist`), suffix-matched on label boundaries. Populated once at boot, never mutated afterward — safely read from both the DNS and HTTP tasks with no lock; only its atomic block counter changes at runtime. | +| `main/dns_metrics.h/.cpp` | Runtime counters/gauges/histogram for `/metrics` (`DnsMetrics`, Phase 3) — query/cache/forward/SERVFAIL counts, an upstream-latency histogram, and cache/in-flight occupancy gauges. Written only by the DNS task, read only by the HTTP task's `/metrics` handler, all via `std::atomic` — the same cross-task shape as `DnsBlocklist`'s counter, generalized into its own module. | +| `main/http_server.h/.cpp` | `esp_http_server` bring-up with four routes: `GET /` (static HTML page with an inline `fetch()` script), `GET /api/records` (JSON array via cJSON), `GET /api/blocklist` (Phase 2), and `GET /metrics` (Prometheus plaintext, Phase 3). | +| `main/mdns_responder.h/.cpp` | mDNS responder bring-up (Phase 4, `espressif/mdns` managed component) — advertises the device as `edge-dns.local` (A + AAAA) plus an `_http._tcp` service pointing at the dashboard. Owns its own internal task, no shared mutable state with the DNS/HTTP tasks. | ## Data flow @@ -89,6 +96,10 @@ Local-table responses still echo the request's question section verbatim and reu **HTTP request** (`http_server.cpp`): - `GET /` → static HTML shell with an inline `