Skip to content

Replace massdns with blastdns 2.0 - #3457

Open
liquidsec wants to merge 31 commits into
devfrom
blastdns-2.0
Open

liquidsec wants to merge 31 commits into
devfrom
blastdns-2.0

Conversation

@liquidsec

@liquidsec liquidsec commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Replaces massdns with blastdns 2.0 for DNS brute-forcing, and drops the massdns dependency entirely.

Changes

  • Brute-force DNS through blastdns instead of massdns; massdns dependency and references removed
  • Brute-force gets its own retry budget, a short per-query timeout, and persistent sockets (one per resolver rather than one per in-flight query)
  • New --custom-nameservers flag, replacing the system resolvers for normal resolution
  • Resolver diagnostics reported when brute-force delivery drops
  • The DNS helper is now built during scan prep. Previously an invalid dns.nameservers value surfaced as a per-event error inside dnsresolve after setup had already reported success, so a typo produced a green "completed" scan with zero results instead of an abort.

baddns 2.5.0

2.5.0 is the release that widened baddns's blastdns pin, so the temporary [tool.uv] override-dependencies block is gone. BBOT now requires baddns~=2.5.0 and caps blastdns at <2.1 to match baddns's own ceiling, so a future blastdns 2.1 cannot silently break module dependency installs the way <1.10 did.

Adapting to what 2.5.0 changed:

  • NS gained a lame-delegation finding, raising its ceiling to HIGH/CONFIRMED in SUBMODULE_MAX_SEVERITY and SUBMODULE_MAX_CONFIDENCE
  • The new DELEGATION submodule is added to both tables at HIGH/CONFIRMED
  • The bigcartel signature the test fixture relied on was dropped upstream, so the fixture now matches the surviving nuclei signature (409 plus error code: 1001)
  • The aws-bucket signature was rescoped to require an amazonaws.com subject. Direct mode uses the target as the subject, so that cannot be satisfied against a localhost test server; the baddns_direct fixture now uses a signature with no CNAME identifier, which is the class a Cloudflare-fronted host actually hits. Real STORAGE_BUCKET detection is unaffected, since those hosts genuinely contain amazonaws.com.

Behavior changes

DELEGATION is enabled by default, both in the module's default submodule list and in the baddns and baddns-heavy presets. It checks _acme-challenge, _dmarc and 15 common DKIM selector labels for dangling delegations and reports at HIGH severity. Two filters keep the volume down: a wildcard CNAME probe per depth drops labels that only resolve because of a wildcard, and generic findings (a CNAME to a name that merely does not exist, which is routine for DKIM selector pairs) are discarded, leaving only claimable outcomes.

baddns's WordMatcher is now passed through. It compiles every signature's word matchers into one YARA ruleset at setup and matches a response against all of them in a single pass, instead of evaluating each signature's words in turn per response. This is what baddns 2.5.0's new yara-python dependency is for, and baddns's own CLI already does it. It is a cache with the previous path as fallback, and was checked for equivalence against that path across 2,044 signature/response pairs with no divergence.

New DELEGATION module tests cover the three label kinds, wildcard suppression, and the generic-finding filter.

docs/scanning/advanced.md is deliberately not regenerated here. It embeds bbot --help, which CI generates on a newer Python whose argparse formats help text differently, so the daily docs job will pick up --custom-nameservers.

liquidsec added 12 commits July 29, 2026 13:36
Replaces the massdns subprocess in the brute helper with a dedicated blastdns
client. It gets its own client rather than sharing the one behind normal
resolution, because brute-forcing runs against a large public resolver list
instead of the system resolvers and needs its own limits.

Uses resolve_batch_full rather than resolve_batch so unanswered queries stay
visible and get reported: a query that runs out of retries is a silently
missed subdomain, which is the one failure brute-forcing must not hide.

Retires dns.brute_threads, which was a massdns passthrough, in favor of
brute_concurrency, brute_inflight_per_resolver, and brute_rate_limit. Enables
the startup resolver probe so dead entries in the public list are dropped once
instead of costing queries all scan.

Drops the now-unused module argument from helpers.dns.brute().

Tests move off the run_live massdns monkeypatch: mock_dns now drives
brute-forcing directly, since _mock_dns swaps the brute client too. Assertions
are unchanged, and the wildcard canary case uses a regex mock key.
Deletes DEP_MASSDNS and the deps_common entries that pulled it in, so BBOT no
longer builds massdns from source at install time.

Documents the new brute-force knobs in place of dns.brute_threads, and swaps
the deps_ansible example in the module howto to masscan, which BBOT still
builds the same way.
uv.lock still pins 1.9.1 and cannot be regenerated until 2.0.0 is published,
so dependency resolution will fail until then. Regenerate the lock once the
release is out.
Supplied nameservers replace the OS-detected ones for normal resolution.
DNSHelper.system_resolvers keeps meaning "what the OS reported" and the new
DNSHelper.resolvers is what actually gets queried.

resolver_file follows the effective set, so external tools pointed at it
(nuclei) honor the setting too. The brute-force pool likewise excludes the
effective set rather than the system one, preserving the reason that exclusion
exists: brute traffic must not rate-limit our own resolution.

Brute-forcing is unaffected and keeps its own dns.brute_nameservers list.

Validation is blastdns's, so an unusable address is reported once, from one
place, rather than reimplementing resolver parsing in Python.
Brute-forcing inherited dns.timeout (5s), which is right for careful resolution
of in-scope hosts but wrong here: blastdns abandons an attempt when the timeout
expires rather than overlapping retransmissions, so retries x timeout serializes.
At 20 retries that meant up to 100s on a single stubborn name, and measured out
at 2.4x slower than a 1-retry run.

massdns re-asks every 500ms for up to 50 tries, a 25s budget per name. A 1s
timeout with 20 retries gives a comparable 20s budget without the serialization
cost. Not shorter than 1s, since blastdns discards late answers that massdns
would still accept.
A brute run ends when its slowest query ends, and stragglers burned
timeout x retries serially: the last 1% of queries took a quarter of each
run's wall clock, paid 43 times per scan. Dropping the timeout to 0.5s
cuts per-run time 32% (3.26s -> 2.23s over 5 interleaved reps) with the
resolved set unchanged, because retries recover anything abandoned early.
Going lower shrinks the usable resolver pool by half.

Also exposes blastdns's persistent_socket, which trades most of that
throughput for ~4x less connection-tracking state, for networks with a
small NAT table.
Per-query sockets bind a fresh source port each time, so connection-tracking
state grows with query volume: a single-domain scan peaked at 114,342
conntrack entries, 44% of a stock Linux table and several times over what a
consumer router holds. One socket per resolver bounds that by the resolver
count instead -- same scan peaked at 6,974, and found the identical 154
subdomains in 158s against 168s.
The subdomain-enum preset still set it, so every bundled-preset validation
failed and took test_python_api and test_modules_basic_checks with it.
dnsbrute also no longer has common deps now that massdns is gone, so the
preload assertion moves to portscan, which still uses them.
# Conflicts:
#	pyproject.toml
#	uv.lock
@github-actions

github-actions Bot commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

📊 Performance Benchmark Report

Comparing dev (baseline) vs blastdns-2.0 (current)

📈 Detailed Results (All Benchmarks)

📋 Complete results for all benchmarks - includes both significant and insignificant changes

🧪 Test Name 📏 Base 📏 Current 📈 Change 🎯 Status
Bloom Filter Dns Mutation Tracking Performance 3.23ms 2.96ms -8.1% ⚪ ✅
Bloom Filter Large Scale Dns Brute Force 13.44ms 11.75ms -12.6% 🟢🟢 🚀
Large Closest Match Lookup 254.18ms 262.34ms +3.2% ⚪ ✅
Realistic Closest Match Workload 139.78ms 137.32ms -1.8% ⚪ ✅
Event Memory Medium Scan 1402 B/event 1403 B/event +0.1% ⚪ ✅
Event Memory Large Scan 1527 B/event 1527 B/event +0.0% ⚪ ✅
Event Validation Full Scan Startup Small Batch 324.67ms 323.10ms -0.5% ⚪ ✅
Event Validation Full Scan Startup Large Batch 413.29ms 412.43ms -0.2% ⚪ ✅
Make Event Autodetection Small 16.16ms 16.09ms -0.4% ⚪ ✅
Make Event Autodetection Large 164.31ms 160.24ms -2.5% ⚪ ✅
Make Event Explicit Types 6.78ms 6.61ms -2.6% ⚪ ✅
Excavate Single Thread Small 3.295s 3.000s -8.9% ⚪ ✅
Excavate Single Thread Large 7.818s 7.573s -3.1% ⚪ ✅
Excavate Parallel Tasks Small 3.477s 3.373s -3.0% ⚪ ✅
Excavate Parallel Tasks Large 5.132s 4.956s -3.4% ⚪ ✅
Intercept Throughput Small 789.61ms 795.68ms +0.8% ⚪ ✅
Intercept Throughput Medium 779.77ms 780.49ms +0.1% ⚪ ✅
Dns Throughput Quiet 2.023s 2.023s +0.0% ⚪ ✅
Dns Throughput Loaded 1.383s 1.369s -1.0% ⚪ ✅
Dns Throughput Inherited 1.720s 1.754s +2.0% ⚪ ✅
Is Ip Performance 1.74ms 1.74ms -0.2% ⚪ ✅
Make Ip Type Performance 159.35µs 160.64µs +0.8% ⚪ ✅
Mixed Ip Operations 1.80ms 1.84ms +1.8% ⚪ ✅
Memory Use Web Crawl 1.2 MB 1.1 MB -3.4% ⚪ ✅
Memory Use Subdomain Enum 28.3 MB 28.3 MB +0.0% ⚪ ✅
Memory Use Deep Chain 5.0 MB 5.1 MB +0.4% ⚪ ✅
Memory Use Parallel Chains 13.4 MB 12.5 MB -6.7% ⚪ ✅
Scan Throughput 100 2.263s 2.149s -5.0% ⚪ ✅
Scan Throughput 1000 14.022s 13.955s -0.5% ⚪ ✅
Typical Queue Shuffle 4.17µs 4.19µs +0.5% ⚪ ✅
Priority Queue Shuffle 21.74µs 20.99µs -3.5% ⚪ ✅

🎯 Performance Summary

+ 1 improvement 🚀
  30 unchanged ✅

🔍 Significant Changes (>10%)

  • Bloom Filter Large Scale Dns Brute Force: 12.6% 🚀 faster

🐍 Python Version 3.11.17

liquidsec added a commit that referenced this pull request Sep 28, 2026
baddns pins blastdns<1.10, so the module-level pip install cannot resolve
against the 2.x that #3457 puts in the venv. Revert once baddns widens it.
@liquidsec liquidsec self-assigned this Oct 1, 2026
@liquidsec liquidsec added this to the BBOT 3.1 - violent_barbara milestone Oct 1, 2026
- Require baddns ~=2.5.0 and cap blastdns at <2.1 to match baddns's own
  ceiling; drop the temporary uv override-dependencies block
- NS is now HIGH/CONFIRMED, and the new DELEGATION submodule is added to
  the severity/confidence tables
- Enable DELEGATION by default, in the module and both presets
- Pass baddns's batched YARA word matcher
- Update fixtures for the rewritten bigcartel and aws-bucket signatures
- Add DELEGATION module tests
@liquidsec
liquidsec marked this pull request as ready for review October 9, 2026 18:55
@codecov

codecov Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 96.01770% with 18 lines in your changes missing coverage. Please review.
✅ Project coverage is 91%. Comparing base (4e4c3b7) to head (67424f1).
⚠️ Report is 2 commits behind head on dev.

Files with missing lines Patch % Lines
bbot/core/helpers/dns/brute.py 80% 14 Missing ⚠️
bbot/test/test_step_1/test_dns.py 99% 4 Missing ⚠️
Additional details and impacted files
@@          Coverage Diff           @@
##             dev   #3457    +/-   ##
======================================
+ Coverage     91%     91%    +1%     
======================================
  Files        463     463            
  Lines      50142   50459   +317     
======================================
+ Hits       45255   45573   +318     
+ Misses      4887    4886     -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

* Let already-resolved events skip ahead in dnsresolve's queue

* Resolve a host's minimal records on every dnsresolve pass

* Make portscan test mock report ports for any covering target
@singlerider
singlerider requested a review from en0f October 21, 2026 04:09

@en0f en0f left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocker: the PR can’t merge into dev because of conflicts. GitHub reports it as not mergeable, and no CI checks have run on the latest commit (73eb9b1), so CI hasn’t tested it yet. Two files conflict:

pyproject.toml: the PR raises blastdns to >=2.0.1,<2.1 and baddns to ~=2.5.0. Meanwhile dev raised pycryptodome to <5, fastapi to <0.144.0 and ruff to 0.17.0, and added a [tool.uv] section that sets a minimum virtualenv version. The fix is to keep both sides’ changes.
uv.lock: regenerate it with uv lock once pyproject.toml is fixed.

@liquidsec

Copy link
Copy Markdown
Contributor Author

Blocker: the PR can’t merge into dev because of conflicts. GitHub reports it as not mergeable, and no CI checks have run on the latest commit (73eb9b1), so CI hasn’t tested it yet. Two files conflict:

pyproject.toml: the PR raises blastdns to >=2.0.1,<2.1 and baddns to ~=2.5.0. Meanwhile dev raised pycryptodome to <5, fastapi to <0.144.0 and ruff to 0.17.0, and added a [tool.uv] section that sets a minimum virtualenv version. The fix is to keep both sides’ changes. uv.lock: regenerate it with uv lock once pyproject.toml is fixed.

We're in a holding pattern here anyway until #3474 merges, which now targets this branch. At that time we will sync up all of the dependencies.

singlerider
singlerider previously approved these changes Oct 21, 2026

@singlerider singlerider left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Clean massdns removal, conflicts from prior review resolved, brute client mocked so tests stay offline.

@liquidsec

Copy link
Copy Markdown
Contributor Author

Resolved in d4b2e68 (merged dev into this branch). pyproject.toml keeps both sides: dev's pycryptodome, fastapi, ruff and virtualenv changes, plus this PR's blastdns and baddns versions. uv.lock has been regenerated and matches. CI passes on 67424f1. Could you re-review?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add --custom-nameservers option Replace massdns with blastdns within dnsbrute

3 participants