Filter CDN/WAF ports by default and remove the portfilter module

[liquidsec]

Closes #3545

CDNs and WAFs like Cloudflare answer on a dozen or so ports (2052-2096, 8080, 8443, 8880) on every IP, so port sources like shodan_idb report all of them as open for every host behind them. Almost none of them lead anywhere, but each one becomes an OPEN_TCP_PORT, gets probed by http, and produces junk URLs. On one real subdomain-enum scan, 2,035 of shodan_idb's 2,560 open ports were non-80/443 ports on CDN/WAF-tagged hosts.

Filtering these out was already possible with the portfilter module, but it was opt-in (--exclude-cdn or a handful of presets). This makes the filter core behavior, on by default, and removes the module.

Behavior

• ScanEgress drops OPEN_TCP_PORT, URL_UNVERIFIED and URL events on hosts tagged cdn or waf, unless the port is 80 or 443.
• Ports you explicitly target are never filtered: -t host:8443 keeps that port and anything discovered on it.
• --allow-cdn-ports turns the filter off and restores the previous default behavior.

Config

cdn_port_filter:
  enabled: true
  tags: [cdn, waf]
  allowed_ports: [80, 443]

Breaking changes

• The default changes: CDN/WAF ports other than 80/443 are dropped unless --allow-cdn-ports or cdn_port_filter.enabled: false is set.
• The portfilter module is removed. Presets that list it, or set modules.portfilter.*, need to drop those lines; the equivalent settings are now cdn_port_filter.tags and cdn_port_filter.allowed_ports.
• --exclude-cdn / -ec is deprecated. It still parses but does nothing except log a warning, since the filter is now the default.

Other changes

• portfilter removed from the nuclei and lightfuzz presets that listed it.
• test_module_portfilter.py and test_exclude_cdn are replaced by test_cdn_port_filter.
• The CDN section of the tips docs is rewritten, and the generated docs are regenerated.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91%. Comparing base (484cfb6) to head (25eeee0).
⚠️ Report is 8 commits behind head on dev.

Additional details and impacted files

@@          Coverage Diff           @@
##             dev   #3534    +/-   ##
======================================
+ Coverage     91%     91%    +1%     
======================================
  Files        463     463            
  Lines      49758   49981   +223     
======================================
+ Hits       44890   45107   +217     
- Misses      4868    4874     +6     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

📊 Performance Benchmark Report

Comparing dev (baseline) vs cdn-port-filter (current)

📈 Detailed Results (All Benchmarks)

📋 Complete results for all benchmarks - includes both significant and insignificant changes

🧪 Test Name	📏 Base	📏 Current	📈 Change	🎯 Status
Bloom Filter Dns Mutation Tracking Performance	3.33ms	3.19ms	-4.3% ⚪	✅
Bloom Filter Large Scale Dns Brute Force	14.96ms	13.90ms	-7.1% ⚪	✅
Large Closest Match Lookup	327.35ms	337.97ms	+3.2% ⚪	✅
Realistic Closest Match Workload	179.92ms	177.65ms	-1.3% ⚪	✅
Event Memory Medium Scan	1402 B/event	1402 B/event	+0.0% ⚪	✅
Event Memory Large Scan	1527 B/event	1527 B/event	+0.0% ⚪	✅
Event Validation Full Scan Startup Small Batch	403.91ms	393.23ms	-2.6% ⚪	✅
Event Validation Full Scan Startup Large Batch	523.19ms	514.87ms	-1.6% ⚪	✅
Make Event Autodetection Small	22.20ms	22.18ms	-0.1% ⚪	✅
Make Event Autodetection Large	225.64ms	226.47ms	+0.4% ⚪	✅
Make Event Explicit Types	9.65ms	9.58ms	-0.8% ⚪	✅
Excavate Single Thread Small	4.010s	3.778s	-5.8% ⚪	✅
Excavate Single Thread Large	10.400s	10.224s	-1.7% ⚪	✅
Excavate Parallel Tasks Small	4.125s	3.890s	-5.7% ⚪	✅
Excavate Parallel Tasks Large	6.768s	6.475s	-4.3% ⚪	✅
Intercept Throughput Small	874.92ms	865.60ms	-1.1% ⚪	✅
Intercept Throughput Medium	871.39ms	862.14ms	-1.1% ⚪	✅
Dns Throughput Quiet	2.400s	2.413s	+0.5% ⚪	✅
Dns Throughput Loaded	1.653s	1.645s	-0.5% ⚪	✅
Dns Throughput Inherited	2.155s	2.294s	+6.5% ⚪	✅
Is Ip Performance	2.08ms	2.05ms	-1.4% ⚪	✅
Make Ip Type Performance	178.81µs	175.39µs	-1.9% ⚪	✅
Mixed Ip Operations	2.15ms	2.11ms	-1.8% ⚪	✅
Memory Use Web Crawl	1.2 MB	1.2 MB	+0.9% ⚪	✅
Memory Use Subdomain Enum	28.3 MB	28.3 MB	+0.0% ⚪	✅
Memory Use Deep Chain	5.0 MB	5.0 MB	+0.0% ⚪	✅
Memory Use Parallel Chains	14.0 MB	12.0 MB	-14.3% 🟢🟢	🚀
Scan Throughput 100	2.645s	2.815s	+6.4% ⚪	✅
Scan Throughput 1000	18.067s	19.431s	+7.6% ⚪	✅
Typical Queue Shuffle	4.97µs	4.93µs	-0.8% ⚪	✅
Priority Queue Shuffle	24.03µs	23.98µs	-0.2% ⚪	✅

🎯 Performance Summary

+ 1 improvement 🚀
  30 unchanged ✅

🔍 Significant Changes (>10%)

• Memory Use Parallel Chains: 14.3% 🚀 less memory

---

🐍 Python Version 3.11.17

[en0f]

See comment if merge fails >

Old CLI flag and module name now stop the scan. The PR removes -ec/--exclude-cdn and the portfilter module, and leaves no alias or deprecation path. I checked both on the PR branch:

bbot -t evilcorp.com -ec exits with bbot: error: unrecognized arguments: -ec.
A preset that lists modules: [portfilter] fails validation with Could not find scan module "portfilter".

That will break any saved preset, script or CI job that uses either one, including any BBOT Enterprise or bbot-server presets or command lines. If the hard break is intended, check those downstream repos before this ships. If it isn’t, the simplest fix is to keep -ec as a no-op alias that prints a deprecation warning, since filtering is now on by default.

[liquidsec]

-ec / --exclude-cdn is now a deprecated no-op that logs a warning instead of erroring (25eeee0). The portfilter removal stays a hard break, same as the other module removals in 3.0. It will be covered in the 3.1 release notes along with the cdn_port_filter.* replacement settings.