Problem
The DuckDuckGo HTML scraping provider (internal/web/search/providers/duckduckgo.go) fails in roughly 98% of real-world searches. DuckDuckGo aggressively blocks automated requests and rate-limits non-browser user-agents, returning empty result sets or CAPTCHA pages instead of results.
Current behavior:
User-Agent: NexusAI-WebSearch/1.0 (compatible; bot) is immediately flagged by DDG's bot detection- HTML structure (
result__a, result__snippet CSS classes) changes frequently, silently breaking the regex parser
- In
auto provider mode, DDG is tried as a fallback — but it almost always returns 0 hits without any error, making it appear as if the search succeeded
Impact
- Users in
auto mode silently get empty search results when all paid providers are unconfigured
- No error is surfaced, so the agent continues with no web context rather than reporting a failure
- There is no official DDG API (their paid API requires approval and has strict ToS restrictions)
Proposed solution
Option A — Remove entirely: Delete duckduckgo.go and remove DDG from the provider list. It provides no reliable value and adds noise to the auto fallback chain.
Option B — Mark as deprecated / disabled by default: Keep the file but exclude it from auto mode; allow opt-in via explicit config. Surface a deprecation warning in the Settings UI.
Option C — Replace with SearXNG as the default free provider: SearXNG is self-hosted, open, and returns structured JSON. Users who want a zero-config free option should be pointed there.
The recommended path is Option A (remove) combined with improving SearXNG onboarding (see companion issue).
Files affected
internal/web/search/providers/duckduckgo.go — provider implementationinternal/web/search/providers/providers.go — provider registry / auto mode chaininternal/nexustui/ui/dialog/settings.go — Settings UI provider list (DDG shown as "built-in fallback, no API key")
Problem
The DuckDuckGo HTML scraping provider (
internal/web/search/providers/duckduckgo.go) fails in roughly 98% of real-world searches. DuckDuckGo aggressively blocks automated requests and rate-limits non-browser user-agents, returning empty result sets or CAPTCHA pages instead of results.Current behavior:
User-Agent: NexusAI-WebSearch/1.0 (compatible; bot)is immediately flagged by DDG's bot detectionresult__a,result__snippetCSS classes) changes frequently, silently breaking the regex parserautoprovider mode, DDG is tried as a fallback — but it almost always returns 0 hits without any error, making it appear as if the search succeededImpact
automode silently get empty search results when all paid providers are unconfiguredProposed solution
Option A — Remove entirely: Delete
duckduckgo.goand remove DDG from the provider list. It provides no reliable value and adds noise to theautofallback chain.Option B — Mark as deprecated / disabled by default: Keep the file but exclude it from
automode; allow opt-in via explicit config. Surface a deprecation warning in the Settings UI.Option C — Replace with SearXNG as the default free provider: SearXNG is self-hosted, open, and returns structured JSON. Users who want a zero-config free option should be pointed there.
The recommended path is Option A (remove) combined with improving SearXNG onboarding (see companion issue).
Files affected
internal/web/search/providers/duckduckgo.go— provider implementationinternal/web/search/providers/providers.go— provider registry /automode chaininternal/nexustui/ui/dialog/settings.go— Settings UI provider list (DDG shown as "built-in fallback, no API key")