Version 4.2 | Author: Lucas
WebDigger performs static and dynamic analysis of websites to detect malicious patterns, obfuscation techniques, and threat indicators. Downloads HTML and JavaScript content, extracts URLs, scans for malware indicators including SocGholish, Parrot TDS, ClickFix payloads, and integrates with VirusTotal and URLScan.io for reputation and behavioral analysis.
- URL normalization and domain validation
- DNS resolution check
- URLScan.io check (async, if you have your own API key)
- VirusTotal reputation check (you need your own API key)
- Website content download (HTML, JavaScript)
- Local pattern scanning (obfuscation, malware indicators)
- SSL certificate analysis
- Domain age check via WHOIS
- URLScan.io results retrieval and analysis
- Summary generation with defanged output
Downloads main HTML page and all referenced JavaScript files. Extracts URLs from HTML content, inline scripts, and external JavaScript files. Filters out image URLs (.jpg, .jpeg, .png, .gif, .bmp, .webp, .svg, .ico). Normalizes URLs by adding https:// prefix if missing. Saves content to domain-named directory with extractedURLs.txt and extractedURLsDefanged.txt outputs.
Obfuscation patterns: hex-encoded arrays (_0x[0-9a-f]{4,6}), multiple consecutive hex characters, eval/Function with encoding functions (atob, unescape, decodeURIComponent), string concatenation chains, array join manipulation, split-reverse-join patterns.
Malware indicators: SocGholish browser update campaigns, notification manipulation, version requirement messages, download patterns, security manipulation, DOM manipulation (createElement, appendChild, document.write), delayed redirects, service worker abuse.
Suspicious functions: eval(), fromCharCode(), document.write(), charCodeAt(), unescape(), navigator properties.
Fingerprinting methods: navigator.userAgent, navigator.plugins, navigator.deviceMemory, navigator.hardwareConcurrency, AudioContext, Canvas fingerprinting, WebGL, RTCPeerConnection.
Base64 content: Detects Base64-encoded strings (minimum 30 chars), decodes and validates, filters benign patterns (data URIs, source maps), checks decoded content for suspicious patterns (eval, document.write, String.fromCharCode, hex escapes, function definitions).
String concatenation obfuscation: Split function names (ev+al, do+cu+ment, wi+ndow), function construction through concatenation, DOM manipulation obfuscation, protocol obfuscation, array-based string manipulation, hex/Unicode escape concatenation.
VirusTotal API v3: Domain reputation check, detection ratio (malicious/total engines), community score and votes, analysis statistics (malicious, suspicious, harmless, undetected), categories and tags, threat names, passive DNS resolutions, IP address details (ASN, owner, country, reputation, threat classification, JARM fingerprint, SSL certificate), related domains from IP resolutions.
URLScan.io: URL private submission for behavioral analysis if you have your own API key, DOM snapshot analysis, screenshot capture, request/response analysis, security header detection, resource tracking (domains, IPs, URLs), cookie analysis, console message extraction, certificate analysis, malicious indicator scoring.
Connects to port 443, retrieves certificate, extracts issuer, subject, validity dates, calculates days until expiry, flags expired certificates and certificates expiring within 30 days.
Uses python-whois to query domain creation date, calculates age in days, flags domains less than 30 days old as potentially suspicious, extracts registrar information.
Directory structure: Creates domain-named directory containing index.html, script0.js, script1.js (etc. for each JavaScript file), extractedURLs.txt, extractedURLsDefanged.txt, analysis_summary.txt, analysis_summary_defanged.txt.
Defanging: Converts http:// to hxxp://, https:// to hxxps://, replaces dots with [.] in domains and IPs.
Config file location: ~/.webdigger/config.json
Default settings: virustotal_api_key: None, urlscan_api_key: None, max_redirects: 10, check_ssl: True, check_domain_age: True, check_reputation: True, first_run: True, urlscan_wait_time: 300 seconds, urlscan_check_interval: 10 seconds.
First run setup: Prompts for VirusTotal API key (64 chars, validated against API), prompts for URLScan.io API key (36 chars), saves keys to config file, sets first_run to False.
--no-ssl-check: Skip SSL certificate analysis --skip-reputation: Skip VirusTotal and URLScan.io reputation checks --skip-domain-age: Skip domain age check --max-redirects N: Maximum redirects to follow (default: 10)
--urlscan-wait N: Maximum wait time for results in seconds (default: 300) --urlscan-interval N: Interval between result checks in seconds (default: 10) --set-urlscan-key KEY: Set URLScan.io API key
--quiet: Reduce output verbosity --json: Output results in JSON format (not fully implemented) --output-dir DIR: Custom output directory (not fully implemented)
--set-vtapi-key KEY: Set VirusTotal API key --reset-config: Reset configuration to defaults
Basic analysis: python web_digger.py https://example.com
Skip SSL check: python web_digger.py https://example.com --no-ssl-check
Skip reputation checks: python web_digger.py https://example.com --skip-reputation
Set API keys: python web_digger.py --set-vtapi-key YOUR_VT_KEY python web_digger.py --set-urlscan-key YOUR_URLSCAN_KEY
Custom redirect limit: python web_digger.py https://example.com --max-redirects 5
Extended URLScan wait time: python web_digger.py https://example.com --urlscan-wait 600
Hex encoding: Detects _0x variable patterns, hex array initialization, multiple consecutive hex characters, hex array access patterns.
Eval patterns: eval/Function with atob/unescape/decodeURIComponent, eval with string concatenation chains, Function constructor with encoding, eval with array access patterns.
String manipulation: String.fromCharCode with multiple codes, long base64 encoding/decoding, long URL-encoded strings, escape/unescape with long strings.
Advanced encoding: Long string concatenation chains (5+), complex array join manipulation, split-reverse-join patterns, number base conversion with concatenation.
SocGholish: Browser update campaign messages, notification manipulation, version requirement patterns, download package patterns, security manipulation, DOM manipulation, delayed redirects, service worker abuse.
Iframe injection: Blank iframe sources, data URI iframes, JavaScript in iframe source, hidden iframes (opacity:0, display:none), zero-height/width iframes.
Script tag analysis: Hex-encoded document.write, Base64 decode and eval, Unicode escape sequences, character code conversion, string escape/unescape manipulation, data/blob URLs as script sources.
DOM manipulation: createElement('script') usage, appendChild patterns, document.write with encoding functions.
Redirect chain: Follows redirects up to max_redirects limit, records each redirect URL, status codes, and Location headers.
Passive DNS: Extracts historical DNS resolutions from VirusTotal, shows IP addresses, hostnames, last resolved timestamps.
IP analysis: Categorizes IPs as cloud_provider (Cloudflare, AWS, Google, Azure, DigitalOcean, Linode), hosting_provider (hosting, datacenter), or other. Extracts ASN, owner, country, reputation, threat classification, JARM fingerprint, SSL certificate details, behavioral data (malicious URLs served, downloaded files, communicating files).
Domain resolution failures: Checks DNS resolution before analysis, displays error if domain not resolvable, attempts VirusTotal historical data lookup if domain unavailable, exits gracefully if domain cannot be resolved.
API errors: Handles VirusTotal 401 (invalid API key), 204 (quota exceeded), validates API key format and length, handles URLScan.io 429 (rate limit), 401 (unauthorized), 400 (bad request), retries with minimal config on 400 errors.
File I/O: Uses encoding='utf-8' with errors='replace' for content reading, handles encoding errors gracefully, creates output directories if missing.
requests: HTTP requests for website download and API calls beautifulsoup4: HTML parsing and content extraction tqdm: Progress bars for long operations termcolor: Colored terminal output whois: Domain age checking ssl, socket: SSL certificate analysis and DNS resolution argparse: Command line argument parsing json: Configuration file handling re: Pattern matching and regex operations math: Entropy calculations datetime: Timestamp handling
URLScan.io results: Requires waiting period (default 300 seconds), may timeout if scan takes longer than max_wait, results may not be immediately available.
VirusTotal API: Rate limits apply, free tier has request limits, requires valid API key for full functionality.
Domain age: WHOIS queries may fail for some TLDs, registrar information may be incomplete.
SSL analysis: Only checks port 443, does not validate certificate chain, does not check certificate revocation.
Pattern detection: May produce false positives on legitimate obfuscated code, pattern matching is regex-based and may miss novel obfuscation techniques.
Console output: Color-coded messages (green: info, yellow: warnings, red: alerts, cyan: headers), progress bars for long operations, structured sections with headers.
File output: Plain text files with UTF-8 encoding, defanged versions for safe sharing, summary files with timestamps and analysis metadata.
Defanging: All URLs and domains are defanged in extractedURLsDefanged.txt and analysis_summary_defanged.txt outputs. Defanging converts http:// to hxxp://, https:// to hxxps://, and replaces dots with [.] in domains and IPs.
API keys: Stored in ~/.webdigger/config.json with user-only permissions, validated on first use, not displayed in output.
Network requests: Uses realistic User-Agent headers, follows redirects up to configured limit, handles SSL certificate validation errors.
VirusTotal: Extracts related domains from passive DNS resolutions, IP address resolutions, subdomains, parent domains, sibling domains.
URLScan.io: Extracts related domains from request URLs, links, cookies, scripts, filters out main domain.
Output: Displays top 10 related domains in console, includes in analysis summary, saves to summary files.