Feature: Performance optimization for large-scale scans

[netanelcyber]

Feature: Performance Optimization for Large-Scale Scans\n\n### Overview\nOptimize adpentest to handle large Active Directory environments with thousands of users/computers.\n\n### Performance Goals\n- Current: Handle ~1000 objects (good)\n- Target: Handle 50,000+ objects efficiently\n- Expected speedup: 3-5x faster scans\n\n### Optimization Strategies\n\n#### 1. Connection Pooling\npython\n# Current: New LDAP connection per query\n# Proposed: Reuse connection pool\n\nclass LDAPConnectionPool:\n def __init__(self, min_size=5, max_size=20):\n self.pool = queue.Queue(maxsize=max_size)\n \n def acquire(self):\n \"\"\"Get connection from pool\"\"\"\n \n def release(self, conn):\n \"\"\"Return connection to pool\"\"\"\n\n\n#### 2. Batch Processing\npython\n# Current: Query users one-by-one\n# Proposed: Batch LDAP queries\n\n# Fetch 1000 users in single query instead of 1000 queries\nfilter_str = '(|(uid=user1)(uid=user2)...(uid=user1000))'\nresults = ldap_conn.search(search_base, filter_str)\n\n\n#### 3. Caching Strategy\npython\n# Cache frequently accessed data\n\nclass ADCache:\n def __init__(self, ttl=300): # 5 minute TTL\n self.cache = {}\n self.ttl = ttl\n \n def get_domain_info(self, domain):\n \"\"\"Cached domain info queries\"\"\"\n \n def get_user_spns(self, username):\n \"\"\"Cached SPN lookups\"\"\"\n\n\n#### 4. Parallel Tool Execution\npython\n# Current: 16 workers\n# Proposed: Adaptive worker pool based on system resources\n\ndef get_optimal_workers():\n \"\"\"Calculate optimal thread count\"\"\"\n cpu_count = os.cpu_count()\n memory_gb = psutil.virtual_memory().total / (1024**3)\n \n # 1 worker per CPU + 1 per 2GB RAM\n return min(cpu_count + int(memory_gb / 2), 64)\n\n\n#### 5. DNS Query Optimization\npython\n# Current: Individual SRV queries\n# Proposed: Batch DNS queries with caching\n\nclass DNSCache:\n def __init__(self):\n self.cache = {} # Query -> [results]\n self.negative_cache = set() # Failed queries\n \n def bulk_resolve(self, hostnames):\n \"\"\"Resolve multiple hostnames in parallel\"\"\"\n\n\n#### 6. Memory Optimization\npython\n# Use generators instead of lists for large result sets\n\ndef enumerate_users(ldap_conn, domain):\n \"\"\"Yields users instead of returning full list\"\"\"\n # Avoids loading 50k+ users into memory at once\n for user in ldap_conn.search_stream(...):\n yield user\n\n\n#### 7. Smart Tool Selection\npython\n# Skip redundant tools based on results\n\nclass SmartToolExecutor:\n def should_execute(self, tool, results_so_far):\n \"\"\"Determine if tool provides new information\"\"\"\n # Skip bloodhound if no AD structure detected\n # Skip kerberos tools if no DCs found\n # Skip email tools if SMTP not discovered\n\n\n### Benchmarking\n\n#### Current Performance (v1.1.2a)\n\nEnvironment: 1000 users, 100 computers\nTime: ~45 seconds\nThreads: 16\nMemory: 150MB\n\n\n#### Target Performance\n\nEnvironment: 50,000 users, 5000 computers\nTime: < 5 minutes (estimated)\nThreads: 32-64 (adaptive)\nMemory: < 500MB (with generators)\n\n\n### Implementation Tasks\n- [ ] Implement connection pooling\n- [ ] Add batch LDAP queries\n- [ ] Implement caching layer\n- [ ] Optimize tool execution order\n- [ ] Add DNS query batching\n- [ ] Convert to generators for large result sets\n- [ ] Add memory profiling\n- [ ] Benchmark improvements\n- [ ] Document performance tuning\n\n### Configuration Options\nbash\n# Allow users to tune for their environment\nadpentest --target domain.local \\\n --max-threads 64 \\\n --batch-size 1000 \\\n --cache-ttl 300 \\\n --memory-limit 1000 \\\n --disable-caching # For very dynamic environments\n\n\n### Monitoring & Profiling\npython\n# Add performance metrics to output\n\n{\n \"performance\": {\n \"total_time\": 45.2,\n \"ldap_time\": 15.3,\n \"dns_time\": 8.2,\n \"tool_execution_time\": 21.7,\n \"avg_thread_utilization\": 0.85,\n \"cache_hits\": 1234,\n \"cache_misses\": 56,\n \"peak_memory_mb\": 150,\n \"queries_per_second\": 45.3\n }\n}\n\n\n### Priority\nMedium-High - Important for enterprise environments\n\n### Related Issues\n- #1: DNS timeout handling\n- #5: Large environment testing\n