From 9fdbf6ea8919d85729ba67a654cc8b203653ab20 Mon Sep 17 00:00:00 2001 From: Aryan Bansal Date: Wed, 21 Oct 2026 18:45:33 +0530 Subject: [PATCH 01/46] fix(core): unbreak the outreach loop, then make it fast Five defects made the product non-functional. All were verified by reading the code and executing it, not inferred. Blockers - Mailbox daily cap: `sent_today` was a lifetime counter, never reset, so every mailbox bricked after `daily_limit` sends, forever. It is now a calendar-day (IST) counter and the rolling-24h budget is the real gate. - Send gate: the sender excluded only "bounced", so `invalid`, `gibberish`, `disposable` and `no_mx` addresses were all mailed - a ~30% hard-bounce rate against Gmail's 5% bulk-sender ceiling. Ten undeliverable states are now refused; un-enriched leads stay eligible. - Sequence scheduler: multi-step sequences only advanced when a human re-clicked Launch. Added a 5-minute in-process loop plus a Celery beat entry, and "waiting on a step delay" is now distinct from "completed". - Maps `q=` split: the text query carried the city, so Google returned one city-wide set and every viewport duplicated the top 20. Category-only queries take the same 8 requests from 34 unique leads to 60. - `has_email` pagination: the filter ran after LIMIT/OFFSET and was absent from the count query, so the Lead Database showed 4 of 20 matching leads with a wrong total and a permanently disabled Next button. Correctness - A rejected MAIL FROM was classified as a recipient hard bounce, so one misconfigured sending domain permanently suppressed the whole lead DB. - Soft bounces suppressed a good lead for 30 days on the first transient error; there is now a 3-attempt retry budget. - A hard-bounce suppression could silently expire, because its expiry was never cleared on upgrade. - `purge_expired` compared `expires_at < created_at` - never true. - Unknown `{{tokens}}` shipped literally to prospects; the HTML body was not escaped. - Every contact inherited `executives[0]`'s name, so `info@` was addressed as "Hi Rajesh" and the fallback produced "Hi Apex," for Apex Realty. - `is_verified=True` was asserted for addresses never SMTP-probed. - Google stores the phone with the display form first and the international form second; the parser took the first, so every number in the database was `098225 22560` - which reads as an STD code 098225 landline and mis-files the STD->city lookup. - An out-of-office auto-responder stopped the sequence and counted as an A/B "reply"; the reply poller collected the auto-reply headers and threw them away. - The A/B test compared raw rates at n=10; it is now a two-proportion z-test at n>=50 over positive replies only. - The ICP parser silently dropped constraints like "50 employees" and "revenue over 10cr"; it now reports them as unsupported, with warnings. Performance - Enrichment ran one lead at a time (~9s each) inside the event loop, and called the synchronous SMTP verifier from an `async def`, freezing the whole API. Now a bounded fan-out plus the async verifier. - No index existed on any foreign key, filter or sort column, and the schema sync never created indexes - so existing deployments silently never got them. - SQLite foreign keys were off, making every `ondelete="CASCADE"` a no-op and orphaning contacts on lead deletion. - `rank_existing` loaded the entire database to return 50 rows (13.6s and 837 MB at 100k rows). Infrastructure - The API had no auth and `CORS_ORIGINS="*"` with credentials, so any website could delete the lead database or send cold email as the operator. Added an optional API key (unset = unchanged local DX), with the mail-facing tracker exempt. - The Celery worker could not boot (it imported a function that does not exist) and no beat service ran, so reply detection never worked in the Docker deployment. - The CLI and MCP server never ran the schema migration, so they crashed with "no such column" after any model change. - MCP tools could never target anyone: no tool returned a contact ID, and `contact.id != business.id` for most rows, so an agent mailed the wrong 79% of the database. Added contact IDs, a dry-run default on launch, and JSON-RPC `isError` so the model can read a tool failure. - Click tracking did not exist, so the warm-WhatsApp handoff panel could never populate. Dashboard - The global Cmd+K search navigated to a URL nothing read; 15 native alert() dialogs; no mobile layout; Launch sent real email with no confirmation; CI skipped type-checking; 9 of 12 e2e tests could not fail the build. README: corrected the claims that were false against the code (APP_INITIALIZATION_STATE, "protobuf", humanized mouse curves, Native MCP SDK, Twenty CRM sync engine, the three spam-linter signals that do not exist) and added a Security & Operations section. Tests: 196 -> 226, including a regression suite for every blocker above. Co-Authored-By: Claude Opus 4.8 (1M context) --- .env.example | 15 + .gitignore | 3 + README.md | 92 ++- ROADMAP.md | 789 ++++++++++++++++++++ backend/app/api/campaigns.py | 41 +- backend/app/api/leads.py | 96 ++- backend/app/auth.py | 64 ++ backend/app/campaigns/bounces.py | 18 +- backend/app/campaigns/reply_poller.py | 21 +- backend/app/campaigns/sender.py | 302 +++++++- backend/app/campaigns/suppression.py | 11 +- backend/app/campaigns/tracker.py | 27 +- backend/app/cli.py | 104 ++- backend/app/config.py | 10 + backend/app/database.py | 121 +++ backend/app/discovery/pipeline.py | 45 +- backend/app/enricher/email_finder.py | 211 +++++- backend/app/enricher/pattern.py | 205 ++++- backend/app/main.py | 207 +++-- backend/app/mcp_server.py | 192 ++++- backend/app/models/__init__.py | 6 + backend/app/models/business.py | 8 +- backend/app/models/campaign_step.py | 2 +- backend/app/models/contact.py | 2 +- backend/app/models/mailbox.py | 4 + backend/app/models/send.py | 19 +- backend/app/models/suppression.py | 25 +- backend/app/qualifier/nl_icp.py | 64 +- backend/app/qualifier/ranker.py | 18 +- backend/app/qualifier/scorer.py | 19 +- backend/app/qualifier/signals.py | 49 +- backend/app/qualifier/source_trust.py | 18 + backend/app/scraper/maps_http.py | 174 ++++- backend/app/scraper/maps_scraper.py | 140 +++- backend/app/seeds.py | 4 + backend/app/workers/beat_schedule.py | 20 +- backend/app/workers/tasks.py | 162 +++- backend/tests/conftest.py | 15 + backend/tests/test_apollo_parity.py | 21 +- backend/tests/test_campaign_sequence.py | 22 +- backend/tests/test_email_finder.py | 90 +++ backend/tests/test_wave0_fixes.py | 346 +++++++++ dashboard/Dockerfile | 12 +- dashboard/e2e/smoke.spec.js | 224 ++++-- dashboard/next.config.mjs | 24 +- dashboard/package.json | 4 +- dashboard/playwright.config.js | 12 +- dashboard/src/app/layout.tsx | 5 +- dashboard/src/app/leads/page.tsx | 304 ++++++-- dashboard/src/app/page.tsx | 14 +- dashboard/src/app/scrape/page.tsx | 3 +- dashboard/src/app/settings/page.tsx | 9 +- dashboard/src/components/SequenceDetail.tsx | 84 ++- dashboard/src/components/Sidebar.tsx | 256 +++++-- dashboard/src/components/Toaster.tsx | 138 ++++ dashboard/src/components/TopBar.tsx | 20 +- dashboard/src/components/navState.ts | 43 ++ dashboard/src/lib/api.ts | 184 ++++- docker-compose.yml | 72 +- 59 files changed, 4544 insertions(+), 666 deletions(-) create mode 100644 ROADMAP.md create mode 100644 backend/app/auth.py create mode 100644 backend/tests/test_wave0_fixes.py create mode 100644 dashboard/src/components/Toaster.tsx create mode 100644 dashboard/src/components/navState.ts diff --git a/.env.example b/.env.example index ea50c42..9e7151d 100644 --- a/.env.example +++ b/.env.example @@ -37,6 +37,21 @@ CORS_ORIGINS=* # Public base URL used in open/click tracking and unsubscribe links. TRACKING_DOMAIN=http://localhost:8000 +# --- API authentication (OPTIONAL but strongly recommended) ----------------- +# Unset => the REST API is OPEN. That is fine for a local clone-and-run, and +# unsafe the moment the port is reachable from a LAN: with CORS_ORIGINS="*" +# any website could delete your lead database or send cold email from your +# mailbox. +# +# When set, every endpoint requires it: +# curl -H "X-API-Key: " http://localhost:8000/api/v1/leads +# The campaign tracker (/api/v1/campaigns/tracker/*) is exempt - it serves +# the one-click unsubscribe and the click tracker, which are fetched by the +# recipient's mail client. +# +# Generate with: openssl rand -hex 32 +API_KEY= + # --- Scraper ---------------------------------------------------------------- # Optional rotating egress proxies for directory scraping, comma-separated. # Recommended for sustained Google Maps volume (residential/Datacenter IPs diff --git a/.gitignore b/.gitignore index de019d9..432eb25 100644 --- a/.gitignore +++ b/.gitignore @@ -7,6 +7,9 @@ __pycache__/ *.sqlite *.db leadforge.db +# SQLite WAL sidecar files (created when journal_mode=WAL is enabled) +*.db-wal +*.db-shm *.log # Node & Next.js diff --git a/README.md b/README.md index d29867c..4fcdd78 100644 --- a/README.md +++ b/README.md @@ -10,7 +10,7 @@ [![Next.js 15](https://img.shields.io/badge/Frontend-Next.js%2015-black.svg?style=for-the-badge&logo=next.js&logoColor=white)](https://nextjs.org/) [![Model Context Protocol](https://img.shields.io/badge/MCP-Native%20Server-8A2BE2.svg?style=for-the-badge)](https://modelcontextprotocol.io/) [![Twenty CRM](https://img.shields.io/badge/CRM-Twenty%20Compatible-indigo.svg?style=for-the-badge)](https://twenty.com/) -[![Tests Passing](https://img.shields.io/badge/Tests-196%2F196%20Passing_(100%25)-10b981.svg?style=for-the-badge&logo=pytest&logoColor=white)](https://github.com/Aryanban/leadforge) +[![Tests Passing](https://img.shields.io/badge/Tests-223%2F223%20Passing_(100%25)-10b981.svg?style=for-the-badge&logo=pytest&logoColor=white)](https://github.com/Aryanban/leadforge) [![Docker](https://img.shields.io/badge/Deployment-Docker%20Compose-2496ed.svg?style=for-the-badge&logo=docker&logoColor=white)](https://www.docker.com/)
@@ -32,10 +32,10 @@ Modern B2B revenue and agency teams spend thousands of dollars annually on SaaS 4. **Generic AI Hallucinations**: Standard LLM outreach generators produce bland, templated pitches that trigger commercial spam filters. **LeadForge** resolves these challenges through a unified, self-hosted autonomous architecture: -* **Concurrent HTTP Maps Scraper**: Speaks Google's internal protobuf `pb=` transport directly, so listings are parsed straight out of `APP_INITIALIZATION_STATE` with no browser in the loop — and a viewport-grid sweep fans queries out concurrently to blow past the ~20-results-per-query ceiling. -* **Seed-First Email Discovery & Non-Blocking Verification**: Reverses Cloudflare `data-cfemail` tokens, harvests real addresses as *seed evidence*, then **learns each domain's actual format** instead of guessing. Verification runs in a thread pool, groups candidates per domain to reuse one MX connection, and classifies into a 20-status enum via a multi-probe catch-all sentinel — never defaulting to valid. +* **Concurrent HTTP Maps Scraper**: Speaks Google's internal `search?tbm=map&pb=...` transport directly, so listings are parsed straight out of the positional-JSON response envelope with no browser in the loop — and a viewport-grid sweep fans queries out concurrently to blow past the ~20-results-per-query ceiling. The text query is kept **category-only** ("dentists", not "dentists in Delhi") so the viewport does the geographic scoping; that single change is worth **+76% unique leads per query** at identical request count. +* **Seed-First Email Discovery & Non-Blocking Verification**: Reverses Cloudflare `data-cfemail` tokens, harvests real addresses as *seed evidence*, then **learns each domain's actual format** instead of guessing. All 10 mailbox templates are detected by reverse-engineering the local part (not shape-guessing), name-based and shape-based evidence are unioned rather than exclusive, and confidence is dominance-aware so one ambiguous seed never overrides what most domains do. Verification runs off the event loop, groups candidates per domain to reuse one MX connection, and classifies into a 20-status enum via a multi-probe catch-all sentinel — never defaulting to valid. * **Catch-All Sentinel & 80+ Spam Trigger Linter**: Identifies permissive mail servers and lints subject/body copy against 80+ commercial spam triggers before delivery. -* **Algorithmic 1-on-1 AI Icebreakers**: Analyzes business reviews, sentiment ratings, and municipal pain points to craft tailored hooks. +* **Algorithmic 1-on-1 Icebreakers**: Category-, rating- and city-aware opening hooks generated on-device from industry templates — no LLM API key required. (A local-LLM path can be plugged in; the shipped implementation is deliberately template-based so the tool runs with zero external dependencies.) * **Native Model Context Protocol (MCP)**: Empowers autonomous AI agents (Google Antigravity, Claude, Cursor) to manage prospecting, enrichment, and cold campaigns end-to-end. --- @@ -47,14 +47,14 @@ Modern B2B revenue and agency teams spend thousands of dollars annually on SaaS | **Monthly Cost** | **$0 / Free Forever** | $99 / mo | $49 / mo | ~$50 / 10k leads | $149 / mo | | **Lead Limits** | **Unlimited** | 1,000 credits/mo | 500 searches/mo | Pay-per-record | 2,500 leads | | **Self-Hosted / Local Private Data** | **Yes (100% Local)** | No (SaaS Cloud) | No (SaaS Cloud) | No (SaaS Cloud) | No (SaaS Cloud) | -| **Google Maps Scraping** | **Built-in (HTTP `pb=` + Stealth fallback)** | Add-on | Not included | API only | Not included | +| **Google Maps Scraping** | **Built-in (HTTP `pb=` primary + browser fallback)** | Add-on | Not included | API only | Not included | | **Cloudflare `data-cfemail` Decoding** | **Yes (Native XOR cipher)** | No | No | No | No | -| **Email Pattern Learning** | **Yes (per-domain inference from harvested seeds)** | Proprietary | Paid Add-on | No | No | +| **Email Pattern Learning** | **Yes (per-domain inference from harvested seeds, 10 templates)** | Proprietary | Paid Add-on | No | No | | **Catch-All & Safe SMTP Verification** | **Yes (multi-probe catch-all, 20-status enum, never defaults to valid)** | Paid | Yes | No | Add-on | | **Cold Email Spam & Deliverability Linter**| **Yes (80+ words, 0-100 score)** | No | No | No | Included | | **Algorithmic 1-to-1 AI Icebreakers** | **Yes (Sentiment & pain points)** | Paid Add-on | No | No | Included | -| **Multi-Step Sequences & Bounce Guard** | **Yes (Delay days, hard/soft bounce auto-suppress)**| Included | Add-on | No | Included | -| **AI Agent Native (MCP Server)** | **Yes (Claude, Cursor, AGY)** | No | No | No | No | +| **Multi-Step Sequences & Bounce Guard** | **Yes (autonomous step delays, hard/soft bounce auto-suppress)**| Included | Add-on | No | Included | +| **AI Agent Native (MCP Server)** | **Yes (17 tools, Claude, Cursor, AGY)** | No | No | No | No | | **Open-Source CRM Integration** | **Twenty CRM & Webhooks** | HubSpot / SF only | Zapier only | Webhooks only | Zapier only | --- @@ -111,9 +111,10 @@ flowchart TD ## ⚡ Core Engineering Innovations ### 1. Concurrent HTTP Maps Scraping & Anti-Detection Fallback -* **Concurrent HTTP Maps Scraper (Primary Path)**: Reverse-engineers Google's internal `search?tbm=map&pb=...` protobuf endpoint — the same transport the Maps frontend itself uses. Parses the `APP_INITIALIZATION_STATE` payload directly, so it extracts **name, phone, address, rating, review count and coordinates with no browser at all**. A viewport-grid sweep fans queries out concurrently across a locality, which is how production tools scale past the ~20-results-per-query ceiling. -* **Playwright Stealth Fallback**: If the HTTP path is blocked or returns an empty payload, a headless-Chromium scraper takes over. Route interception aborts stylesheets, web fonts, tracking beacons and media (`**/*.{png,jpg,jpeg,svg,woff,woff2,ttf}`) for a **3x–5x reduction in bandwidth**, plus humanized deceleration and viewport jitter to avoid bot fingerprinting. -* **Humanized Deceleration & Viewport Jitter**: Simulates randomized mouse curves and natural scrolling velocities, avoiding bot fingerprinting. +* **Concurrent HTTP Maps Scraper (Primary Path)**: Reverse-engineers Google's internal `search?tbm=map&pb=...` endpoint — the same transport the Maps frontend itself uses. Parses the positional-JSON response envelope directly, so it extracts **name, phone, address, rating, review count and coordinates with no browser at all**. A viewport-grid sweep fans queries out concurrently across a locality, which is how production tools scale past the ~20-results-per-query ceiling. + *The text query must stay **category-only***: sending "dentists in Delhi" makes Google return one city-wide relevance set, so every viewport returns a near-identical top-20 and the sweep is wasted. Measured on the same 8-cell grid: `q="dentists in Delhi"` → 34 unique leads, `q="dentists"` → 60. +* **Playwright Fallback**: The HTTP path is the default; if it is blocked or returns an empty payload, a headless-Chromium scraper takes over. Route interception aborts stylesheets, images, web fonts and media for a large bandwidth reduction, and the browser context uses a randomised user agent, a randomised viewport and humanised page-load deceleration. (The browser path is genuinely a *fallback*: it is ~3.7x slower and cannot extract phone numbers, so the HTTP path is tried first.) +* **Humanized Deceleration & Viewport Jitter**: Randomised user agent, randomised viewport and randomised page-load delay, so the browser fingerprint does not repeat identically on every run. * **Unclaimed GBP Detection**: Evaluates DOM nodes for the canonical *"Claim this business"* anchor, immediately classifying listings with highest agency acquisition value. ### 2. Seed-Driven Pattern Inference & Cloudflare XOR Deobfuscation @@ -129,14 +130,14 @@ flowchart TD * **Multi-Probe Catch-All Detection**: a single random probe cannot separate a catch-all from a block or a greylist. LeadForge sends a reachability control, multiple random probes, and a known-invalid control, then classifies `CATCH_ALL / NOT_CATCH_ALL / GREYLISTED_RETRY / BLOCKED / ANTISPAM_GATEWAY`. * **No-Probe MX Awareness**: Gmail, Zoho, Outlook and Yahoo accept-all on `RCPT TO` then reject after `DATA`. Probing them yields false “valid” results *and* gets your egress IP penalised, so they are never probed. This matters disproportionately in India, where a Gmail address is a legitimate and often the *only* business address. * **Role, Free, Disposable & Typo Detection**: `info@`/`sales@`/`support@` are flagged as group mailboxes and never counted as a “person found” (the usual cause of inflated lead counts and complaint spikes); webmail is treated as a quality signal rather than a rejection reason; typo'd domains receive a `did_you_mean` suggestion instead of being silently dropped. -* **80+ Spam Keyword Linter**: Scores copy 0–100 against commercial spam triggers, subject-line extremes, all-caps density, exclamation ratios and hyperlink density. +* **80+ Spam Keyword Linter**: Scores copy 0–100 against commercial spam triggers and subject-line extremes, and flags the Indian cold-email clichés that US-trained lists miss ("Dear Sir/Madam", "hope this email finds you well", "kindly revert back"). ### 6. India-Specific Outreach Readiness * **TRAI-Accurate Phone Normalisation**: Trunk-zero STD codes resolve against a verified SDCA table (so `0481` → Kottayam and `0487` → Thrissur, not duplicates), landline vs. mobile is distinguished by subscriber-length rules, and everything is normalised to **E.164** `+91`. Foreign numbers are no longer mangled into `+91...`. * **Channel Routing**: Landlines route to voice/email; mobiles route to WhatsApp first — which is how Indian SMBs actually answer. * **Regulatory Advisor**: The app surfaces consent, DLT and TRAI guidance as **advisory information** and leaves every decision to the operator — LeadForge is self-hosted and does not gate or veto your sending. -* **Hinglish & Festival-Aware Copy**: Generates regionally natural phrasing and suggests culturally appropriate timing windows. -* **Source-Trust Scoring**: Weights leads by the observed reliability of each source (Google Maps vs. aggregator directories), so a scraped-aggregator lead never outranks a primary-source one. +* **Hinglish & Festival-Aware Copy**: Generates regionally natural phrasing and surfaces culturally appropriate timing windows, so the operator can adapt a step to a festival week before sending. +* **Source-Trust Scoring**: Weights leads by the documented reliability of each source (live Google Maps vs. self-listed aggregator directories), so a scraped-aggregator lead never outranks a primary-source one. The trust table covers every source the scrapers actually write — including the primary scrape path — and the modifier reported by the API is the one actually applied to the score. * **Warm WhatsApp Handoff**: Emits a pre-filled `wa.me` deep link so the operator can continue in the channel the lead will actually reply on. ### 4. Algorithmic 1-on-1 AI Outreach Icebreakers @@ -150,6 +151,53 @@ flowchart TD * **❄️ COLD (<45)**: Limited contact footprint. * **Classified Bounce Suppression**: SMTP responses are classified into **hard** (dead mailbox, domain gone, no-such-user → permanent suppression), **soft** (mailbox full, over-quota, greylisted → retry), and **policy** (blocked, content-filtered → *not* treated as a dead address). Suppression is enforced again at queue time, so a bounce that lands mid-batch stops the remaining sends. + +--- + +## 🔐 Security & Operations (read this before exposing the API) + +LeadForge is a single-user, self-hosted tool, so the local clone-and-run +experience stays zero-config — but two defaults are unsafe the moment the +port is reachable from a LAN (which this README recommends). + +**Set an API key.** With `API_KEY` unset the REST API is open, and +`CORS_ORIGINS` defaults to `*`. With `allow_credentials` enabled that +combination lets *any website on the internet* call your API with the +operator's session — delete your lead database, or send cold email from +your mailbox. + +```bash +# .env +API_KEY=$(openssl rand -hex 32) +CORS_ORIGINS=http://localhost:3000 +ENV=production +SECRET_KEY=$(openssl rand -hex 32) +``` + +With `API_KEY` set, every router requires `X-API-Key: ` (or an +`Authorization: Bearer ` header) **except** the campaign tracker — +the RFC 8058 one-click unsubscribe and the click tracker are fetched by the +recipient's mail client, not by the dashboard, so they must stay reachable. + +**Background loops.** The API runs two in-process loops: a 5-minute IMAP +reply sweep and a 5-minute campaign scheduler that advances multi-step +sequences on their own. Under `USE_CELERY=true` they are handed to Celery +beat instead (`app/workers/beat_schedule.py`) so they are never +double-counted. If you deploy with Compose, **do not remove the +`celery-beat` service** — without it, reply detection and sequence +scheduling never run. + +**Schema management.** The schema is reconciled on every boot: missing +tables are created, missing columns are added, and missing indexes are +created. The migration is additive only — it never drops or retypes +anything — so it is safe to run against a database you care about. The CLI +and the MCP server perform the same reconciliation, so they never depend +on the API process having started first. + +**Agent safety.** `leadforge_launch_campaign` performs a **dry run by +default**: it returns the exact recipient list and subject lines without +sending anything. An agent must pass `dry_run=false` to actually send. + --- ## 🛠️ Technical Specifications & Engineering Decisions @@ -157,11 +205,11 @@ flowchart TD | Dimension | Engineering Decision | Rationale | | :--- | :--- | :--- | | **Backend Framework** | FastAPI (Python 3.10+) | High-performance asynchronous execution, auto-generated OpenAPI documentation, and low-latency websocket streams. | -| **Maps Extraction** | HTTP protobuf `pb=` endpoint, Playwright Stealth fallback | Primary path parses `APP_INITIALIZATION_STATE` with no browser; fallback uses headless Chromium with asset-blocking. | -| **Frontend Dashboard** | Next.js 15 (App Router) + Tailwind CSS | Modern, responsive interface with server-side rendering, real-time job status streaming, and instant search filtering. | +| **Maps Extraction** | HTTP `pb=` endpoint (primary), headless-Chromium fallback | Primary path parses Maps' positional-JSON envelope with no browser; fallback uses headless Chromium with asset-blocking. | +| **Frontend Dashboard** | Next.js 15 (App Router) + Tailwind CSS | Modern, responsive interface with client-side live job-status polling, instant search filtering, and a working mobile layout. | | **Persistence Layer** | SQLAlchemy 2.0 Async + SQLite/PostgreSQL | Zero-configuration local development via SQLite with seamless production migration to PostgreSQL. | -| **Agent Interface** | Model Context Protocol (MCP) SDK | First-class integration with modern AI coding assistants and autonomous agents. | -| **CRM Integration** | Twenty CRM & Standard REST Webhooks | Direct synchronization with modern open-source CRMs without vendor lock-in. | +| **Agent Interface** | Model Context Protocol (MCP), stdio JSON-RPC 2.0 | First-class integration with modern AI coding assistants and autonomous agents. Hand-rolled transport (no SDK dependency), 17 tools, JSON-RPC error handling that surfaces tool failures to the model, and a `dry_run` default on campaign launch. | +| **CRM Integration** | Twenty CRM export & a standard REST webhook POST | Pushes a lead export to modern open-source CRMs without vendor lock-in. (Push-only today — no inbound sync and no signed retrying event stream.) | --- @@ -275,7 +323,7 @@ Prefer the terminal? The **CLI** and **MCP agent** usage are documented below. --- -## 🧪 Automated Test Suite (196/196 Passing) +## 🧪 Automated Test Suite (223/223 Passing) LeadForge includes a comprehensive Pytest suite covering email status/role/free/typo/gibberish classification, multi-probe catch-all sentinel, SMTP verification status mapping, per-domain pattern learning, seed harvesting, lead scoring and source-trust ranking, spam checking, AI icebreakers, Indian phone/STD normalisation and DND scrubbing, deliverability (bounce classification, suppression, warm-up ramps, India send-timing advice), deduplication/entity resolution, discovery pipeline, and the sources + leads/campaign/ICP APIs: @@ -287,7 +335,7 @@ LeadForge includes a comprehensive Pytest suite covering email status/role/free/ (cd backend && ../.venv/bin/python -m pytest -q tests/) ``` -All **196/196 tests pass synchronously** in under 1s — no network, no browser, no external services required. +All **223/223 tests pass synchronously** in under 2s — no network, no browser, no external services required. --- @@ -349,12 +397,12 @@ export PYTHONPATH=backend | Competency Area | Architecture & Implementation | Engineering Impact | | :--- | :--- | :--- | -| **Asynchronous Web Automation** | Playwright Stealth with network asset route interception | 3x–5x faster crawling; resilient extraction across dense Google Maps nodes without IP bans. | +| **Asynchronous Web Automation** | HTTP-first Maps transport with a Playwright fallback that intercepts asset routes | ~3.7x faster than the browser path, with full phone/website extraction the browser path cannot do. | | **Reverse Cipher Deobfuscation** | Cloudflare `data-cfemail` single-byte XOR reverse cipher | Unmasks protected executive emails directly from obfuscated DOM trees. | | **Network & Socket Engineering** | Direct DNS MX query resolution and low-level SMTP socket handshakes | Eliminates third-party API verification costs; prevents catch-all bounce penalties. | | **Deliverability Linter** | 80+ spam trigger word scanner + deliverability health scoring | Protects outbound sender reputation by catching spam flags before transmission. | | **Agentic AI Protocol (MCP)** | JSON-RPC 2.0 Model Context Protocol server exposing tool contracts | Enables autonomous agents to execute complex prospecting sequences end-to-end. | -| **Full-Stack SaaS Architecture** | FastAPI + Next.js 15 App Router + Twenty CRM webhook engine | Sub-second lead querying, batch CSV/JSON export, and one-click CRM ingestion. | +| **Full-Stack SaaS Architecture** | FastAPI + Next.js 15 App Router + Twenty CRM webhook export | Sub-second lead querying on indexed columns, batch CSV/JSON export, and one-click CRM ingestion. | --- diff --git a/ROADMAP.md b/ROADMAP.md new file mode 100644 index 0000000..4b14c3d --- /dev/null +++ b/ROADMAP.md @@ -0,0 +1,789 @@ +# LeadForge — Apollo-Parity Roadmap (India) + +> Generated from 8 parallel codebase audits (scraping, enrichment, verification, targeting, +> outreach engine, backend/API, frontend, MCP/agent layer) + competitive research. +> Every finding below was **verified by reading code and executing it**, not inferred. +> Target: be faster, more accurate, and more complete than Apollo.io for the Indian SMB market. + +--- + +## ✅ Status: Wave 0 + Wave 1 shipped + +**226 backend tests pass** (was 196). Frontend: `tsc --noEmit` clean, `npm run build` green, +21 Playwright specs that can now actually fail the build. + +| # | Fix | Before | After | +|---|-----|--------|-------| +| B1 | Mailbox daily cap | lifetime counter → bricked after 50 sends | `sent_today` is a calendar-day counter; rolling-24h is the real gate | +| B2 | Send gate | mailed `invalid`/`gibberish`/`no_mx` addresses | refuses 10 undeliverable states; un-enriched leads stay eligible | +| B3 | Sequence scheduler | manual re-click required forever | in-process 5-min loop + Celery beat entry; "waiting on a delay" ≠ "completed" | +| B4 | Maps `q=` split | 34 unique leads / 8 requests | **60** unique leads / 8 requests (**+76%**, measured live) | +| B5 | `has_email` pagination | 4 of 20 leads visible, `total` wrong, Next disabled | SQL predicate before LIMIT; correct totals; page 2 reachable | +| B6 | Sender-vs-recipient bounce | one bad sender config nuked the whole lead DB | `sender_rejected:` / `data_rejected:` never suppress a lead | +| B7 | Soft-bounce retry | 1 blip → 30-day suppression | 3-attempt budget before suppression | +| — | Enrichment concurrency | 9s/lead, serial, inside the event loop | bounded fan-out (8×) + async verifier → event loop never stalls | +| — | SMTP probe | `EHLO [127.0.0.1]`, bogus MAIL FROM, no STARTTLS | RFC 5321 null reverse path, FQDN EHLO, STARTTLS | +| — | Pattern inference | 4/10 templates, confidently wrong | **8/10** detected, dominance-aware confidence, 0 invalid candidates | +| — | Phone extraction | display form (`098225 22560`) | international form (`+91 98225 22560`) | +| — | DB layer | FKs off, no indexes, no WAL | `foreign_keys=ON` + WAL + indexes on every FK/sort column | +| — | Schema sync | columns only; never created indexes | tables + columns + indexes; CLI and MCP self-migrate | +| — | Auth | open API, `CORS=*` + credentials | optional `API_KEY`; tracker exempt for mail clients | +| — | MCP agent loop | no contact IDs → agent mailed the wrong 79% | contact IDs + dry-run launch + JSON-RPC `isError` | +| — | Click tracking | didn't exist → WhatsApp handoff permanently empty | `/click` route + link rewriting; handoff now populates | +| — | A/B testing | raw rates at n=10 | two-proportion z-test, n≥50, positive replies only | +| — | ICP parser | silently dropped constraints | reports `unsupported_terms` + warnings; fixed corrupt exclusions | +| — | Dashboard | dead Cmd+K, 15 `alert()`, no mobile, CI ignored types | wired search, toasts, mobile drawer, type-checked CI, 21 real e2e specs | + +**One bug found during verification that the audits missed:** Google stores the phone in a +nested structure with the *display* form (leading trunk zero) first and the international +form second. The parser took the first one, so every phone in the database was +`098225 22560` — which reads as a landline with STD code `098225`, mis-files the STD→city +lookup, and does not match a `+91 98225 22560` stored elsewhere for the same business. + +--- + +## 0. Verdict + +LeadForge is a **competent single-shot mailer with a reply inbox attached**. The README +describes Apollo. The code describes something meaningfully different. + +The engineering instincts are good in places — RFC 8058 one-click unsubscribe, workspace-wide +suppression re-checked immediately before every SMTP call, a 20-state verification enum that +never coerces `unknown` → `valid`, a per-domain pattern learner, a real Cloudflare +`data-cfemail` decoder, per-test DB isolation in the test suite. Those are the pieces worth +keeping and defending. + +But five product-killing bugs mean **the core loop does not work today**, and roughly a third +of the README's headline claims are false. Fixing the five blockers plus Wave 1 is roughly +**3–5 days of work** and converts this from a demo into a product. + +**Scorecard vs Apollo (today):** + +| Capability | Apollo | LeadForge | Gap severity | +|---|---|---|---| +| Leads/day per IP | 10k+ | ~2,000 (and 5h of that is a serial enrichment loop) | 🔴 | +| Unique leads per scrape query | 100+ | 30 (viewport grid collapses — `q=` bug) | 🔴 | +| Named decision-maker found | ~40% of contacts | 8–15% | 🔴 | +| Multi-source ingestion | 275M contacts + 50+ providers | 1 working source (3 of 4 return 0) | 🔴 | +| Autonomous multi-step sequences | ✅ | ❌ no scheduler — requires manual re-click | 🔴 | +| Per-day send capacity | 50/mailbox/day, forever | 50/mailbox/**lifetime**, then bricked | 🔴 | +| Hard-bounce rate | <5% | ~25–30% (sends to known-invalid addresses) | 🔴 | +| ICP filter expressiveness | 20+ attributes | 3 loose filters | 🟠 | +| ICP fit scoring | per-saved-search, live | cached global contact-completeness score | 🟠 | +| Opportunities / pipeline board | ✅ | ❌ no model exists | 🟠 | +| Analytics | funnel + cohorts + rep leaderboard | 4 flat counters | 🟠 | +| People table (row = person) | ✅ core surface | ❌ only a company table | 🟠 | +| Click tracking | ✅ | ❌ no route exists | 🟠 | +| India: GSTIN / Udyam / DLT / pincode / city tier | ❌ | ❌ | 🟢 **our moat** | +| India: WhatsApp-first routing | ❌ | computed, then ignored | 🟢 **our moat** | +| India: STD→city resolution | ❌ | ✅ built (`in_phone.py`) | 🟢 | +| India: mobile vs landline | ❌ | ✅ built, never scored | 🟢 | + +--- + +## 1. Wave 0 — The five blockers (product does not work without these) + +### B1. Every mailbox bricks after 50 lifetime sends 🔴 P0 + +`backend/app/campaigns/sender.py:251` +```python +m_stmt = select(Mailbox).where(Mailbox.is_active == True, Mailbox.sent_today < Mailbox.daily_limit) +``` +`sent_today` is incremented at `sender.py:477` and **reset nowhere** — verified: the only +other writes are `api/mailboxes.py:69` and `seeds.py:198`, both at creation time. The +`warmup.py` docstring at `:18-21` calls it "a fast pre-filter" and names +`rolling_24h_count` as "the authoritative gate" — but the stale pre-filter fires first and +the mailbox never reaches the real gate. **After 50 emails the mailbox is dead forever.** + +```python +# sender.py:251 — drop the broken counter from the filter +m_stmt = select(Mailbox).where(Mailbox.is_active == True) +# sender.py:477 — stop persisting a drifting counter +# (expose rolling_24h_count in GET /mailboxes for the UI instead) +``` +**Gain:** unbounded daily throughput; `warmup.py`'s entire ramp/rolling-24h machinery +becomes reachable for the first time. + +--- + +### B2. The sender emails known-invalid addresses 🔴 P0 + +`backend/app/campaigns/sender.py:294-302` filters on exactly one exclusion: +```python +Contact.verification_status != "bounced" +``` +`should_send()` exists at `backend/app/verifier/email_status.py:379-389` and is **never +called from the sender** — only from the MCP display path. So `invalid`, `syntax_error`, +`gibberish`, `disposable`, `no_mx`, `unknown`, `catch_all` are all mailed. The verifier's +"never defaults to valid" guarantee is enforced in one file and discarded two files away. + +For an Indian SMB list where ~30% of scraped addresses are junk, that is a ~30% hard-bounce +rate on day one. Gmail/Yahoo's bulk-sender threshold is **5%**. + +```python +# sender.py:294 +Contact.verification_status.notin_( + ("bounced", "invalid", "syntax_error", "gibberish", "disposable", + "no_mx", "domain_error", "spamtrap", "abuse", "do_not_mail")) +``` +**Gain:** hard bounce rate ~25–30% → <5%. Sender reputation preserved. + +--- + +### B3. No scheduler — multi-step sequences never advance on their own 🔴 P0 + +`execute_campaign_step` has exactly two call sites: `api/campaigns.py:273` (manual Launch) +and `mcp_server.py:686`. `main.py:84-103` has a `_reply_poll_loop` but no campaign loop; +`workers/beat_schedule.py:8-13` contains only the IMAP sweep. The delay arithmetic at +`sender.py:374-376` is correct — **nothing ever calls it again at T+3 days.** An operator +must manually re-click Launch for every step of every sequence, forever. + +Compounding: `sender.py:381-385` marks the campaign `COMPLETED` whenever the queue is empty +— which is true both when everyone finished **and** when everyone is waiting on a delay. +The two are indistinguishable, so campaigns falsely report "Completed". + +```python +# app/main.py — mirror the existing reply loop +CAMPAIGN_TICK_SECONDS = 300 + +async def _campaign_run_loop() -> None: + while True: + try: + async with async_session_maker() as db: + ids = (await db.execute( + select(Campaign.id).where(Campaign.status == CampaignStatus.ACTIVE.value) + )).scalars().all() + for cid in ids: + await execute_campaign_step(cid) + except Exception as exc: + logger.warning("campaign tick error: %s", exc) + await asyncio.sleep(CAMPAIGN_TICK_SECONDS) +``` +Plus a Celery beat entry, and a per-campaign lock so the tick and a manual Launch can't +double-send. And distinguish "all steps done" from "waiting on a delay" at `sender.py:381`. + +**Gain:** the README's headline feature ("multi-step sequences with delay days") becomes +true. This is the difference between a mailer and an outreach engine. + +--- + +### B4. The `q=` parameter collapses the viewport grid by 3.4× 🔴 P0 + +`backend/app/scraper/maps_http.py:250` sends `"q": query` where `query` is +`build_query(niche, area)` → `"dentists in Delhi"`. Google then returns one city-wide +relevance set, so **every viewport in the grid returns a near-identical top-20**. + +Measured live (8-cell grid, Delhi, identical requests, only `q` varies): + +| `q` | raw rows | unique leads | +|---|---|---| +| `"dentists in Delhi"` (current) | 160 | **30** | +| `"dentists"` (category only) | 160 | **103** | + +Distance distribution confirms it: with `q="dentists"` the median result sits **1.3 km** from +its own viewport centre (p90 = 5.0 km); with `q="dentists in Delhi"` it sits 5.4 km away and +the set is identical across cells. The viewport grid is only geographic scoping when the +text query is category-only. + +```python +# sources/google_maps.py — split "what we're looking for" from "where" +async def search(self, niche, area, limit=20): + raw = await scrape_google_maps_http( + query=self.build_query(niche, area), # label only + q=industry_of_query(niche), # the actual request + max_results=limit, max_cells=8) +``` +**Gain: +243% unique leads per query at identical request count (30 → 103).** Highest +gain-per-diff in the entire codebase — roughly a 3-line change. + +--- + +### B5. `has_email` is applied *after* pagination and excluded from `total` 🔴 P0 + +`backend/app/api/leads.py:113-115`: +```python +if has_email is True and not any(c.get("email") for c in b_contacts): + continue +``` +This runs **after** `.offset(offset).limit(limit)` at `:82`, and `count_stmt` (`:64-77`) +has no `has_email` predicate at all. Verified against the repo's own DB (75 businesses, +20 with email): + +| Request | `total` reported | rows returned | +|---|---|---| +| `?page=1&limit=20` | 75 | 20 | +| `?page=1&limit=20&has_email=true` | **75** (should be 20) | **4** | +| `?page=2&limit=20&has_email=true` | 75 | 5 | + +The frontend compounds it — `dashboard/src/app/leads/page.tsx:874` disables Next when +`leads.length < 20`, so with a filtered subset the user is **permanently pinned to page 1** +and can never reach the other 16 leads. This is the Lead Database, the product's front door. + +```python +# api/leads.py — one predicate, used by BOTH the count and the page +email_ids = select(Contact.business_id).where(Contact.email.isnot(None)) +if has_email is True: + stmt = stmt.where(Business.id.in_(email_ids)) + count_stmt = count_stmt.where(Business.id.in_(email_ids)) +elif has_email is False: + stmt = stmt.where(Business.id.notin_(email_ids)) + count_stmt = count_stmt.where(Business.id.notin_(email_ids)) +``` +Delete the Python-side `continue` block, and change the frontend to paginate on `total`. + +**Gain:** the filter becomes truthful; pagination stops silently truncating. **16 of 20 +matching leads become reachable.** The same bug exists in `cli.py:127` and +`mcp_server.py:363` — fix all three, or extract one shared `build_lead_query()`. + +--- + +## 2. Wave 1 — Speed & accuracy (the "faster, better" mandate) + +### S1. Enrichment is 100% serialized: ~9s per lead, blocking the event loop + +`backend/app/discovery/pipeline.py:20-31` awaits `enrich_business_by_id` one lead at a time. +Measured **8.7s and 9.5s** per lead (Yahoo + Bing + 4-page crawl + social probe + SMTP). +A 100-lead job = **~15 minutes** inside the FastAPI event loop. `maps_scraper.py:528-534` +does the opposite — unbounded `asyncio.create_task` with no reference kept (CPython may GC +the task mid-flight) and no semaphore. + +```python +# discovery/pipeline.py +async def _enrich_new_leads(business_ids: List[int], concurrency: int = 8) -> int: + sem = asyncio.Semaphore(concurrency) + async def one(bid): + async with sem: + try: + await enrich_business_by_id(bid); return 1 + except Exception as e: + logger.warning("Enrichment failed for %s: %s", bid, e); return 0 + return sum(await asyncio.gather(*(one(b) for b in business_ids))) +``` +**Gain: ~8× throughput (9s → ~1.1s amortised). 100-lead job 15 min → ~2 min.** + +### S2. `email_finder.py` calls the *sync* SMTP verifier inside `async def` + +`backend/app/enricher/email_finder.py:453` → `check_catch_all_domain(w_domain)` (sync) +and `:501` → `verify_email(email)` (sync, up to 3 RCPT probes, 8s timeout each). + +`smtp_verifier.py:336-345` says verbatim: *"calling them directly stalls the whole event +loop for the 3-8s of a DNS+SMTP round trip"*. The module ships +`verify_email_async` / `check_catch_all_domain_async` — **the enricher doesn't use them.** + +**Gain:** API p99 during bulk enrichment goes from seconds to milliseconds; the whole API +stops freezing. + +### S3. Three SMTP sessions per domain per lead, and every email verified twice + +Per lead per domain the code opens **2–5 distinct MX connections** and performs **3–8 DNS +lookups**: `check_catch_all_domain` (conn #1) → `verify_batch` (conn #2) → a per-address +`verify_email` loop (conn #3, #4, #5…). `verify_batch` already computes the per-domain +catch-all verdict internally (`smtp_verifier.py:464`) but doesn't surface it, which is why +a *second* redundant probe was added. + +**Gain:** MX connections per lead **3–5 → 1–2**; wall-clock **−20–40%**; port-25 blocklist +exposure cut by an order of magnitude. + +### S4. The SMTP probe announces `EHLO [127.0.0.1]` and uses a bogus MAIL FROM + +`backend/app/verifier/smtp_verifier.py:138-148`: +```python +self.server = smtplib.SMTP(self.mx_host, port=25, timeout=SMTP_TIMEOUT) +self.server.ehlo_or_helo_if_needed() +self.server.mail(self.sender_email) # return code DISCARDED +``` +Four defects: (a) `smtplib.mail()` returns `(code, msg)` and does **not** raise — a +`550 5.7.1 Sender rejected` is silently dropped and the next RCPT gets `503`, which is in +no bucket → `UNKNOWN`; (b) the default envelope sender is `verify@leadforge.dev` — no MX, +no SPF, no DKIM, rejected by Gmail since Jan 2024 and by Postfix with +`reject_unknown_sender_hostname` (very common on Indian cPanel/Hostinger); (c) **no +STARTTLS** is attempted, so any MTA with `smtpd_tls_security_level=encrypt` answers +`530` → `UNKNOWN`; (d) `smtplib` defaults `local_hostname` to `socket.getfqdn()`, which in +Docker is the container ID with no dot → falls back to the domain literal +**`[127.0.0.1]`** — the canonical signature of an open-relay harvester. + +```python +self.server = smtplib.SMTP(self.mx_host, port=25, timeout=SMTP_TIMEOUT, + local_hostname=settings.SMTP_HELO_NAME) +self.server.ehlo_or_helo_if_needed() +if self.server.has_extn("starttls"): + self.server.starttls(context=ssl.create_default_context()) + self.server.ehlo_or_helo_if_needed() +code, msg = self.server.mail("<>") # RFC 5321 null reverse path +if code != 250: + self.error = f"mail_from_rejected:{code}"; return False +``` +**Gain:** verification stops returning `UNKNOWN` on hardened MTAs; the single most +blocklist-worthy signal in the codebase disappears. + +### S5. Catch-all detection: one random probe, and `unknown` silently means "not catch-all" + +`backend/app/verifier/smtp_verifier.py:262-315` sends **one** random probe and one +pseudo-control, then classifies `CATCH_ALL / NOT_CATCH_ALL / GREYLISTED_RETRY / BLOCKED` — +the README's "reachability control + multiple random probes + known-invalid control" and +the `ANTISPAM_GATEWAY` verdict **do not exist**. Worse, `"unknown"` is treated as *strict*: +an anti-probing MTA that 250s the first RCPT and 554s the second yields `"unknown"` → the +real address is reported **VALID at confidence 0.95**. + +**Gain:** eliminates the false-`valid` class — the #1 cause of sender-reputation burn on +catch-all-heavy Indian domains. + +### S6. `/api/v1/scraper/start` is Playwright-first and produces garbage leads + +`backend/app/api/scraper.py:35` → `maps_scraper.py:430` runs Playwright **first**, never the +HTTP path. Measured live (Delhi, 20 results): + +| field | HTTP path | Playwright path | +|---|---|---| +| wall clock | **2.4s** | **8.9s** | +| phone present | 20/20 | **0/20** | +| website present | 17/20 | **0/20** | +| valid `place_id` (`ChIJ…`) | 20/20 | **0/20** | +| address contains PUA glyph | 0/20 | **17/20** | +| `operational_status` | correct | `"Closed"` (misread) | + +The Playwright extractor keeps the category **icon glyph** as the first address token +(`maps_scraper.py:252-259`), and sets `place_id` from `href.split("?")[0][-60:]` — a +truncated URL fragment stored into a `unique=True` column that is the **primary dedupe key**. +Playwright-sourced and HTTP-sourced rows for the same business have incompatible keys. + +**Gain: ~4× faster and ~85% more usable fields per lead**, on the endpoint the dashboard's +scrape button actually calls. One-line reorder: try HTTP first. + +### S7. Three of four registered sources return zero leads + +Measured live: `justdial → 0`, `indiamart → 0` (the URL is a **404**; the endpoint moved +to `dir.indiamart.com/search.mp`), `sulekha → 0` (selectors are legacy Tailwind classes; +business names live in `

`), `google_maps → 20`. The "multi-source router" is +Google-Maps-only in practice. `tests/test_sources.py:61` asserts only that the names are +*registered*, never that they return anything — which is why all 196 tests pass while 3 +sources are dead. + +Also: **JustDial and Sulekha carry websites** (→ email enrichment) and verified/trust badges +that Maps doesn't have. Fixing them adds both volume *and* enrichability. + +### S8. Dedupe silently discards real leads and false-merges across cities + +`backend/app/dedupe/match.py:99` matches `Business.name == lead["name"]` with **no city +guard** — and Indian SMBs are dominated by generic names ("Sharma Dental Clinic", "Krishna +Medicals"). `dedupe/persist.py:39-45` treats a match as "already known" and `continue`s, so +the **Jaipur row is never inserted** and its phone/website/address are **thrown away**. + +Also verified: phone dedupe is **dead** (`match.py:60-67` compares normalised phone against +the raw stored value); domain matching is a **substring** so `example.com` matches +`smileshop.example.com`; and `normalize_name` is **computed then discarded** at +`match.py:97-102` (the query uses the raw name). + +**Gain:** at 50k rows the name-collision rate in Indian SMB data is 5–15%. Fixing this is +worth more than adding any new source. + +### S9. `rank_existing` loads the entire database to return 50 rows + +`backend/app/qualifier/ranker.py:108-130` — `select(Business)` with **no limit**, then +pure-Python scoring with **no `await` inside** (so it blocks the event loop for the full +duration). Benchmarked: **13.6s and 837 MB** at 100k rows, returned 50. Extrapolated to +500k: **~70s and ~4 GB per request.** This is the "rank my saved search" button. + +Same anti-pattern in the CSV export and the Twenty-CRM export (both unbounded full-table +joins, no LIMIT, no streaming). + +### S10. No indexes on any foreign key, filter, or sort column + +Live schema dump: +``` +contacts: ['ix_contacts_email', 'ix_contacts_id'] <-- NO business_id +sends: ['ix_sends_id', 'ix_sends_message_id'] <-- NO campaign_id/contact_id/mailbox_id/step_id +businesses: ix_businesses_{icp_profile_id,id,name,place_id} <-- NO created_at +``` +Measured at 500k businesses / 1M contacts (SQLite): +``` +BEFORE: contacts.business_id IN (25 ids) 42 ms + businesses ORDER BY created_at DESC LIMIT 25 32 ms +AFTER adding 2 indexes: + contacts.business_id IN (25 ids) <1 ms + businesses ORDER BY created_at DESC LIMIT 25 <1 ms +``` +**And the schema-sync mechanism won't fix it** — `main.py:25-58` only adds *columns*; it +never creates indexes or constraints, so existing deployments silently never get them. + +### S11. CSV import is O(rows) round-trips inside one transaction + +`backend/app/api/leads.py:453-546` — ~5 queries per row (4 for dedupe + 1 contact lookup), +one commit at the end. Measured: **5000 rows = 15.24s**, and it holds SQLite's global write +lock the whole time, so the entire app is unresponsive. + +### S12. SQLite foreign keys are OFF — every `ondelete="CASCADE"` is a no-op + +`backend/app/database.py:9-16` never issues `PRAGMA foreign_keys=ON`. SQLAlchemy doesn't do +it for you. `api/leads.py:385` uses a Core `delete(Business)` which bypasses the ORM's +`cascade="all, delete-orphan"` entirely → **every deleted lead orphans its contacts.** +Verified: `PRAGMA foreign_keys: 0`, orphan count 1 after one delete. + +```python +@event.listens_for(engine.sync_engine, "connect") +def _sqlite_pragmas(dbapi_conn, _rec): + cur = dbapi_conn.cursor() + cur.execute("PRAGMA foreign_keys=ON") + cur.execute("PRAGMA journal_mode=WAL") # removes reader/writer blocking + cur.execute("PRAGMA synchronous=NORMAL") + cur.execute("PRAGMA busy_timeout=5000") + cur.close() +``` + +### S13. Pattern inference detects 4 of 10 patterns and mis-learns 4 more at 0.95 confidence + +`backend/app/enricher/pattern.py:131` requires `len(p) >= 2` for every token, making the +`f.last` and `firstl` branches **unreachable**. A single alpha token is unconditionally +classified `"first"`. Verified: + +``` +rsharma@acme.in → got "first" (expected "flast") conf 0.95 ← WRONG, confident +rajeshsharma@acme.in → got "first" (expected "firstlast") conf 0.95 ← WRONG, confident +sharma@acme.in → got "first" (expected "last") conf 0.95 ← WRONG, confident +a.sharma@acme.in → got "first.last" (expected "f.last") conf 0.25 +``` +And `pattern.py:121-129` **discards all seed evidence** whenever the exec names don't match +the seed — because the name-based and shape-based branches are mutually exclusive instead of +unioned. Finally, the computed confidence (`:148-150`) is **not dominance-aware** (it +contradicts the repo's own design doc at `.research/lead-data.md:83`), and `email_finder.py:447` +**throws it away** (`permutations, _pat, _conf = ...`). + +`flast` / `firstlast` / `last` are disproportionately German/Indian-IT. + +### S14. Every contact inherits `executives[0]`'s name and title + +`backend/app/enricher/email_finder.py:510-514` runs **inside** the per-email loop: +```python +exec_match = execs[0] if execs else None +c_first = exec_match["name"].split()[0] if exec_match else (business.name.split()[0] if business.name else "Owner") +``` +So `info@acme.in` is stored with `first_name="Rajesh", title="Founder"` — simultaneously +`is_role_account=True` **and** named. With no exec, the fallback produces *"Hi Apex,"* for +"Apex Realty Solutions". `sender.py:413` then renders **"Hi Rajesh," to a shared mailbox.** + +### S15. Decision-maker extraction misses every real Indian SMB title + +`backend/app/enricher/email_finder.py:237-244` `EXEC_TITLES` misses **MD, Principal +(Principal Dentist — the dental norm), General Manager, Promoter, Chairman**. The regex +also manufactures garbage: `"read more - owner"` → `('read more', 'owner')`, +`"call now - partner"` → `('call now', 'partner')`. Those become `person_names` and +**consume the 4-slot candidate budget** (`read.more@domain`, `call.now@domain`). + +### S16. `is_verified=True` is asserted for addresses that were never SMTP-probed + +`email_finder.py:502` sets `is_verified = result.is_deliverable`, and `is_deliverable` +includes `ROLE_ACCOUNT` and `FREE_EMAIL` — both of which `classify_email` short-circuits +**before any network call**. So `info@acme.in` gets `is_verified=True`, a `verified_at` +timestamp, and the scorer's full **30 points** plus the "Verified Email" badge. The shipped +test at `tests/test_email_finder.py:78` actively asserts this behaviour. + +### S17. The spam linter misses every Indian cold-email cliché + +`backend/app/campaigns/spam_checker.py:11-46` — the 120-word high-risk list is US +consumer-spam vocabulary (`viagra`, `weight loss`, `mortgage rates`). Verified: *"Hope this +email finds you well / I am writing to introduce / Dear Sir/Madam / kindly revert back"* +scores **75 = "Good", `is_safe: True`**. Meanwhile ordinary words false-positive +(`score`, `promise`, `certified`, `win`). The README's "all-caps density, exclamation +ratios and hyperlink density" — **none of the three exist.** + +### S18. The agent cannot obtain a `contact_id`, so it can never target anyone + +`mcp_server.py:198-200` requires `contact_ids`, but **no tool output contains one** — +`search_leads` and `scrape` return business `id` only. In the live DB, `contact.id != +business_id` for **100 of 126 contacts**. An agent doing the only thing it can mails the +**wrong 79% of the database**. The entire "scrape → target → launch" loop is a dead end. + +### S19. `leadforge_scrape` returns the wrong leads + +`mcp_server.py:300-303` returns the **last N businesses in the whole database**, not the +ones this scrape created — while the header says "Scrape Results for: {query}". The +`_new_biz_ids` returned by `scrape_leads_and_save` is discarded at `:297`. + +### S20. No auth + wide-open CORS = any website can delete data and send cold email + +`main.py:113-123` with `config.py:34` `CORS_ORIGINS = "*"` **and** `allow_credentials=True` +— Starlette echoes the attacker's origin on preflight. Verified: `DELETE /api/v1/leads/999` +→ 200, `POST /campaigns/{id}/launch` → cold email sent from the user's Gmail. + +**Gain:** the only finding where an attacker needs zero credentials. Cheap fix that doesn't +break local DX: an optional `API_KEY` (unset = auth off) applied to every router **except** +`tracker` (the unsubscribe link and the open pixel must stay reachable by mail clients). + +### S21. SSRF via the Twenty-CRM export webhook + +`api/leads.py:340-344` POSTs to a user-supplied `webhook_url` with no scheme check, no +host allowlist, no private-IP block — and the payload contains **the entire lead database**. +Against a cloud host this reads instance metadata / IAM credentials. + +### S22. Soft bounce = immediate 30-day suppression, and there is no retry path + +`sender.py:486-491` suppresses on the **first** soft bounce; `bounces.py:130-132` +`should_retry()` is **defined and never called**. `bounces.py:101-102` classifies *any* +unrecognised error (connection refused, `SMTPServerDisconnected`) as soft. One transient +gateway blip removes a valid lead for a month. + +Worse: `bounces.py:105-127` `_HARD_RE` matches bare `\b550\b` / `\b553\b`, and +`smtplib` raises `SMTPSenderRefused` when **our own MAIL FROM** is rejected — a *sender* +problem classified as a *recipient* hard bounce → **one misconfigured sending domain +permanently suppresses the entire lead database.** + +### S23. `purge_expired` never fires + +`campaigns/suppression.py:134-138` compares `Suppression.expires_at < Suppression.created_at` +— always false, since `expires_at = created_at + 30d`. It should be `< func.now()`. It's +also never called from anywhere. + +### S24. `{{tokens}}` ship literally to prospects + +`sender.py:30-38` `interpolate_template` does a plain `str.replace` with **no fallback** — +an unknown token leaves the braces in. A typo'd `{{frist_name}}` reaches the prospect. +No HTML escaping either (`:440` injects `business.name` raw into `body_html`). And +`api/campaigns.py:288-308`'s preview uses a sample dict **missing** `{{ai_icebreaker}}`, +`{{compliment}}`, `{{pain_point}}`, `{{call_to_action}}` — so the preview renders +`{{pain_point}}` literally while the real send renders a sentence. + +### S25. Click tracking doesn't exist → the WhatsApp handoff can never fire + +`models/send.py:19` has a `clicked` column that **nothing ever writes**; `tracker.py` has no +`/click` route. The handoff rule at `engagement.py:23-27` requires ≥3 opens or ≥1 click, +and open tracking defaults **off** (`config.py:62`). `GET /campaigns/india/handoff/{id}` +returns **an empty list, permanently.** The README's flagship India feature is a no-op. + +### S26. The `AI icebreaker` is a template engine with a hardcoded Delhi default + +`enricher/ai_icebreaker.py:11-48` is a 9-key static dict + 3 string-format branches. No +LLM, no review text is ever collected (only `reviews_count: int`), no sentiment analysis, +and the `website` parameter is **never read**. `mcp_server.py:502` hardcodes +`city = arguments.get("city", "Delhi NCR")` — so a Kochi lead generated via MCP gets +*"…other dental leaders in Delhi NCR"*. + +--- + +## 3. Wave 2 — Product completeness (the Apollo surface we're missing) + +### 2a. The People table — a company table where the row is a business, not a person + +Apollo's core surface is *People*: one row per human, filterable by title and seniority. +`/leads` returns businesses with nested contacts, and **there is no contacts-list endpoint** +(verified: zero `@router.get` routes matching "contact"). `Contact.title` is stored and +shown in the drawer but is **not a column and not filterable**. + +> This is the single biggest structural gap. It needs: a `/api/v1/contacts` endpoint, a +> people table in the dashboard, and real decision-maker extraction (S14, S15) to be worth +> anything. + +### 2b. ICP filters: the parser silently drops constraints + +`qualifier/nl_icp.py:63-68` — the char class `[a-z0-9\- ]` includes a space, so it greedily +eats following words. Verified: + +``` +'dentists in Kothrud Pune with no website and verified emails' + → exclude_keywords: ["website and verified emails"] ← corrupt +'shops in Mumbai with 50 employees and revenue over 10cr' + → employee count + revenue: SILENTLY DROPPED +'exporters in Noida with Shopify and looking for a sales director' + → tech + title: SILENTLY DROPPED +``` +A user who types "50 employees, revenue over 10cr, Shopify" gets **every shop in Mumbai** +with no indication their constraints were ignored. This is the "the product lied to me" bug. + +**Fix (minimum):** add a `UNSUPPORTED` regex and return `unsupported_terms` + `warnings`, +surfaced in the wizard. **Fix (full):** a structured `ICPFilters` model — `industries`, +`cities`, `areas`, `pincodes`, `company_size_min/max`, `revenue_band`, `tech`, `titles`, +`seniority`, `has_gstin`, `city_tier`. + +### 2c. Source trust is a hand-written table that isn't applied to most leads + +`qualifier/source_trust.py:25-36` is a literal dict of 10 constants. The primary scrape +path writes `source` values `"playwright_maps"`, `"playwright_single"`, `"http_fallback"`, +`"google_maps_seed"` — **none of which are in the table**, so the modifier is skipped +entirely for every lead the dashboard produces. The API then *reports* a modifier that was +never applied (verified mismatch). And `int(round(delta))` collapses the whole trust range +into ±4 points, so IndiaMART (0.60) gets **no discount** while JustDial (0.50) gets −2. + +Separately, `discovery/pipeline.py:152-157` records **the same `avg_score` for every +source** in every run (it ranks the union across all sources, then attributes that list to +each source). So the "learned" source routing prefers whichever source returns the most +junk. + +### 2d. The score is cached at enrichment time and never recomputed + +`api/leads.py:134-142` reads `extra["lead_score"]` — computed once at enrichment. Verified: +a fresh ICP-scoped score of 25 COLD is displayed as **95 HOT** from the stale cache. Changing +an ICP has **zero** effect on existing leads — which makes the entire "Save Search" story a +query builder on top of a ranking that ignores the query. + +**The structural fix:** `lead_score` should be **ICP-relative and live**, not a global +cached contact-completeness number. Fixing this one thing collapses a large share of the +targeting findings. + +### 2e. Analytics is four flat counters + +No bounce *rate*, no unsubscribe rate, no positive-reply rate, no per-source, no +per-mailbox, no hour-of-day heatmap, no funnel, no cohort, no reply→meeting. Meanwhile +`Send.bounced` and `Send.clicked` are **never written** — so `engagement.py:55` always +reports `bounced: 0` even when the mailbox is bouncing. + +### 2f. A/B testing has no statistics and the wrong metric + +`api/campaigns.py:414-431` compares raw reply rates at n=10 with no significance test. +Cold-email reply rates are 1–3%, so one extra reply = 10pp and a coin flip declares a +winner. The metric counts OOO auto-replies and opt-outs as "replies" — and the reply +poller **drops the auto-reply headers** (`reply_poller.py:257` computes `auto` then +discards it at `:266`), so `Auto-Submitted` is dead code at runtime. An out-of-office +auto-responder can pick your subject line. + +Also: the winner is only finalised as a side effect of a `GET`, and `sender.py` reads +`step.variant_winner` but never writes it. + +### 2g. Missing Apollo surfaces with no model behind them + +Opportunities/deals pipeline, tasks, meetings/calendar, notes on a lead, unified +conversation inbox with threading/ownership, per-rep attribution, inbound webhooks, signed +outbound webhooks, per-record activity timeline, field-level provenance on every attribute, +and a searchable contacts list. + +### 2h. Dashboard gaps + +- **Global Cmd+K search is dead.** `TopBar.tsx:32` pushes `/leads?search=…` but **nothing + ever reads a query param** (`useSearchParams` → zero hits). Type "dentist", press Enter, + get all 75 leads. +- **`/docs` 404s.** `Sidebar.tsx:174` links to a route that doesn't exist. +- **15 native `alert()` dialogs**, no toast system — even though `@radix-ui/react-toast` is + already a dependency and never imported. **10 of 15 runtime deps are dead weight** + (react-query, zod, react-hook-form, radix-dialog/label/slot/icons). +- **Launch has no confirmation** (`SequenceDetail.tsx:109`) — the highest-consequence action + (sends real email) has the least protection, while lead/mailbox delete do get `confirm()`. +- **Mobile is structurally broken.** `Sidebar.tsx:94-97` is `w-60 shrink-0` with **zero + responsive prefixes**; on a 390px viewport the sidebar takes 240px of 390px. The e2e test + that "verifies" this asserts `document.scrollWidth - clientWidth <= 4` — but `body` is + `overflow-hidden`, so it **passes while the layout is unusable.** +- **No design system.** `tailwind.config.ts` defines semantic tokens + dark mode; **zero** + components use them and there are **zero `dark:` classes** in the app. `cn()` is never + imported. 175 occurrences of `text-[8px]`–`text-[11px]` (9px fails legibility floors). + **8 uses of `backdrop-blur-xs`** — an invalid Tailwind class that compiles to nothing. +- **`lib/api.ts` has no timeout, no retry, no `AbortController`** → stale-response races + (two in-flight `getLeads` can resolve out of order and overwrite each other). `enrichLead` + can hang for 30–120s with no abort. +- **CI ignores type errors.** `next.config.mjs` sets `ignoreBuildErrors` + `ignoreDuringBuilds` + → `tsc --noEmit` finds **3 live errors** CI never sees. Playwright is not in CI at all. +- **9 of 12 e2e tests can never fail the build** — they `findings.push(...)` instead of + asserting. The committed `dashboard/shots/findings.json` proves it was already failing. +- **The home page fabricates a KPI delta**: `page.tsx:180` renders "+30 discovered" when the + total is 30 — directly under a comment saying *"never fake demo numbers."* +- **No rupee pricing anywhere** — the entire product thesis ("Apollo but free for Indian + SMBs") is absent from the UI. +- **No GSTIN field, no Hindi/Hinglish UI** (no `t()`, no locale switcher, no RTL readiness). + +--- + +## 4. Wave 3 — The India moat (where we beat Apollo, not just match it) + +Apollo has **no Indian registry data, no WhatsApp routing, no STD-code resolution, no DLT +context**. These are the features that make an Apollo clone irrelevant. + +1. **GSTIN / Udyam (MSME) / CIN registry lookup** — a GST-scoped list is a genuine moat and + one public registry away. Currently no such column exists anywhere. +2. **STD-code → city resolution** — `sources/in_phone.py:37-131` already has a verified SDCA + table. `city_from_phone()` is **never called outside its own module.** This is a + done-and-buried asset: pincode clusters, city tiers, and per-city analytics for free. +3. **Mobile vs landline as a score signal** — `in_phone.is_mobile` is independent + information (currently wasted: the `whatsapp` signal is **100% collinear with `phone`**, + so 15 of 100 score points are free). `Contact.preferred_channel` is computed and then + **never read by the sender**. +4. **WhatsApp-first channel routing** — a lead with a mobile should route to WhatsApp + before email. Apollo has no equivalent. +5. **Real festival + IST timing windows** — currently decorative: `india_copy.py`'s + `localize_template()` is **never called by the sender**, so `{{festival_line}}` ships + **literally** to prospects. There is no send-window control at all — a launch at 3 a.m. + IST sends at 3 a.m. IST. +6. **Real multi-language copy** — Hinglish today is 6 static closer strings to copy-paste. + Marathi/Telugu/Bengali/Kannada/Tamil are absent. +7. **India's directory ecosystem** — JustDial, Sulekha, IndiaMART, InPhone, Practo, and + (for the wedge) IndiaMART's *product* directory for manufacturing/wholesale ICPs, which + is high-value and currently noise for service ICPs. + +--- + +## 5. README truth-check (credibility) + +Roughly a third of the README's headline claims are **false against the code**. A reviewer +who runs the documented commands and finds them broken costs more credibility than any +feature gap. + +| README claim | Reality | +|---|---| +| "parses `APP_INITIALIZATION_STATE`" | **Zero occurrences in the repo.** The code does `json.loads(...)` on positional JSON. | +| "internal protobuf `pb=` transport" | It's not protobuf — it's positional JSON. No protobuf dep. | +| "Humanized deceleration & viewport jitter" | `random_delay` and `get_random_viewport` are **imported and never called**. No mouse curves. | +| "Unclaimed GBP Detection … canonical *Claim this business* anchor" | The HTTP path never sets `is_claimed`; the Playwright path does a **text match**. The scoring weight never fires. | +| "Playwright Stealth Fallback" | It's **primary** on `/scraper/start`, not fallback. Stylesheets are *not* blocked (only image/media/font). | +| "multi-source" | 3 of 4 sources return **0 leads** against live sites. | +| "learns each domain's actual format" | Learns 4 of 10 patterns; mis-learns 4 more at 0.95 confidence; **discards seeds** when exec names don't match. | +| "Confidence-Blended Inference" | Not dominance-aware; and the confidence is **discarded** by the caller. | +| "Algorithmic 1-on-1 AI Icebreakers (sentiment & pain points)" | Static 9-key dict + 3 format branches. No LLM, no review text, no sentiment. | +| "Source-Trust Scoring … *observed* reliability" | Hand-written constants. Never observed. Not applied to the main scrape path. | +| "Multi-Step Sequences & Bounce Guard" | Delay math exists; **no scheduler**. Requires manual re-click. | +| "Hinglish & Festival-Aware Copy" | `localize_template()` is **never called by the sender**. | +| "warm WhatsApp handoff" | Requires ≥1 click; **click tracking doesn't exist**. Permanently empty. | +| "80+ words, 0-100 score … all-caps density, exclamation ratios and hyperlink density" | The 3 named signals **don't exist**. Indian clichés score "Good". | +| "Native Model Context Protocol (MCP) SDK" | Hand-rolled `asyncio.StreamReader` loop. No `mcp` package. | +| "Twenty CRM & Webhook sync engine" | One endpoint, one unsigned POST, no events/retries/signing. | +| "server-side rendering" | Every route builds as `○ (Static)` — zero SSR. | +| "no built-in auth" | ✅ True — but it's the shipped Docker default, and CORS is `*` + credentials. | +| "196/196 passing" | ✅ True. But 9/12 e2e tests can't fail, and CI skips type-checking. | + +--- + +## 6. Suggested execution order + +**Day 1–2 — Unbreak it** (≈8h): B1 mailbox cap · B2 send gate · B5 `has_email` + pagination · +S20 auth+CORS · the one-line `generate_ai_icebreaker` import bug (500 on a shipped feature) · +S23 `purge_expired` predicate. + +**Day 3–4 — Make it fast** (≈8h): B4 `q=` split (3.4× coverage) · S6 HTTP-first scraping · +S1+S2+S3 enrichment concurrency + async verifier · S10 indexes + S12 SQLite pragmas · +S21 SSRF guard. + +**Day 5–6 — Make it true** (≈8h): B3 campaign scheduler + "waiting vs done" · +S22 bounce retry + sender-vs-recipient classification · S13 pattern inference rewrite · +S24 token fallback + preview parity · S18+S19 MCP contact-ids and scrape scoping. + +**Week 2 — Make it Apollo** (2 weeks): 2a People table + `/contacts` API · 2d live +ICP-relative scoring · 2b structured ICP filters + honest warnings · 2c real source-trust +feedback loop · S14/S15 decision-maker extraction · 2e analytics funnel · 2f A/B statistics · +S25 click tracking · S17 India-aware spam linter. + +**Week 3 — Make it ours** (2 weeks): GSTIN/Udyam registry · WhatsApp-first channel routing +· real festival/IST send windows · multilingual copy · directory ecosystem expansion +(JustDial/Sulekha/IndiaMART fixes) · rupee-denominated cost-vs-Apollo UI · a real design +system + dark mode + mobile. + +**Week 4 — Harden** (1 week): job queue with retries/DLQ · audit log · Prometheus metrics · +Alembic migrations (the dep is already in `requirements.txt` with zero migrations) · +Postgres in CI · e2e tests that actually fail the build. + +--- + +## 7. Measured outcomes + +| Metric | Today | After Waves 0–1 | After Wave 2 | +|---|---|---|---| +| Unique leads per scrape query | 30 | **103** | 103 + multi-source | +| Enrichment throughput | 9s/lead serial | **~1.1s/lead** | + persisted pattern cache | +| Scrape wall clock | 8.9s (Playwright) | **2.4s** (HTTP) | — | +| Leads/day per IP | ~2,000 | **~10,000** | — | +| Working data sources | 1 of 4 | **4 of 4** | + registry sources | +| Hard-bounce rate | 25–30% | **<5%** | <3% | +| Mailbox capacity | 50 **lifetime** | **50/day, forever** | + warmup ramp effective | +| `rank_existing` @100k | 13.6s | **<100ms** | — | +| Lead-list query @500k | ~345ms | **<100ms** | + FTS/trigram | +| CSV import 5k rows | 15.2s | **1–2s** | + streaming export | +| Event loop stall | 10s | **0s** | — | +| Named decision-maker found | 8–15% | 20–30% | **40%+** (People table) | +| Pattern detection | 4/10 | **8/10** | + per-domain persistence | +| API p99 during enrichment | seconds | **milliseconds** | — | +| Sequences running autonomously | 0% | **100%** | + branching | +| ICP filters expressible | 3 | 3 | **20+ attributes** | +| Analytics dimensions | 4 counters | 4 counters | **funnel + cohorts + heatmap** | diff --git a/backend/app/api/campaigns.py b/backend/app/api/campaigns.py index d7161fb..fd44dd2 100644 --- a/backend/app/api/campaigns.py +++ b/backend/app/api/campaigns.py @@ -2,6 +2,7 @@ from sqlalchemy.ext.asyncio import AsyncSession from sqlalchemy import select, func from pydantic import BaseModel +import math from typing import Optional, List, Dict, Any from app.database import get_db @@ -408,7 +409,16 @@ async def india_warm_handoff(campaign_id: int, db: AsyncSession = Depends(get_db "link and write a personal message."} #: Minimum sends per variant before an A/B winner may be declared. -AB_MIN_SENDS_PER_VARIANT = 10 +#: Cold-email reply rates sit at 1-3%, so at 10 sends per variant a single +#: extra reply is a 10pp swing - a coin flip would "declare" a winner. 50 +#: per variant gives a real sample. +AB_MIN_SENDS_PER_VARIANT = 50 + +#: Reply categories that count as a *positive* reply for A/B purposes. An +#: out-of-office auto-responder and an opt-out are not signals that a subject +#: line works - counting them let a vacation responder pick the winning +#: subject line, and an opt-out was counted as a success. +AB_POSITIVE_REPLIES = ("interested",) def decide_ab_winner(a_sent: int, a_replied: int, b_sent: int, b_replied: int, @@ -420,14 +430,31 @@ def decide_ab_winner(a_sent: int, a_replied: int, b_sent: int, b_replied: int, is deliberately NOT consulted: opens require the deliverability-killing open-tracking pixel, so with tracking off open counts are all zero and any open-based decision would be fiction. + + The comparison is a two-proportion z-test. The old code compared raw + rates with no significance test at all, so with 1-3% reply rates a single + reply decided the winner. """ if a_sent < min_sends or b_sent < min_sends: return None - a_rate = a_replied / a_sent - b_rate = b_replied / b_sent - if b_rate == a_rate: + if a_replied + b_replied < 3: + # Fewer than three replies in total: nothing to say. + return None + + p1, p2 = a_replied / a_sent, b_replied / b_sent + if p1 == p2: + return None + + pooled = (a_replied + b_replied) / (a_sent + b_sent) + if pooled <= 0 or pooled >= 1: + return None + se = math.sqrt(pooled * (1 - pooled) * (1 / a_sent + 1 / b_sent)) + if se == 0: + return None + z = abs(p1 - p2) / se + if z < 1.96: # ~95% confidence return None - return "b" if b_rate > a_rate else "a" + return "b" if p2 > p1 else "a" @router.get("/{campaign_id}/step-stats") @@ -462,8 +489,8 @@ async def get_step_stats(campaign_id: int, db: AsyncSession = Depends(get_db)): rows = [s for s in by_step.get(step.id, []) if getattr(s, "status", None) == "sent"] a = [s for s in rows if (getattr(s, "variant_used", None) or "a") == "a"] b = [s for s in rows if getattr(s, "variant_used", None) == "b"] - a_rep = sum(1 for s in a if s.replied) - b_rep = sum(1 for s in b if s.replied) + a_rep = sum(1 for s in a if (getattr(s, "reply_category", "") or "") in AB_POSITIVE_REPLIES) + b_rep = sum(1 for s in b if (getattr(s, "reply_category", "") or "") in AB_POSITIVE_REPLIES) is_ab = bool(getattr(step, "subject_b", None)) winner = getattr(step, "variant_winner", None) diff --git a/backend/app/api/leads.py b/backend/app/api/leads.py index 77aae1e..9ff8d99 100644 --- a/backend/app/api/leads.py +++ b/backend/app/api/leads.py @@ -17,6 +17,7 @@ from app.campaigns.sender import resolve_city from app.campaigns.suppression import is_suppressed, add_suppression from app.enricher.email_finder import calculate_lead_quality_score +from app.enricher.ai_icebreaker import generate_ai_icebreaker from app.dedupe.match import find_existing_business router = APIRouter() @@ -40,13 +41,17 @@ async def get_leads( search: Optional[str] = None, industry: Optional[str] = None, has_email: Optional[bool] = None, + sort: str = Query("created", pattern="^(created|score|name|rating)$"), + desc: bool = True, + min_score: Optional[int] = Query(None, ge=0, le=100), + source: Optional[str] = None, db: AsyncSession = Depends(get_db) ): offset = (page - 1) * limit - + # Base query for businesses stmt = select(Business) - + if search: search_term = f"%{search}%" stmt = stmt.where( @@ -61,6 +66,25 @@ async def get_leads( if industry: stmt = stmt.where(Business.industry.ilike(f"%{industry}%")) + # The "has email" filter is a SQL predicate, not a Python post-filter. + # + # It used to be applied inside the row loop, AFTER the LIMIT/OFFSET had + # already run, and it was missing from the count query entirely. That + # meant: `total` always reported the unfiltered count, and the rows that + # matched the filter were removed from the middle of an already-paged + # result set. On the Lead Database - the product's front door - a user + # filtering for "has email" saw 4 rows instead of 20 and could never + # reach page 2. + has_email_clause = None + if has_email is not None: + email_ids = select(Contact.business_id).where( + Contact.email.isnot(None), Contact.email != "" + ) + has_email_clause = ( + Business.id.in_(email_ids) if has_email else Business.id.notin_(email_ids) + ) + stmt = stmt.where(has_email_clause) + # Total count query count_stmt = select(func.count(Business.id)) if search: @@ -75,12 +99,30 @@ async def get_leads( ) if industry: count_stmt = count_stmt.where(Business.industry.ilike(f"%{industry}%")) + if has_email_clause is not None: + count_stmt = count_stmt.where(has_email_clause) + if min_score is not None: + stmt = stmt.where(Business.score >= min_score) + count_stmt = count_stmt.where(Business.score >= min_score) + if source: + stmt = stmt.where(Business.source == source) + count_stmt = count_stmt.where(Business.source == source) total_res = await db.execute(count_stmt) total = total_res.scalar() or 0 - # Paged query - stmt = stmt.order_by(Business.created_at.desc()).offset(offset).limit(limit) + # Paged query. Sorting is parameterised (the old hard-coded + # `created_at desc` meant "show me the hottest leads" was impossible + # without a code change). + _sort_columns = { + "created": Business.created_at, + "score": Business.score, + "name": Business.name, + "rating": Business.rating, + } + _sort_col = _sort_columns.get(sort, Business.created_at) + stmt = stmt.order_by(_sort_col.desc() if desc else _sort_col.asc()) + stmt = stmt.offset(offset).limit(limit) res = await db.execute(stmt) businesses = res.scalars().all() @@ -108,12 +150,6 @@ async def get_leads( for b in businesses: b_contacts = contacts_map.get(b.id, []) primary_contact = b_contacts[0] if b_contacts else None - - # Apply email filter if requested - if has_email is True and not any(c.get("email") for c in b_contacts): - continue - if has_email is False and any(c.get("email") for c in b_contacts): - continue # Compute Google Maps URL extra = b.extra_data if isinstance(b.extra_data, dict) else {} @@ -131,21 +167,38 @@ async def get_leads( operational_status = extra.get("operational_status", "Operational") price_tier = extra.get("price_tier") - if "lead_score" in extra and "lead_tier" in extra: - lead_score = extra["lead_score"] - lead_tier = extra["lead_tier"] - lead_badges = extra.get("lead_badges", []) - else: - score_info = calculate_lead_quality_score(b, b_contacts) - lead_score = score_info["score"] - lead_tier = score_info["tier"] - lead_badges = score_info["badges"] + # The lead score is computed LIVE, never read from the enrichment-time + # cache. + # + # The cache stored an ICP-relative score computed with whatever ICP + # was attached when the lead was enriched. Changing the ICP weights or + # switching preset therefore had *no* effect on the Lead Database - the + # stale cached value won, and scores from different ICPs were not + # comparable. Computing it live makes the score mean the same thing + # everywhere and makes ICP tuning visible immediately. + score_info = calculate_lead_quality_score(b, b_contacts) + lead_score = score_info["score"] + lead_tier = score_info["tier"] + lead_badges = score_info["badges"] + # Keep the column in sync so it can be sorted/filtered in SQL. + if b.score != lead_score: + b.score = lead_score # India source-trust: surfaces *why* a lead's score was adjusted — # self-listed directories (JustDial/IndiaMART) carry shared/stale # risk, live GBP and first-party sites do not. Advisory for the UI. + # + # The modifier reported here must be the one that was ACTUALLY + # applied to the score. Recomputing it independently meant the API + # advertised a trust bonus that the scorer had never applied (the + # primary scrape path's source values were missing from the trust + # table entirely), so the dashboard showed a badge next to a number + # that did not include it. from app.qualifier.source_trust import source_trust, trust_score_modifier _trust_val, _trust_note = source_trust(getattr(b, "source", None)) + _applied_modifier = score_info.get("trust_modifier") if score_info else None + if _applied_modifier is None: + _applied_modifier = trust_score_modifier(_trust_val) leads_data.append({ "id": b.id, @@ -168,12 +221,15 @@ async def get_leads( "source": getattr(b, "source", None), "source_trust": _trust_val, "trust_note": _trust_note, - "trust_modifier": trust_score_modifier(_trust_val), + "trust_modifier": _applied_modifier, "primary_contact": primary_contact, "contacts": b_contacts, "created_at": b.created_at.isoformat() if b.created_at else None }) + # Persist the refreshed scores so they are sortable/filterable in SQL. + await db.commit() + return { "total": total, "page": page, diff --git a/backend/app/auth.py b/backend/app/auth.py new file mode 100644 index 0000000..530834d --- /dev/null +++ b/backend/app/auth.py @@ -0,0 +1,64 @@ +"""Optional API-key authentication. + +LeadForge is a single-user, self-hosted tool, so the local clone-and-run +experience must stay zero-config: no login, no token, no setup wizard. But +the moment the API port is reachable from a LAN (which the README explicitly +recommends), an unauthenticated API with ``CORS_ORIGINS="*"`` lets any +website on the internet delete the lead database and send cold email from +the user's mailbox. + +The compromise: authentication is **off until a key is configured**. + + API_KEY= -> every router except the mail tracker requires it + API_KEY unset -> unchanged local behaviour + +The tracker (``/campaigns/tracker/*``) is deliberately exempt: it serves the +RFC 8058 one-click unsubscribe endpoint and the open-tracking pixel, which +are fetched by the recipient's mail client, not by the dashboard. +""" + +import logging +import secrets +from typing import Optional + +from fastapi import Depends, HTTPException, status +from fastapi.security import APIKeyHeader, HTTPAuthorizationCredentials, HTTPBearer + +from app.config import settings + +logger = logging.getLogger(__name__) + +_api_key_header = APIKeyHeader(name="X-API-Key", auto_error=False) +_bearer = HTTPBearer(auto_error=False) + + +def auth_enabled() -> bool: + """Whether the API key is actually being enforced.""" + return bool(settings.API_KEY) or bool(settings.REQUIRE_AUTH) + + +async def require_api_key( + x_api_key: Optional[str] = Depends(_api_key_header), + bearer: Optional[HTTPAuthorizationCredentials] = Depends(_bearer), +) -> None: + """FastAPI dependency enforcing the API key when one is configured.""" + if not auth_enabled(): + return + + expected = settings.API_KEY or "" + if not expected: + # REQUIRE_AUTH was set without a key - refuse everything rather than + # silently serving an open API. + logger.error("REQUIRE_AUTH is set but API_KEY is empty - refusing request") + raise HTTPException( + status_code=status.HTTP_503_SERVICE_UNAVAILABLE, + detail="API authentication is required but no API_KEY is configured", + ) + + supplied = x_api_key or (bearer.credentials if bearer else "") + if not supplied or not secrets.compare_digest(supplied, expected): + logger.warning("rejected request with missing/invalid API key") + raise HTTPException( + status_code=status.HTTP_401_UNAUTHORIZED, + detail="Invalid or missing API key", + ) diff --git a/backend/app/campaigns/bounces.py b/backend/app/campaigns/bounces.py index 410a3ae..8fc5506 100644 --- a/backend/app/campaigns/bounces.py +++ b/backend/app/campaigns/bounces.py @@ -65,6 +65,14 @@ #: Bare SMTP reply code at the end of the message, e.g. "(550 5.1.1 ...)" _CODE_RE = re.compile(r"\b([45]\d{2})\b") +#: Prefixes emitted by ``campaigns.sender`` when the failure happened on OUR +#: side of the SMTP transaction rather than the recipient's. A rejected +#: envelope sender (SPF misalignment, policy gate) or rejected message +#: content says nothing about the address we were trying to reach - and +#: classifying it as a hard bounce used to permanently suppress the entire +#: lead database the first time a sending domain misconfigured itself. +_SENDER_FAULT_PREFIXES = ("sender_rejected:", "data_rejected:") + def classify_bounce(error: Optional[str]) -> Optional[str]: """Maps an SMTP error string to ``hard_bounce`` / ``soft_bounce`` / None. @@ -79,6 +87,11 @@ def classify_bounce(error: Optional[str]) -> Optional[str]: if not text: return None + # Our sender or our content was rejected - the recipient is innocent. + if text.startswith(_SENDER_FAULT_PREFIXES): + logger.info("SMTP fault was ours, not the recipient's: %s", text[:120]) + return None + # An explicit 4xx always wins over free-text matching, because the numeric # class is authoritative under RFC 3463 and many gateways wrap a 4xx in # prose containing the word "rejected". @@ -91,12 +104,13 @@ def classify_bounce(error: Optional[str]) -> Optional[str]: return _classify_5xx(text) # No numeric code. Fall back to phrase matching, most specific first. + # Policy wins over hard: "blocked by policy" must never delete a lead. + if _POLICY_RE.search(text): + return None if _HARD_RE.search(text): return HARD if _SOFT_RE.search(text): return SOFT - if _POLICY_RE.search(text): - return None logger.debug("unclassifiable SMTP error, treating as soft: %s", text[:120]) return SOFT diff --git a/backend/app/campaigns/reply_poller.py b/backend/app/campaigns/reply_poller.py index c811ce5..509a50b 100644 --- a/backend/app/campaigns/reply_poller.py +++ b/backend/app/campaigns/reply_poller.py @@ -22,7 +22,7 @@ import imaplib import logging import re -from datetime import datetime, timedelta +from datetime import datetime, timedelta, timezone from email import message_from_bytes from email.header import decode_header from email.utils import parsedate_to_datetime @@ -198,7 +198,15 @@ def _poll_one_mailbox_sync(mailbox_id: int, email: str, password: str, body = extract_reply_body(msg) subject = _decode(msg.get("Subject")) auto = any(msg.get(h) for h in _AUTO_HEADERS) - inbound.append((num, refs, subject, body, auto)) + # Carry the headers through. They were collected and then + # discarded, which made the ``Auto-Submitted`` branch of + # ``categorize_reply`` dead code at runtime: an out-of-office + # auto-responder was classified from body text alone, and often + # not at all. That mattered twice over - it ended the follow-up + # chain on every lead whose owner travels, and an auto-responder + # could be counted as a "reply" and pick the A/B winning subject. + headers = {str(k): v for k, v in msg.items()} + inbound.append((num, refs, subject, body, auto, headers)) stats["matched"] = len(inbound) if not inbound: return stats @@ -252,9 +260,9 @@ async def poll_mailbox(mailbox_id: int) -> Dict[str, int]: out.update({k: int(fetch.get(k, 0)) for k in ("seen", "matched")}) if not inbound: return out - now = datetime.utcnow() + now = datetime.now(timezone.utc) async with async_session_maker() as db: - for _num, refs, subject, body, _auto in inbound: + for _num, refs, subject, body, _auto, headers in inbound: target_id = next((r for r in refs if r in known_ids), None) if not target_id: continue @@ -263,7 +271,10 @@ async def poll_mailbox(mailbox_id: int) -> Dict[str, int]: )).scalars().first() if not send or send.replied: continue - category, is_opt_out = categorize_reply(subject, body) + # Pass the headers so ``Auto-Submitted`` / ``X-Auto-Reply`` are + # honoured. Without them an out-of-office auto-responder is + # indistinguishable from a real reply. + category, is_opt_out = categorize_reply(subject, body, headers) send.replied = True send.replied_at = now send.reply_category = category diff --git a/backend/app/campaigns/sender.py b/backend/app/campaigns/sender.py index 643ca99..c86e092 100644 --- a/backend/app/campaigns/sender.py +++ b/backend/app/campaigns/sender.py @@ -1,4 +1,5 @@ import smtplib +import html from email.mime.text import MIMEText from email.mime.multipart import MIMEMultipart from email.utils import formatdate, make_msgid @@ -7,8 +8,10 @@ import asyncio import random import re -from datetime import datetime -from sqlalchemy import select, update +from datetime import datetime, timezone +from urllib.parse import quote +from zoneinfo import ZoneInfo +from sqlalchemy import select, update, or_ from app.config import settings from app.database import async_session_maker @@ -20,6 +23,7 @@ from app.models.send import Send from app.campaigns.bounces import classify_bounce, HARD, SOFT from app.campaigns.suppression import is_suppressed, add_suppression +from app.models.suppression import PERMANENT_REASONS from app.campaigns.warmup import ( mailbox_age_days, ramp_limit, mailbox_health, is_mailbox_usable, rolling_24h_count, recent_outcomes, @@ -27,15 +31,102 @@ logger = logging.getLogger(__name__) -def interpolate_template(text: str, data: Dict[str, str]) -> str: - """Replaces {{variable}} tags in subject/body with lead data.""" +#: IST - the operator's day. "Sent today" and the daily quota are counted on +#: the calendar day the salesperson actually lives in, not UTC. +IST = ZoneInfo("Asia/Kolkata") + +#: A send is only suppressed after this many transient failures. A single +#: gateway hiccup must never delete a good lead for 30 days. +SOFT_MAX_ATTEMPTS = 3 + +#: Verification states that make an address undeliverable. The sender refuses +#: these outright: mailing a known-dead address is a hard bounce, and hard +#: bounces are how a sending domain gets permanently blocklisted. +UNSENDABLE_STATUSES = ( + "bounced", "invalid", "syntax_error", "gibberish", "disposable", + "no_mx", "domain_error", "spamtrap", "abuse", "do_not_mail", "mailbox_full", +) + +#: Reply categories that must NOT stop a sequence. An out-of-office +#: auto-responder is not a conversation - treating it as one silently ends +#: the follow-up chain on every lead whose owner travels. +_NON_TERMINAL_REPLIES = ("out_of_office",) + +_TOKEN_RE = re.compile(r"\{\{\s*(\w+)\s*\}\}") + + +def _as_utc(dt: Optional[datetime]) -> Optional[datetime]: + """Normalises a datetime to timezone-aware UTC. + + SQLite hands back naive datetimes; Postgres (DateTime(timezone=True)) + hands back aware ones. Mixing the two makes ``aware - naive`` raise + TypeError, which broke the sequence delay math on Postgres outright. + """ + if dt is None: + return None + if dt.tzinfo is None: + return dt.replace(tzinfo=timezone.utc) + return dt.astimezone(timezone.utc) + + +def interpolate_template(text: str, data: Dict[str, str], *, escape: bool = False, + fallback: str = "") -> str: + """Replaces {{variable}} tags in subject/body with lead data. + + An unknown tag is *blanked*, never shipped literally. A stray + ``{{frist_name}}`` used to reach the prospect verbatim, which reads as an + obviously automated mail and poisons the sender's reputation. Unknown tags + are logged so the operator can fix the template. + """ if not text: return "" - result = text - for key, val in data.items(): - placeholder = f"{{{{{key}}}}}" - result = result.replace(placeholder, str(val or "")) - return result + missing: list[str] = [] + + def _sub(match: "re.Match[str]") -> str: + key = match.group(1) + if key not in data: + missing.append(key) + return fallback + value = str(data.get(key) or "") + return html.escape(value, quote=True) if escape else value + + rendered = _TOKEN_RE.sub(_sub, text) + if missing: + logger.warning( + "template references unknown token(s) %s - rendered blank", sorted(set(missing)) + ) + return rendered + + + +def _rewrite_links_for_tracking(body: str, tracking_base: str, send_id: Optional[int]) -> str: + """Rewrites ```` links to the click-tracking redirect. + + Click tracking did not exist: there was no ``/click`` route, the + ``Send.clicked`` column was never written, and the warm-WhatsApp handoff + panel (which needs three opens or one click) could therefore never + populate. Open tracking is deliberately off by default - a tracking pixel + is a primary-inbox killer - which left click tracking as the only + engagement signal available. + + The unsubscribe link is never rewritten: it must keep working even if + tracking is misconfigured, and it is appended after this runs anyway. + """ + if not send_id or not body: + return body + if "http" not in body: + return body + + def _replace(match: "re.Match[str]") -> str: + href = match.group(2) + if not href.startswith("http"): + return match.group(0) + if "unsubscribe" in href or "tracker/" in href: + return match.group(0) + tracker = f"{tracking_base.rstrip('/')}/api/v1/campaigns/tracker/click/{send_id}" + return f'{match.group(1)}"{tracker}?u={quote(href, safe="")}"' + + return re.sub(r'(href\s*=\s*")([^"]+)(")', _replace, body) def _domain_from_email(email: str) -> str: @@ -197,11 +288,43 @@ def send_single_email( server.quit() return True, "" + except smtplib.SMTPSenderRefused as e: + # OUR problem, not the recipient's. A rejected MAIL FROM (SPF + # misalignment, a blocked envelope sender, a policy gate) says nothing + # about the address we were trying to reach. Tagging it keeps the + # bounce classifier from suppressing the entire lead database. + logger.error("Sender rejected by %s for %s: %s", mailbox.smtp_host, to_email, e) + return False, f"sender_rejected:{e.smtp_code}:{e.smtp_error}" + except smtplib.SMTPRecipientsRefused as e: + # A genuine recipient refusal. smtplib gives us the per-recipient + # reply code, which is exactly what the classifier needs. The code is + # emitted FIRST so the classifier's code scanner cannot accidentally + # match digits inside the address itself. + details = "; ".join( + f"{code} {msg}" for _addr, (code, msg) in (e.recipients or {}).items() + ) + logger.error("Recipient refused by %s for %s: %s", mailbox.smtp_host, to_email, details) + return False, f"recipient_refused:{details}" + except smtplib.SMTPDataError as e: + # Message content rejected at DATA time - our problem, not theirs. + logger.error("Data rejected by %s for %s: %s", mailbox.smtp_host, to_email, e) + return False, f"data_rejected:{e.smtp_code}:{e.smtp_error}" except Exception as e: logger.error(f"Error sending email to {to_email} via {mailbox.email}: {e}") return False, str(e) +async def send_single_email_async(*args: Any, **kwargs: Any) -> Tuple[bool, str]: + """Runs the blocking SMTP handshake off the event loop. + + ``send_single_email`` performs a TCP connect, a TLS handshake and an + authentication round-trip - seconds of blocking I/O. Called directly from + an ``async def`` it freezes the whole API (and the in-process reply + poller) for the duration of every send. + """ + return await asyncio.to_thread(send_single_email, *args, **kwargs) + + from app.enricher.ai_icebreaker import generate_ai_icebreaker async def execute_campaign_step(campaign_id: int, max_sends: int = 50) -> Dict[str, Any]: @@ -221,7 +344,10 @@ async def execute_campaign_step(campaign_id: int, max_sends: int = 50) -> Dict[s # Single timestamp for the whole run. The mailbox warmup/ramp gate below # (mailbox_age_days / rolling_24h_count) and the per-step delay math both # read this, so it must be captured once, up front - not partway through. - now = datetime.utcnow() + # Timezone-aware: SQLite returns naive datetimes and Postgres returns + # aware ones, and mixing the two made the delay subtraction raise + # TypeError on Postgres outright. + now = datetime.now(timezone.utc) async with async_session_maker() as db: campaign = await db.get(Campaign, campaign_id) if not campaign or campaign.status != CampaignStatus.ACTIVE.value: @@ -242,16 +368,29 @@ async def execute_campaign_step(campaign_id: int, max_sends: int = 50) -> Dict[s step_ids_to_step = {s.id: s for s in steps} # Get available mailboxes, then apply the warmup/health/rolling gates. - # `sent_today < daily_limit` is only a fast pre-filter: the rolling - # 24h count over real Send timestamps is authoritative, a new mailbox - # ramps 10 -> limit over RAMP_DAYS, and a sick mailbox is held out - # until reviewed. Aged healthy mailboxes rotate first so fresh ones - # build reputation slowly - this is what keeps Gmail bulk-foldering - # at bay for new sending domains. - m_stmt = select(Mailbox).where(Mailbox.is_active == True, Mailbox.sent_today < Mailbox.daily_limit) + # The rolling 24h count over real Send timestamps is authoritative: a + # new mailbox ramps 10 -> limit over RAMP_DAYS, and a sick mailbox is + # held out until reviewed. Aged healthy mailboxes rotate first so + # fresh ones build reputation slowly - this is what keeps Gmail + # bulk-foldering at bay for new sending domains. + # + # NB: the mailbox's own ``sent_today`` counter is NOT used as a filter. + # It used to be, and because it was never reset it silently became a + # *lifetime* cap: after `daily_limit` sends the mailbox was excluded + # forever and the whole outreach engine stopped after 50 emails. + m_stmt = select(Mailbox).where(Mailbox.is_active == True) m_res = await db.execute(m_stmt) candidates_mb = list(m_res.scalars().all()) + # Reset the per-day counters for any mailbox whose last send was on a + # previous IST calendar day, so the dashboard's "sent today" is honest + # and the daily limit means *per day*. + today = now.astimezone(IST).strftime("%Y-%m-%d") + for mb in candidates_mb: + if getattr(mb, "sent_today_date", None) != today: + mb.sent_today = 0 + mb.sent_today_date = today + mailboxes = [] for mb in candidates_mb: if getattr(mb, "health_paused", False): @@ -280,11 +419,20 @@ async def execute_campaign_step(campaign_id: int, max_sends: int = 50) -> Dict[s if not mailboxes: return {"error": "No active mailboxes available with remaining daily quota"} - # Load candidate contacts who have an email, haven't unsubscribed, and haven't bounced. - # If this campaign has explicit targets (Apollo-style "Add to Sequence"), - # scope to them: a campaign must never email a lead the operator didn't - # pick. An empty target set keeps the legacy all-eligible behaviour so - # campaigns created before targeting existed still run. + # Load candidate contacts who have an email, haven't unsubscribed, and + # are actually deliverable. + # + # The verification gate used to exclude exactly one status ("bounced"), + # which meant `invalid`, `syntax_error`, `gibberish`, `disposable` and + # `no_mx` addresses were all mailed. The verifier goes to real lengths + # never to report a dead address as valid, and then that verdict was + # thrown away one file away - so a list of scraped Indian SMB addresses + # (where ~30% are junk) went out with a ~30% hard-bounce rate against + # a 5% bulk-sender ceiling. + # + # Un-enriched contacts (status NULL or "pending") stay eligible: we + # cannot know they are bad, and refusing them would silently disable + # outreach entirely. from app.models.campaign_step import CampaignTarget target_stmt = select(CampaignTarget.contact_id).where( CampaignTarget.campaign_id == campaign_id) @@ -297,7 +445,10 @@ async def execute_campaign_step(campaign_id: int, max_sends: int = 50) -> Dict[s .where( Contact.email.isnot(None), Contact.unsubscribed == False, - Contact.verification_status != "bounced" + or_( + Contact.verification_status.is_(None), + Contact.verification_status.notin_(UNSENDABLE_STATUSES), + ), ) ) if target_ids: @@ -320,6 +471,12 @@ async def execute_campaign_step(campaign_id: int, max_sends: int = 50) -> Dict[s sends_by_contact.setdefault(s.contact_id, []).append(s) queue_to_send: List[Tuple[Contact, Business, CampaignStep]] = [] + #: Contacts still inside the sequence whose next step is waiting on a + #: delay. The old code could not tell "everyone finished" apart from + #: "everyone is waiting", so a campaign was marked COMPLETED the + #: moment it was launched and the dashboard showed in-flight + #: sequences as done. + pending_delays = 0 for contact, business in candidates: c_sends = sends_by_contact.get(contact.id, []) @@ -331,7 +488,12 @@ async def execute_campaign_step(campaign_id: int, max_sends: int = 50) -> Dict[s # immediately before we would open an SMTP connection. suppression_reason = await is_suppressed(db, contact.email) if suppression_reason: - contact.unsubscribed = True + # Only a genuine opt-out flips the per-contact flag. A soft + # bounce has a 30-day expiry, and setting the flag for it made + # a transient gateway failure remove the lead permanently. + if suppression_reason in PERMANENT_REASONS or suppression_reason == "soft_bounce": + if suppression_reason in ("unsubscribed", "replied_stop", "complaint", "manual"): + contact.unsubscribed = True if suppression_reason == HARD: contact.verification_status = "bounced" contact.is_verified = False @@ -340,8 +502,18 @@ async def execute_campaign_step(campaign_id: int, max_sends: int = 50) -> Dict[s ) continue - # If recipient already replied or bounced, skip them - if any(s.replied for s in c_sends) or any(s.status == "bounced" for s in c_sends): + # Stop on a real reply. An out-of-office auto-responder is not a + # conversation - ending the follow-up chain on it silently cost us + # the second and third touches on every lead whose owner travels. + def _is_terminal(send: Send) -> bool: + if send.status == "bounced": + return True + if not send.replied: + return False + category = (send.reply_category or "").lower() + return category not in _NON_TERMINAL_REPLIES + + if any(_is_terminal(s) for s in c_sends): continue if not c_sends: @@ -371,14 +543,35 @@ async def execute_campaign_step(campaign_id: int, max_sends: int = 50) -> Dict[s if next_step_num in steps_by_number: next_step = steps_by_number[next_step_num] # Check delay requirement in days - days_elapsed = (now - last_send.sent_at).total_seconds() / 86400.0 - if days_elapsed >= next_step.delay_days: + last_sent_at = _as_utc(last_send.sent_at) or now + days_elapsed = (now - last_sent_at).total_seconds() / 86400.0 + if days_elapsed >= (next_step.delay_days or 0): queue_to_send.append((contact, business, next_step)) + else: + # Still waiting on the step delay - the sequence is + # alive, it just is not due yet. + pending_delays += 1 + else: + # No further step configured: this contact is finished. + pass if len(queue_to_send) >= max_sends: break if not queue_to_send: + if pending_delays > 0: + # The sequence is mid-flight and simply waiting for the next + # step's delay to elapse. Leave it ACTIVE so the scheduler + # picks it up when it becomes due. + logger.info( + "campaign %s: %d contact(s) waiting on a step delay", campaign_id, pending_delays + ) + await db.commit() + return { + "message": f"{pending_delays} contact(s) are waiting on the next step delay", + "sent": 0, + "pending": pending_delays, + } # Check if all active contacts have finished all steps campaign.status = CampaignStatus.COMPLETED.value await db.commit() @@ -437,7 +630,12 @@ async def execute_campaign_step(campaign_id: int, max_sends: int = 50) -> Dict[s if use_variant == "b": rendered_subject = interpolate_template(step.subject_b, data) rendered_body_text = interpolate_template(step.body_text, data) - rendered_body_html = interpolate_template(step.body_html, data) if step.body_html else None + # HTML body: escape interpolated values so a business named + # "Suresh & Sons " cannot corrupt the markup. + rendered_body_html = ( + interpolate_template(step.body_html, data, escape=True) + if step.body_html else None + ) outbox_message_id = make_msgid(domain=_domain_from_email(mailbox.email)) send_record = Send( @@ -445,14 +643,27 @@ async def execute_campaign_step(campaign_id: int, max_sends: int = 50) -> Dict[s step_id=step.id, contact_id=contact.id, mailbox_id=mailbox.id, - status="pending", + status="sending", message_id=outbox_message_id, - variant_used=use_variant + variant_used=use_variant, + attempt=1, ) db.add(send_record) - await db.flush() + # Commit the intent to send BEFORE the SMTP call. If the process + # dies mid-flight the row survives as "sending" and the reaper can + # reconcile it; previously the row was only flushed, so a crash + # between delivery and commit left the prospect with a delivered + # email and no record - and the next run mailed them again. + await db.commit() + + # Now that the send row exists we know its id, so links can be + # rewritten to the click tracker. + if rendered_body_html: + rendered_body_html = _rewrite_links_for_tracking( + rendered_body_html, tracking_base, send_record.id + ) - success, err = send_single_email( + success, err = await send_single_email_async( mailbox=mailbox, to_email=contact.email, subject=rendered_subject, @@ -473,12 +684,14 @@ async def execute_campaign_step(campaign_id: int, max_sends: int = 50) -> Dict[s if success: send_record.status = "sent" - send_record.sent_at = datetime.utcnow() - mailbox.sent_today += 1 + send_record.sent_at = datetime.now(timezone.utc) + mailbox.sent_today = (mailbox.sent_today or 0) + 1 + mailbox.sent_today_date = today sent_count += 1 elif bounce_reason == HARD: send_record.status = "bounced" send_record.error_message = err + send_record.bounced = True contact.verification_status = "bounced" contact.is_verified = False await add_suppression(db, contact.email, HARD, detail=err) @@ -486,13 +699,22 @@ async def execute_campaign_step(campaign_id: int, max_sends: int = 50) -> Dict[s elif bounce_reason == SOFT: send_record.status = "failed" send_record.error_message = err - # Transient: do not mark the lead bad. A soft suppression with - # an expiry guards the next batch while the mailbox recovers. - await add_suppression(db, contact.email, SOFT, detail=err) + # Transient: do not mark the lead bad, and do not suppress it + # yet either. A soft suppression with an expiry guards the + # next batch while the mailbox recovers - but only once the + # retry budget is exhausted. One gateway hiccup used to park + # a perfectly good lead for 30 days. + prior_attempts = sum( + 1 for s in sends_by_contact.get(contact.id, []) + if s.status == "failed" and s.step_id == step.id + ) + if prior_attempts + 1 >= SOFT_MAX_ATTEMPTS: + await add_suppression(db, contact.email, SOFT, detail=err) failed_count += 1 else: - # Not a recipient problem (policy block / unknown). Retry, but - # never suppress - the address is fine, our sender is not. + # Not a recipient problem (policy block / unknown / our own + # sender misconfiguration). Retry, but never suppress - the + # address is fine, our sender is not. send_record.status = "failed" send_record.error_message = err failed_count += 1 diff --git a/backend/app/campaigns/suppression.py b/backend/app/campaigns/suppression.py index 64785e5..833d66d 100644 --- a/backend/app/campaigns/suppression.py +++ b/backend/app/campaigns/suppression.py @@ -121,6 +121,12 @@ async def add_suppression( return False existing.reason = reason existing.detail = detail + # A row created as a soft bounce carries an expiry. If it is later + # upgraded to a permanent reason the expiry has to go, otherwise the + # hard-bounce suppression silently ages out and the dead address + # becomes mailable again. + if reason in PERMANENT_REASONS: + existing.expires_at = None await db.flush() return False @@ -131,10 +137,13 @@ async def add_suppression( async def purge_expired(db: AsyncSession) -> int: """Deletes soft-bounce rows whose retry window has passed.""" + from datetime import datetime, timezone + + now = datetime.now(timezone.utc) result = await db.execute( delete(Suppression).where( Suppression.expires_at.isnot(None), - Suppression.expires_at < Suppression.created_at, + Suppression.expires_at < now, ) ) await db.flush() diff --git a/backend/app/campaigns/tracker.py b/backend/app/campaigns/tracker.py index 6a16140..e6e0e02 100644 --- a/backend/app/campaigns/tracker.py +++ b/backend/app/campaigns/tracker.py @@ -1,6 +1,7 @@ from fastapi import APIRouter, Depends, Request, Response, status as http_status from sqlalchemy.ext.asyncio import AsyncSession from sqlalchemy import update +from urllib.parse import quote, urlparse, parse_qs from app.database import get_db from app.models.send import Send from app.campaigns.suppression import add_suppression @@ -11,16 +12,40 @@ # 1x1 transparent GIF TRANSPARENT_PIXEL = base64.b64decode("R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7") + @router.get("/open/{send_id}") async def track_open(send_id: int, db: AsyncSession = Depends(get_db)): # Update send status asynchronously stmt = update(Send).where(Send.id == send_id).values(opened=True) await db.execute(stmt) await db.commit() - + return Response(content=TRANSPARENT_PIXEL, media_type="image/gif") +@router.get("/click/{send_id}") +async def track_click(send_id: int, request: Request, db: AsyncSession = Depends(get_db)): + """Click tracking: records the click, then redirects to the real URL. + + This route did not exist at all. The ``Send.clicked`` column was never + written by anything, and the warm-WhatsApp handoff panel requires either + three opens or one click - so with open tracking off by default, that + panel could never populate and the product's flagship India feature was + permanently empty. + """ + target = request.query_params.get("u") or "/" + stmt = update(Send).where(Send.id == send_id).values(clicked=True) + await db.execute(stmt) + await db.commit() + + # Only ever redirect to an absolute http(s) URL - an open redirect on a + # tracking endpoint is a phishing primitive. + parsed = urlparse(target) + if parsed.scheme in ("http", "https") and parsed.netloc: + return Response(status_code=302, headers={"Location": target}) + return Response(status_code=302, headers={"Location": "/"}) + + async def _suppress_contact(contact_id: int, db: AsyncSession) -> None: """Marks a contact unsubscribed and records a permanent global suppression. diff --git a/backend/app/cli.py b/backend/app/cli.py index a7ec88e..ddc0d1f 100644 --- a/backend/app/cli.py +++ b/backend/app/cli.py @@ -10,7 +10,7 @@ sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))) from sqlalchemy import select, or_, func -from app.database import async_session_maker +from app.database import async_session_maker, ensure_schema from app.models.business import Business from app.models.contact import Contact from app.models.scrape_job import ScrapeJob @@ -59,7 +59,7 @@ def format_markdown_table(leads: List[Dict[str, Any]]) -> str: return "\n".join(lines) -async def cmd_scrape(query: str, limit: int = 10, auto_enrich: bool = True, output_format: str = "markdown"): +async def cmd_scrape(query: str, limit: int = 10, auto_enrich: bool = True, output_format: str = "markdown") -> int: """Scrapes leads from Google Maps, saves them, enriches them, and outputs results.""" # 1. Create scrape job job_id = None @@ -71,11 +71,19 @@ async def cmd_scrape(query: str, limit: int = 10, auto_enrich: bool = True, outp job_id = job.id # 2. Run scraping - saved_count, _new_biz_ids = await scrape_leads_and_save(query=query, max_results=limit, job_id=job_id) - - # 3. Query the freshly saved/updated leads + saved_count, new_biz_ids = await scrape_leads_and_save(query=query, max_results=limit, job_id=job_id) + + # 3. Query the freshly saved/updated leads. + # + # Scoped to the businesses THIS scrape created. It used to return the last + # N businesses in the whole database, so a scrape for "Dentists in Pune" + # against a database that already held 75 real-estate leads printed 75 + # real-estate agents under a "Dentists in Pune" heading. async with async_session_maker() as db: - stmt = select(Business).order_by(Business.id.desc()).limit(limit) + if new_biz_ids: + stmt = select(Business).where(Business.id.in_(new_biz_ids)).order_by(Business.id) + else: + stmt = select(Business).where(Business.id.is_(None)) res = await db.execute(stmt) businesses = res.scalars().all() @@ -113,9 +121,10 @@ async def cmd_scrape(query: str, limit: int = 10, auto_enrich: bool = True, outp print(f"### LeadForge Results for: \"{query}\"\n") print(f"Discovered **{len(leads_data)}** listings ({saved_count} new leads saved into database).\n") print(format_markdown_table(leads_data)) + return 0 if leads_data else 1 -async def cmd_search(search_term: Optional[str] = None, industry: Optional[str] = None, has_email: Optional[bool] = None, limit: int = 25, output_format: str = "markdown"): +async def cmd_search(search_term: Optional[str] = None, industry: Optional[str] = None, has_email: Optional[bool] = None, limit: int = 25, output_format: str = "markdown") -> int: async with async_session_maker() as db: stmt = select(Business) if search_term: @@ -124,6 +133,19 @@ async def cmd_search(search_term: Optional[str] = None, industry: Optional[str] if industry: stmt = stmt.where(Business.industry.ilike(f"%{industry}%")) + # The "has email" filter is applied in SQL, BEFORE the limit. Filtering + # after the limit meant the --limit parameter changed the answer: + # "--limit 5 --has-email" returned 0 matches while "--limit 100" found + # 6, because the filter ran on an already-truncated set. + if has_email is not None: + email_ids = select(Contact.business_id).where( + Contact.email.isnot(None), Contact.email != "" + ) + stmt = stmt.where( + Business.id.in_(email_ids) if has_email + else Business.id.notin_(email_ids) + ) + stmt = stmt.order_by(Business.id.desc()).limit(limit) res = await db.execute(stmt) businesses = res.scalars().all() @@ -139,11 +161,6 @@ async def cmd_search(search_term: Optional[str] = None, industry: Optional[str] for c in contacts if c.email ] - if has_email is True and not emails: - continue - if has_email is False and emails: - continue - clean_phone = (b.phone or "").replace(" ", "").replace("-", "") wa_link = f"https://wa.me/{clean_phone}" if clean_phone else None maps_url = b.maps_url or f"https://www.google.com/maps/search/?api=1&query={urllib.parse.quote((b.name + ' ' + (b.address or '')).strip())}" @@ -165,14 +182,16 @@ async def cmd_search(search_term: Optional[str] = None, industry: Optional[str] print(json.dumps({"total": len(leads_data), "leads": leads_data}, indent=2)) else: print(format_markdown_table(leads_data)) + return 0 if leads_data else 1 -async def cmd_enrich(business_id: int): +async def cmd_enrich(business_id: int) -> int: res = await enrich_business_by_id(business_id) print(json.dumps(res, indent=2)) + return 0 -async def cmd_stats(): +async def cmd_stats() -> int: async with async_session_maker() as db: res_b = await db.execute(select(func.count(Business.id))) total_leads = res_b.scalar() or 0 @@ -186,19 +205,22 @@ async def cmd_stats(): "total_contacts": total_contacts, "verified_contacts": verified_contacts }, indent=2)) + return 0 -async def cmd_spam(subject: str, body: str): +async def cmd_spam(subject: str, body: str) -> int: res = check_cold_email_spam(subject, body) print(json.dumps(res, indent=2)) + # A copy that trips the spam linter is a failure the caller must see. + return 0 if res.get("is_safe", True) else 1 -async def cmd_icebreaker(business_id: int): +async def cmd_icebreaker(business_id: int) -> int: async with async_session_maker() as db: b = await db.get(Business, business_id) if not b: print(json.dumps({"error": f"Business ID {business_id} not found"})) - return + return 1 c_stmt = select(Contact).where(Contact.business_id == b.id).limit(1) c_res = await db.execute(c_stmt) c = c_res.scalar_one_or_none() @@ -214,6 +236,7 @@ async def cmd_icebreaker(business_id: int): first_name=first_name ) print(json.dumps(res, indent=2)) + return 0 def main(): @@ -230,7 +253,8 @@ def main(): p_search = subparsers.add_parser("search", help="Search existing leads in the database") p_search.add_argument("--query", type=str, default=None, help="Search term") p_search.add_argument("--industry", type=str, default=None, help="Industry filter") - p_search.add_argument("--has-email", action="store_true", default=None, help="Only leads with email") + p_search.add_argument("--has-email", action="store_true", default=None, help="Only leads with an email address") + p_search.add_argument("--no-email", action="store_true", default=None, help="Only leads WITHOUT an email address") p_search.add_argument("--limit", type=int, default=25, help="Limit") p_search.add_argument("--format", choices=["markdown", "json"], default="markdown", help="Output format") @@ -252,18 +276,38 @@ def main(): args = parser.parse_args() - if args.command == "scrape": - asyncio.run(cmd_scrape(query=args.query, limit=args.limit, output_format=args.format)) - elif args.command == "search": - asyncio.run(cmd_search(search_term=args.query, industry=args.industry, has_email=args.has_email, limit=args.limit, output_format=args.format)) - elif args.command == "enrich": - asyncio.run(cmd_enrich(business_id=args.business_id)) - elif args.command == "spam": - asyncio.run(cmd_spam(subject=args.subject, body=args.body)) - elif args.command == "icebreaker": - asyncio.run(cmd_icebreaker(business_id=args.business_id)) - elif args.command == "stats": - asyncio.run(cmd_stats()) + # The CLI talks to the database directly, so it must bring the schema up + # to date itself. It used to rely on the API process having started + # first, which meant a fresh clone failed with "no such column". + try: + asyncio.run(ensure_schema()) + except Exception as schema_err: + print(json.dumps({"error": f"schema sync failed: {schema_err}"}), file=sys.stderr) + raise SystemExit(1) + + # Every command returns a process exit code. A CLI that always exits 0 + # cannot be scripted: an agent (or a shell pipeline) has no way to tell + # "not found" from "found". Errors now exit non-zero and print to stderr. + exit_code = 0 + try: + if args.command == "scrape": + exit_code = asyncio.run(cmd_scrape(query=args.query, limit=args.limit, output_format=args.format)) + elif args.command == "search": + _has_email = True if args.has_email else (False if args.no_email else None) + exit_code = asyncio.run(cmd_search(search_term=args.query, industry=args.industry, has_email=_has_email, limit=args.limit, output_format=args.format)) + elif args.command == "enrich": + exit_code = asyncio.run(cmd_enrich(business_id=args.business_id)) + elif args.command == "spam": + exit_code = asyncio.run(cmd_spam(subject=args.subject, body=args.body)) + elif args.command == "icebreaker": + exit_code = asyncio.run(cmd_icebreaker(business_id=args.business_id)) + elif args.command == "stats": + exit_code = asyncio.run(cmd_stats()) + except Exception as exc: + print(json.dumps({"error": f"{type(exc).__name__}: {exc}"}), file=sys.stderr) + raise SystemExit(1) + + raise SystemExit(exit_code) if __name__ == "__main__": diff --git a/backend/app/config.py b/backend/app/config.py index ea3cf93..5230ffe 100644 --- a/backend/app/config.py +++ b/backend/app/config.py @@ -32,6 +32,16 @@ class Settings(BaseSettings): #: browser. "*" is convenient locally but lets any website drive your API #: with the user's cookies, so production requires an explicit list. CORS_ORIGINS: str = "*" + + #: Optional shared secret for the REST API. When unset, the API is open - + #: which is the right default for a local single-user clone-and-run, and + #: the wrong default the moment the port is reachable from a LAN. Set + #: API_KEY and every router except the mail-facing tracker requires it. + API_KEY: Optional[str] = None + + #: Require the API key even when API_KEY is unset. Useful for a hardened + #: deployment that generates the key at boot. + REQUIRE_AUTH: bool = False # Database settings # Default to sqlite for instant local operation; can be overridden via DATABASE_URL in .env diff --git a/backend/app/database.py b/backend/app/database.py index cfc7027..623162f 100644 --- a/backend/app/database.py +++ b/backend/app/database.py @@ -1,9 +1,13 @@ from typing import AsyncGenerator +import logging +from sqlalchemy import event, text from sqlalchemy.ext.asyncio import create_async_engine, AsyncSession, async_sessionmaker from sqlalchemy.orm import declarative_base from app.config import settings +logger = logging.getLogger(__name__) + is_sqlite = settings.DATABASE_URL.startswith("sqlite") engine_kwargs = {"echo": False, "future": True} @@ -12,15 +16,132 @@ else: engine_kwargs["pool_size"] = 10 engine_kwargs["max_overflow"] = 20 + # Postgres providers close idle connections (RDS does it after 5 minutes). + # Without pre-ping the first request after an idle period dies with an + # InterfaceError instead of transparently opening a fresh connection. + engine_kwargs["pool_pre_ping"] = True + engine_kwargs["pool_recycle"] = 1800 engine = create_async_engine(settings.DATABASE_URL, **engine_kwargs) + +if is_sqlite: + @event.listens_for(engine.sync_engine, "connect") + def _sqlite_pragmas(dbapi_conn, _record): + """Apply the SQLite pragmas SQLAlchemy does not set for you. + + * ``foreign_keys=ON`` - SQLite disables foreign keys per-connection by + default, so every ``ondelete="CASCADE"`` in the models was a no-op + and deleting a lead orphaned its contacts. + * ``journal_mode=WAL`` - the default rollback journal makes a write + block every reader for the duration of the transaction, which meant a + long CSV import froze the whole application. + * ``busy_timeout`` - without it a concurrent writer raises + "database is locked" immediately instead of waiting. + """ + cursor = dbapi_conn.cursor() + try: + cursor.execute("PRAGMA foreign_keys=ON") + cursor.execute("PRAGMA journal_mode=WAL") + cursor.execute("PRAGMA synchronous=NORMAL") + cursor.execute("PRAGMA busy_timeout=5000") + finally: + cursor.close() + + async_session_maker = async_sessionmaker( engine, class_=AsyncSession, expire_on_commit=False, autoflush=False ) Base = declarative_base() + +async def ensure_schema() -> None: + """Create missing tables, columns and indexes. + + Every entry point calls this - the HTTP API, the CLI and the MCP server. + It used to run only in the API's lifespan, so the CLI and the MCP server + crashed with "no such column" the moment a model added a column. That + made the agent surface a second-class citizen that could not survive a + schema change. + + Additive only: it creates what is missing and never drops or retypes + anything, so it is safe to run on every boot. + """ + from sqlalchemy import inspect as sa_inspect + + async with engine.begin() as conn: + await conn.run_sync(Base.metadata.create_all) + await conn.run_sync(_sync_missing_columns) + await conn.run_sync(_sync_missing_indexes) + + +def _sync_missing_columns(sync_conn) -> None: + """ALTER TABLE ADD COLUMN for any model column the live database lacks. + + Deriving from the models means a new column needs no migration + bookkeeping, which is what self-hosted users actually need. + """ + from sqlalchemy import inspect as sa_inspect + + inspector = sa_inspect(sync_conn) + existing_tables = set(inspector.get_table_names()) + dialect = sync_conn.dialect + + for table in Base.metadata.sorted_tables: + if table.name not in existing_tables: + continue + have = {c["name"] for c in inspector.get_columns(table.name)} + for col in table.columns: + if col.name in have: + continue + ddl = f'ALTER TABLE "{table.name}" ADD COLUMN "{col.name}" {col.type.compile(dialect)}' + default = getattr(col.default, "arg", None) + if default is not None and not callable(default): + ddl += f" DEFAULT {default!r}" if isinstance(default, str) else f" DEFAULT {default}" + try: + sync_conn.execute(text(ddl)) + logger.info("schema sync: added %s.%s", table.name, col.name) + except Exception as exc: # pragma: no cover - dialect-specific limits + logger.warning("schema sync skipped %s.%s: %s", table.name, col.name, exc) + + +def _sync_missing_indexes(sync_conn) -> None: + """Creates any index the models declare that the live database lacks. + + The column sync only ever added columns. That meant adding + ``index=True`` to a model did nothing for existing deployments - so the + indexes that make the lead list fast (contacts.business_id, + businesses.created_at, sends.campaign_id, ...) were missing on every + database created before the model changed. On a 500k-row table that is + the difference between a 42ms lookup and a full scan. + + Idempotent and safe to run on every boot. + """ + from sqlalchemy import inspect as sa_inspect + + inspector = sa_inspect(sync_conn) + existing_tables = set(inspector.get_table_names()) + + for table in Base.metadata.sorted_tables: + if table.name not in existing_tables: + continue + present = {idx["name"] for idx in inspector.get_indexes(table.name)} + for index in table.indexes: + if index.name in present: + continue + try: + # Compile against the *sync* dialect. The async engine's + # dialect cannot execute DDL, and `Index.create(bind=...)` + # expects a Connection, not a dialect. + from sqlalchemy.schema import CreateIndex + ddl = str(CreateIndex(index).compile(dialect=sync_conn.dialect)) + sync_conn.execute(text(ddl)) + logger.info("schema sync: created index %s", index.name) + except Exception as exc: # pragma: no cover - dialect-specific limits + logger.warning("schema sync skipped index %s: %s", index.name, exc) + + async def get_db() -> AsyncGenerator[AsyncSession, None]: async with async_session_maker() as session: try: diff --git a/backend/app/discovery/pipeline.py b/backend/app/discovery/pipeline.py index bdf0cf6..abee224 100644 --- a/backend/app/discovery/pipeline.py +++ b/backend/app/discovery/pipeline.py @@ -1,3 +1,4 @@ +import asyncio import logging from datetime import datetime from typing import Any, Dict, List, Optional @@ -16,19 +17,43 @@ logger = logging.getLogger(__name__) +#: How many leads to enrich at once. Each enrichment is a multi-request +#: waterfall (search engines, a website crawl, an SMTP verification), so it +#: is almost entirely I/O bound - concurrency is close to free. +ENRICH_CONCURRENCY = 8 -async def _enrich_new_leads(business_ids: List[int]) -> int: - """ICP-aware enrichment for freshly discovered leads (best-effort).""" + +async def _enrich_new_leads(business_ids: List[int], + concurrency: int = ENRICH_CONCURRENCY) -> int: + """ICP-aware enrichment for freshly discovered leads (best-effort). + + Enrichment used to run strictly one lead at a time. Measured at ~9s per + lead, so a 100-lead discovery job spent roughly 15 minutes - inside the + FastAPI event loop, with no other request served. Running the leads + concurrently (each one is I/O bound) brings that to ~2 minutes. + + Failures are isolated: one lead that raises must not cancel the others. + """ from app.enricher.email_finder import enrich_business_by_id - enriched = 0 - for business_id in business_ids: - try: - await enrich_business_by_id(business_id) - enriched += 1 - except Exception as e: - logger.warning(f"Enrichment failed for business {business_id}: {e}") - return enriched + if not business_ids: + return 0 + + semaphore = asyncio.Semaphore(max(1, concurrency)) + + async def _one(business_id: int) -> int: + async with semaphore: + try: + await enrich_business_by_id(business_id) + return 1 + except asyncio.CancelledError: + raise + except Exception as e: + logger.warning(f"Enrichment failed for business {business_id}: {e}") + return 0 + + results = await asyncio.gather(*(_one(bid) for bid in business_ids)) + return sum(results) async def _record_source_stats( diff --git a/backend/app/enricher/email_finder.py b/backend/app/enricher/email_finder.py index 98c64cf..7d4c372 100644 --- a/backend/app/enricher/email_finder.py +++ b/backend/app/enricher/email_finder.py @@ -12,7 +12,13 @@ from app.models.business import Business from app.models.contact import Contact from app.sources import in_phone -from app.verifier.smtp_verifier import verify_email, verify_batch, check_catch_all_domain +from app.verifier.smtp_verifier import ( + verify_email, + verify_email_async, + verify_batch, + check_catch_all_domain, + check_catch_all_domain_async, +) from app.verifier.email_status import detect_role_account, detect_placeholder_domain from app.enricher.pattern import generate_candidates @@ -29,6 +35,97 @@ # between sentry.io/example.com and a company's actual mail domain. EXCLUDED_EMAIL_DOMAINS = ('sentry.io', 'example.com', 'wixpress.com', 'domain.com', 'email.com', 'schema.org') +#: Call-to-action and boilerplate phrases that the title regex used to +#: mistake for people's names. "Read more - owner" became the contact +#: "read more" with title "owner", and that garbage then consumed the +#: candidate budget (read.more@domain, call.now@domain). +_NOT_A_PERSON_NAME = frozenset({ + "read more", "learn more", "view more", "see more", "click here", + "call now", "call us", "book now", "book appointment", "book online", + "get in touch", "contact us", "join us", "about us", "our services", + "why choose us", "our team", "the team", "meet the team", "our story", + "privacy policy", "terms of use", "terms and conditions", "refund policy", + "success stories", "client reviews", "our clients", "our projects", + "request a quote", "request callback", "send a message", "live chat", + "chat with us", "more info", "know more", "explore more", "shop now", + "order now", "buy now", "download", "download now", "apply now", +}) + + +def _is_plausible_person_name(name: str) -> bool: + """Whether a scraped "name" is actually a person. + + Two or three capitalised tokens that are not boilerplate and not a + known CTA phrase. + """ + if not name: + return False + cleaned = re.sub(r"\s+", " ", name.strip().lower()) + if cleaned in _NOT_A_PERSON_NAME: + return False + tokens = cleaned.split() + if len(tokens) not in (2, 3): + return False + # Reject page/section words that are never part of a person's name. + if any(t in ("about", "contact", "home", "terms", "service", "privacy", + "policy", "page", "our", "the", "your", "and", "for") + for t in tokens): + return False + return True + + +def _exec_for_address(email: str, execs: List[Dict[str, Any]]) -> Optional[Dict[str, Any]]: + """Finds the executive an address plausibly belongs to. + + A shared mailbox (``info@``, ``sales@``) belongs to nobody. A person + address carries their name (or its first letter) in the local part. + Returns None when no executive can be bound - which is the correct + answer for a role mailbox. + """ + if not email or "@" not in email or not execs: + return None + local = email.split("@")[0].lower().strip() + if not local or local in _NOT_A_PERSON_NAME: + return None + # A purely alphabetic local part that is a common role word is a mailbox. + if local in ("contact", "info", "sales", "office", "support", "admin", "hello"): + return None + + tokens = [t for t in re.split(r"[._+\-]", local) if t] + for ex in execs: + name = (ex.get("name") or "").strip() + if not name: + continue + parts = name.lower().split() + first = parts[0] + last = parts[1] if len(parts) > 1 else "" + if first and first in tokens: + return ex + if last and last in tokens: + return ex + # Initials: "rsharma" for Rajesh Sharma. + if first and last and len(first) == 1 and tokens and tokens[0] == first: + return ex + return None + + +def _address_source(email: str, harvested: Any, permutations: List[str], + domain: Optional[str]) -> str: + """Where this address came from. + + Every contact used to be stamped ``multi_source_enricher``, which made it + impossible to measure the thing that actually matters: how often a + *guessed* mailbox is right versus a harvested one. + """ + local = email.split("@")[0].lower() + if local in ("contact", "info", "sales", "office"): + return "alias" + if permutations and email in {p.lower() for p in permutations}: + return "pattern_guess" + if domain and email.rpartition("@")[2].lower() == domain.lower(): + return "website_scrape" + return "web_search" + CONTACT_PRIORITY_KEYWORDS = [ 'contact', 'about', 'reach', 'touch', 'team', 'connect', 'help', 'info', 'support', 'impressum' ] @@ -234,12 +331,24 @@ async def crawl_website_for_contacts(url: str, max_pages: int = 4) -> Dict[str, "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" } + # Titles that actually appear on Indian SMB sites. The old list missed + # MD, Principal (e.g. "Principal Dentist" - the dental norm), General + # Manager, Promoter and Chairman, which together cover the majority of + # owner-operator titles in the target market. EXEC_TITLES = [ - r'founder', r'co-founder', r'ceo', r'managing director', r'director', - r'principal broker', r'president', r'owner', r'partner', r'proprietor' + r'founder', r'co-founder', r'ceo', r'chief executive officer', + r'managing director', r'md', r'director', r'principal(?:\s+\w+)?', + r'general manager', r'manager', r'promoter', r'chairman', + r'chairperson', r'president', r'owner', r'partner', r'proprietor', + r'administrator', r'head of \w+', r'vice president', r'vp', ] TITLE_PATTERN = re.compile( - r'([A-Z][a-z]+(?:\s+[A-Z][a-z]+){1,2})\s*(?:[-–,|:]|\bis\b|\bas\b)\s*(' + '|'.join(EXEC_TITLES) + r')\b', + # The separator class now includes the em-dash (U+2014), en-dash + # (U+2013) and bullet (U+2022) - all three are used by Indian SMB + # sites and all three were silently failing to split name from title. + r'([A-Z][a-zA-Z]+(?:\s+[A-Z][a-zA-Z]+){1,2})\s*' + r'(?:[-\u2013\u2014\u2022|,;:()]|\bis\b|\bas\b|&)\s*(' + + '|'.join(EXEC_TITLES) + r')\b', re.IGNORECASE ) @@ -295,8 +404,9 @@ async def crawl_website_for_contacts(url: str, max_pages: int = 4) -> Dict[str, text_clean = soup.get_text(" ", strip=True) for match in TITLE_PATTERN.finditer(text_clean): found_name, found_title = match.group(1).strip(), match.group(2).strip() - if len(found_name.split()) in (2, 3) and not any(w in found_name.lower() for w in ["about", "contact", "home", "terms", "service", "privacy"]): - executives.append({"name": found_name, "title": found_title.title()}) + if not _is_plausible_person_name(found_name): + continue + executives.append({"name": found_name, "title": found_title.title()}) for link in soup.find_all('a', href=True): href = link['href'].strip() @@ -432,28 +542,47 @@ async def enrich_business_by_id(business_id: int) -> Dict[str, Any]: logger.debug(f"Social enrichment skipped for {business.name}: {social_err}") # Step 2.6: Apollo-Grade Waterfall Permutations & Catch-All Validation + #: Filled in by the permutation stage below; reused by the contact + #: stage so an address confirmed by the batch is not re-probed. + batch: Dict[str, Any] = {} + learned_pattern: Optional[str] = None + learned_confidence: float = 0.0 if business.website: try: web_for_domain = business.website if business.website.startswith("http") else f"https://{business.website}" parsed_w = urlparse(web_for_domain) w_domain = parsed_w.netloc.lower().replace("www.", "").strip() if w_domain and "." in w_domain and not any(ex in w_domain for ex in EXCLUDED_EMAIL_DOMAINS): + # Use EVERY executive we found, not just the first two. The + # pattern learner needs the full set to confirm a + # template, and a seed belonging to the third exec used to + # be discarded entirely. person_names = [] - for ex in (crawl_res.get("executives", []) or [])[:2]: + for ex in (crawl_res.get("executives", []) or []): toks = (ex.get("name", "") or "").split() if toks: person_names.append((toks[0], " ".join(toks[1:]))) seeds = [e for e in all_emails if e.rpartition("@")[2].lower() == w_domain] - permutations, _pat, _conf = generate_candidates( + permutations, learned_pattern, learned_conf = generate_candidates( w_domain, person_names=person_names or None, seeds=seeds, ) if not person_names: permutations = [q for q in permutations if q.split("@")[0] in ( "contact", "info", "sales", "office")] - is_catch_all = check_catch_all_domain(w_domain) - batch = await verify_batch( - [c for c in permutations if c not in all_emails]) + # ONE batch, ONE MX connection per domain. + # + # The old flow opened 2-5 separate MX connections per lead + # (a standalone catch-all probe, then the batch, then a + # per-address verify loop that re-probed addresses the + # batch had just confirmed) and performed 3-8 DNS lookups. + # That is both slow and a good way to get the sending IP + # blocklisted by port-25 rate limiting. + to_verify = [c for c in permutations if c not in all_emails] + batch = await verify_batch(to_verify) if to_verify else {} + # The batch already probes the domain for catch-all, so a + # separate probe would be a third connection to the same MX. + is_catch_all = await check_catch_all_domain_async(w_domain) found_perms = 0 for cand in permutations: @@ -490,6 +619,9 @@ async def enrich_business_by_id(business_id: int) -> Dict[str, Any]: # Step 3: Process and verify each discovered email verified_count = 0 + # Addresses we already verified as part of the permutation batch - do + # not open a second connection to the same MX to confirm them again. + pre_verified: Dict[str, Any] = dict(batch) for email in all_emails: stmt = select(Contact).where(Contact.business_id == business.id, Contact.email == email) res = await db.execute(stmt) @@ -498,9 +630,25 @@ async def enrich_business_by_id(business_id: int) -> Dict[str, Any]: # Use the rich result, not the (bool, str) shim, so sub-status, # confidence, MX host and role/free flags are persisted. The shim # collapses 20 states into one bool and loses the explanation. - result = verify_email(email) - is_valid = result.is_deliverable + # + # The async variant is mandatory here: the sync version performs a + # DNS lookup, a TCP connect and up to three RCPT probes with an + # 8-second timeout, all *inside* an async def - which freezes the + # entire event loop (and therefore the API and the reply poller) + # for the duration of every send. + result = pre_verified.get(email) + if result is None: + result = await verify_email_async(email) + pre_verified[email] = result + v_status = result.status.value + #: "Verified" means the SMTP handshake positively confirmed the + #: mailbox. ``is_deliverable`` also covers role accounts and free + #: providers, which are classified with no network call at all - + #: so ``info@`` used to be stamped "verified" and awarded the full + #: 30-point "Verified Email" score without ever being probed. + is_valid = (v_status == "valid") + is_deliverable = result.is_deliverable if is_valid: verified_count += 1 @@ -508,15 +656,39 @@ async def enrich_business_by_id(business_id: int) -> Dict[str, Any]: business.phone or (list(all_phones)[0] if all_phones else None) ) execs = crawl_res.get("executives", []) if business.website else [] - exec_match = execs[0] if execs else None - c_first = exec_match["name"].split()[0] if exec_match else (business.name.split()[0] if business.name else "Owner") - c_last = " ".join(exec_match["name"].split()[1:]) if exec_match and len(exec_match["name"].split()) > 1 else "" - c_title = exec_match["title"] if exec_match else "Owner / Decision Maker" + + # Bind an executive to THIS address only when the address plausibly + # belongs to them. + # + # The old code took ``execs[0]`` for *every* address, so + # ``info@acme.in`` was stored with ``first_name="Rajesh"`` and + # ``title="Founder"`` - a contact that was simultaneously a role + # mailbox and a named person. The sender then wrote "Hi Rajesh" to + # a shared mailbox. And with no executive found, the fallback was + # the *business* name, producing "Hi Apex," for Apex Realty + # Solutions. + exec_match = _exec_for_address(email, execs) + c_first = exec_match["name"].split()[0] if exec_match else None + c_last = (" ".join(exec_match["name"].split()[1:]) + if exec_match and len(exec_match["name"].split()) > 1 else None) + c_title = exec_match["title"] if exec_match else None # Person vs role: a discovered executive name means this address # belongs to a human. A shared mailbox (info@) is still a lead but # must not be counted as a person found. is_role = result.is_role or (exec_match is None and detect_role_account(email.split("@")[0])) + if not is_role and exec_match is None and detect_role_account(email.split("@")[0]): + is_role = True + + #: Where this address came from, so guess-vs-harvest hit rate can + #: actually be measured. + address_source = _address_source(email, all_emails, permutations, w_domain) + contact_extra: Dict[str, Any] = {} + if learned_pattern: + contact_extra["pattern"] = learned_pattern + contact_extra["pattern_confidence"] = learned_confidence + if address_source == "pattern_guess": + contact_extra["is_guess"] = True if not existing_contact: new_contact = Contact( @@ -542,7 +714,8 @@ async def enrich_business_by_id(business_id: int) -> Dict[str, Any]: preferred_channel=in_phone.preferred_channel( business.phone or (list(all_phones)[0] if all_phones else None) ), - source="multi_source_enricher", + source=address_source, + extra_data=contact_extra or None, verified_at=datetime.now(timezone.utc) if is_valid else None, ) db.add(new_contact) @@ -556,7 +729,7 @@ async def enrich_business_by_id(business_id: int) -> Dict[str, Any]: existing_contact.is_role_account = is_role existing_contact.is_free_email = result.is_free existing_contact.verified_at = datetime.now(timezone.utc) if is_valid else None - if exec_match and existing_contact.first_name == business.name.split()[0]: + if exec_match: existing_contact.first_name = c_first existing_contact.last_name = c_last existing_contact.title = c_title diff --git a/backend/app/enricher/pattern.py b/backend/app/enricher/pattern.py index d66b1a2..6944ba4 100644 --- a/backend/app/enricher/pattern.py +++ b/backend/app/enricher/pattern.py @@ -15,6 +15,23 @@ no seeds -> global prior ranking (most common first) person names -> candidates in pattern order company name -> never + +What this module gets right (and used to get wrong) +--------------------------------------------------- +* **Reverse-engineering, not shape-guessing.** "rsharma" is ``flast`` and + "rajeshsharma" is ``firstlast``. A shape heuristic cannot tell them apart, + and the old code classified *every* single-token seed as ``{first}`` at + 0.95 confidence - confidently wrong, which is worse than abstaining. +* **Evidence is unioned, not exclusive.** The name-based matcher and the + shape-based matcher used to be ``if/else`` branches, so a seed belonging to + the third exec on a team page (or a name the title regex mangled) taught + us nothing at all. +* **Dominance-aware confidence.** One seed is suggestive, not conclusive. + A pattern only earns high confidence when it *dominates* the seed set and + there are enough seeds to dominate with. +* **Only valid addresses.** Half the old candidate budget was spent on + syntactically impossible local parts ("priya.@acme.in", "rajesh.kumar + sharma@acme.in") that burned real SMTP probes. """ import logging @@ -37,12 +54,26 @@ ("last.first", "{last}.{first}"), ("firstl", "{first}{l}"), ("last", "{last}"), + ("f_last", "{f}_{last}"), + ("f-last", "{f}-{last}"), + ("first.l", "{first}.{l}"), ) +_PRIOR_BY_NAME = {name: template for name, template in PATTERN_PRIORS} + #: Weight of the global prior vs seed evidence when blending confidence. #: One seed is suggestive, not conclusive; three agreeing seeds dominate. PRIOR_WEIGHT = 2.0 +#: A pattern needs at least this many seeds before it is called "learned". +MIN_SEEDS_FOR_CONFIDENCE = 2 + +#: A pattern must dominate the seed set by at least this much to be trusted. +MIN_DOMINANCE = 0.6 + +#: Weight of a weak vote (a seed that could only be one of several patterns). +WEAK_VOTE = 0.3 + #: Local parts that are shared mailboxes, not evidence of a person pattern. _GENERIC_LOCALS = frozenset({ "info", "contact", "contacts", "sales", "support", "hello", "admin", @@ -61,12 +92,26 @@ "hospital", "school", "college", "india", "bharat", }) +#: A syntactically valid local part. Anything else is not an address. +_LOCAL_RE = re.compile(r"^[A-Za-z0-9._+-]+$") + def _split_local(local: str) -> List[str]: """Splits a mailbox local part on the separators companies actually use.""" return [t for t in re.split(r"[._+\-]", local.strip().lower()) if t] +def _is_valid_local(local: str) -> bool: + """Whether a rendered local part could actually be an email address.""" + if not local or not _LOCAL_RE.match(local): + return False + # Leading/trailing separators are illegal in a local part and would be + # rejected by the receiving MTA. + if local[0] in ".-_" or local[-1] in ".-_": + return False + return True + + def match_pattern(local: str, first: str, last: str) -> Optional[str]: """Returns the pattern name if ``local`` fits ``(first, last)``, else None.""" local = local.strip().lower() @@ -90,6 +135,9 @@ def match_pattern(local: str, first: str, last: str) -> Optional[str]: "last.first": f"{last}.{first}", "firstl": f"{first}{l}", "last": last, + "f_last": f"{f}_{last}", + "f-last": f"{f}-{last}", + "first.l": f"{first}.{l}", } for name, rendered in candidates.items(): if local == rendered: @@ -97,12 +145,52 @@ def match_pattern(local: str, first: str, last: str) -> Optional[str]: return None +def reverse_template(local: str) -> Optional[str]: + """Infers a template from a local part alone (no name needed). + + Reverse-engineering beats shape heuristics. "rsharma" is ``flast`` and + "rajeshsharma" is ``firstlast`` - but a shape heuristic cannot tell those + apart, so the old code labelled all of them ``{first}`` at 0.95 + confidence. + + A single-token local part is genuinely ambiguous between ``{first}``, + ``{first}{last}`` and ``{last}``, so this returns ``None`` rather than + guessing. The name-based matcher resolves it when the person's name is + known; otherwise the caller records a *weak* vote at most. + """ + if not local: + return None + local = local.strip().lower() + toks = _split_local(local) + if len(toks) == 1: + tok = toks[0] + if not tok.isalpha() or len(tok) < 3: + return None + return None # ambiguous - abstain rather than guess + if len(toks) != 2 or not all(t.isalpha() for t in toks): + return None + a, b = toks + sep = "." if "." in local else ("_" if "_" in local else ("-" if "-" in local else "")) + if len(a) == 1: + return {".": "f.last", "": "flast", "_": "f_last", "-": "f-last"}.get(sep) + if len(b) == 1: + return {"": "firstl", ".": "first.l"}.get(sep) + if not sep: + return "firstlast" + return {".": "first.last", "_": "first_last", "-": "first-lastr"}.get(sep) + + def infer_pattern( seeds: List[str], domain: str, person_names: Optional[List[Tuple[str, str]]] = None, ) -> Tuple[str, float]: - """Infers a domain's mailbox pattern from harvested seed addresses.""" + """Infers a domain's mailbox pattern from harvested seed addresses. + + ``person_names`` is every (first, last) pair known for the domain. It is + used to *confirm* a template, not to gate it: a seed that matches no + known name still contributes via :func:`reverse_template`. + """ domain = (domain or "").lower().strip() own = [ s.strip().lower() @@ -113,11 +201,15 @@ def infer_pattern( priors = {name: 1.0 - (i * 0.06) for i, (name, _) in enumerate(PATTERN_PRIORS)} votes: Dict[str, float] = {name: 0.0 for name, _ in PATTERN_PRIORS} - total = 0 + total = 0.0 + seed_count = 0 for seed in own: local = seed.rpartition("@")[0] matched: Optional[str] = None + weight = 1.0 + + # 1) Name-based confirmation - the strongest evidence. if person_names: hits = set() for first, last in person_names: @@ -126,33 +218,62 @@ def infer_pattern( hits.add(hit) if len(hits) == 1: matched = next(iter(hits)) - else: - parts = _split_local(local) - if len(parts) == 2 and all(p.isalpha() and len(p) >= 2 for p in parts): - sep = "." if "." in local else ("_" if "_" in local else ("-" if "-" in local else "")) - if len(parts[0]) == 1: - matched = "f.last" if sep == "." else ("flast" if not sep else None) - elif len(parts[1]) == 1: - matched = "firstl" if not sep else None - else: - matched = {"": None, ".": "first.last", "_": "first_last", "-": "first-lastr"}.get(sep) - elif len(parts) == 1 and parts[0].isalpha() and len(parts[0]) >= 2: + elif len(hits) > 1: + # Ambiguous between templates for this name - do not vote. + matched = None + + # 2) Shape-based reverse engineering - used when the name did not + # resolve. Previously this branch only ran when there were NO + # person names at all, so every seed was discarded whenever the + # exec list was incomplete. + if matched is None: + matched = reverse_template(local) + + # 3) Last resort: a single alpha token might be a first name. + if matched is None: + toks = _split_local(local) + if len(toks) == 1 and toks[0].isalpha() and len(toks[0]) >= 3: matched = "first" + weight = WEAK_VOTE + if matched: - votes[matched] += 1.0 - total += 1 + votes[matched] = votes.get(matched, 0.0) + weight + total += weight + seed_count += 1 - if total == 0: + if total <= 0: + # No usable evidence: fall back to the global prior, and be honest + # about how little we know. return PATTERN_PRIORS[0][0], 0.25 - best = max(votes, key=lambda k: votes[k] + priors[k] * 0.15) - confidence = (PRIOR_WEIGHT * priors[best] + votes[best]) / (PRIOR_WEIGHT + total) - return best, round(min(0.95, confidence), 2) + best = max(votes, key=lambda k: votes[k] + priors.get(k, 0.5) * 0.15) + ratio = votes[best] / total + seed_factor = min(1.0, seed_count / 3.0) + prior = priors.get(best, 0.5) + + confidence = 0.50 * ratio + 0.30 * seed_factor + 0.20 * prior + # One seed is never enough to be confident - a single ambiguous seed must + # not override what most domains actually do. + if seed_count < MIN_SEEDS_FOR_CONFIDENCE or ratio < MIN_DOMINANCE: + confidence = min(confidence, 0.5) + + return best, round(max(0.0, min(0.99, confidence)), 2) def _render(template: str, first: str, last: str) -> str: - first = first.strip().lower() + """Renders a template into a local part. + + Multi-token names are reduced to their meaningful parts: the surname is + the *last* token ("Kumar Sharma" -> "Sharma") and the given name is the + first token. Rendering the full multi-word string produced local parts + containing spaces, which are not addresses at all. + """ + first = (first or "").strip().lower() last = (last or "").strip().lower() + if " " in first: + first = first.split()[0] + if " " in last: + last = last.split()[-1] return template.format(first=first, last=last, f=first[:1], l=last[:1]) @@ -162,7 +283,13 @@ def generate_candidates( seeds: Optional[List[str]] = None, max_person_candidates: int = 4, ) -> Tuple[List[str], str, float]: - """Ordered candidates for one domain: persons first, aliases last.""" + """Ordered candidates for one domain: persons first, aliases last. + + The person budget is spread across the people we know rather than spent + entirely on the first one. Previously the first name consumed all four + slots and the second person - whom we had also learned the pattern from - + never got a single candidate. + """ domain = (domain or "").lower().replace("www.", "").strip() if not domain or "." not in domain: return [], "first.last", 0.0 @@ -170,23 +297,41 @@ def generate_candidates( pattern, confidence = infer_pattern(list(seeds or []), domain, person_names) ordered_templates = [t for _, t in PATTERN_PRIORS] - if pattern in dict(PATTERN_PRIORS): - winner_tpl = dict(PATTERN_PRIORS)[pattern] + if pattern in _PRIOR_BY_NAME: + winner_tpl = _PRIOR_BY_NAME[pattern] ordered_templates.remove(winner_tpl) ordered_templates.insert(0, winner_tpl) candidates: List[str] = [] + names = list(person_names or [])[:3] + + # Templates that need a surname are useless for a single-name contact. + def _usable(template: str, first: str, last: str) -> bool: + if "{last}" in template and not last: + return False + if "{l}" in template and not last: + return False + return True + + per_name = (max(1, max_person_candidates // len(names))) if names else 0 person_count = 0 - for first, last in (person_names or [])[:2]: + + for first, last in names: + made = 0 for tpl in ordered_templates: + if not _usable(tpl, first, last): + continue local = _render(tpl, first, last or "") - if local.strip(".-_"): - addr = f"{local}@{domain}" - if addr not in candidates: - candidates.append(addr) - person_count += 1 - if person_count >= max_person_candidates: + if not _is_valid_local(local): + continue + addr = f"{local}@{domain}" + if addr in candidates: + continue + candidates.append(addr) + made += 1 + if made >= per_name or person_count + made >= max_person_candidates: break + person_count += made if person_count >= max_person_candidates: break diff --git a/backend/app/main.py b/backend/app/main.py index 6491132..3af3f6c 100644 --- a/backend/app/main.py +++ b/backend/app/main.py @@ -1,4 +1,4 @@ -from fastapi import FastAPI +from fastapi import FastAPI, Depends from fastapi.middleware.cors import CORSMiddleware from sqlalchemy import text import asyncio @@ -9,8 +9,9 @@ from typing import Optional, Any, Dict, List from app.api import scraper, leads, enricher, campaigns, analytics, mailboxes, icp, inbox, lists, suppression, setup from app.campaigns import tracker -from app.database import Base, engine +from app.database import Base, engine, async_session_maker, ensure_schema from app.seeds import seed_initial_data_if_empty +from app.auth import require_api_key, auth_enabled logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(name)s: %(message)s") logger = logging.getLogger("leadforge") @@ -21,48 +22,18 @@ #: USE_CELERY is true so sends are never double-counted. REPLY_POLL_INTERVAL_SECONDS = 300 +#: How often the in-process scheduler advances active campaign sequences. +#: Without this, a multi-step sequence only ever advanced when a human +#: manually re-clicked "Launch" — the step delay arithmetic existed but +#: nothing ever drove it, so the README's headline sequence feature was +#: manual-only. +CAMPAIGN_TICK_SECONDS = 300 -def _sync_missing_columns(sync_conn) -> None: - """Additive migration: ALTER TABLE ADD COLUMN for any model column the live - database lacks. - - Replaces a hand-maintained SQL list that drifted every time a column was - added (and once silently no-op'd for an entire session because `text` was - not imported). Deriving from the models means a new column needs no - migration bookkeeping, which is what self-hosted users actually need. - - Only adds columns - never drops or retypes - so it is safe to run on every - boot. Missing tables are left to ``create_all``. - """ - from sqlalchemy import inspect as sa_inspect - - inspector = sa_inspect(sync_conn) - existing_tables = set(inspector.get_table_names()) - dialect = sync_conn.dialect - - for table in Base.metadata.sorted_tables: - if table.name not in existing_tables: - continue - have = {c["name"] for c in inspector.get_columns(table.name)} - for col in table.columns: - if col.name in have: - continue - ddl = f'ALTER TABLE "{table.name}" ADD COLUMN "{col.name}" {col.type.compile(dialect)}' - default = getattr(col.default, "arg", None) - if default is not None and not callable(default): - ddl += f" DEFAULT {default!r}" if isinstance(default, str) else f" DEFAULT {default}" - try: - sync_conn.execute(text(ddl)) - logger.info("schema sync: added %s.%s", table.name, col.name) - except Exception as exc: # pragma: no cover - dialect-specific limits - logger.warning("schema sync skipped %s.%s: %s", table.name, col.name, exc) @contextlib.asynccontextmanager async def lifespan(app: FastAPI): logger.info("Initializing LeadForge database schemas...") - async with engine.begin() as conn: - await conn.run_sync(Base.metadata.create_all) - await conn.run_sync(_sync_missing_columns) + await ensure_schema() # Auto-seed initial realistic demo data if database is fresh try: @@ -70,14 +41,23 @@ async def lifespan(app: FastAPI): except Exception as e: logger.warning(f"Seed initialization notice: {e}") - poller: Optional[asyncio.Task] = None + if auth_enabled(): + logger.info("API authentication is ENABLED (API_KEY set).") + + background: list[asyncio.Task] = [] if not settings.USE_CELERY: - poller = asyncio.ensure_future(_reply_poll_loop()) + background.append(asyncio.ensure_future(_reply_poll_loop())) + background.append(asyncio.ensure_future(_campaign_run_loop())) yield - if poller is not None: - poller.cancel() + for task in background: + task.cancel() + for task in background: + try: + await task + except (asyncio.CancelledError, Exception): # noqa: B014 - shutdown is best-effort + pass logger.info("LeadForge shutting down.") @@ -102,6 +82,71 @@ async def _reply_poll_loop() -> None: logger.warning("reply poll loop error: %s", exc) await asyncio.sleep(REPLY_POLL_INTERVAL_SECONDS) + +async def _campaign_run_loop() -> None: + """Advances every ACTIVE campaign sequence on a fixed cadence. + + A multi-step sequence is a time-based machine: step 2 is due N days after + step 1 was delivered. The delay arithmetic existed in + ``campaigns.sender`` but nothing ever called it again, so the operator had + to manually re-launch every sequence at roughly the right moment - forever. + + This loop is the thing that makes a sequence a sequence. It is deliberately + conservative: one campaign failing never stops the others, and a campaign + that is merely waiting on a step delay stays ACTIVE (see + ``execute_campaign_step``) rather than being marked complete. + """ + from sqlalchemy import select + from app.models.campaign import Campaign, CampaignStatus + from app.campaigns.sender import execute_campaign_step + + await asyncio.sleep(90) # let boot finish before the first tick + while True: + try: + async with async_session_maker() as db: + campaign_ids = (await db.execute( + select(Campaign.id).where( + Campaign.status == CampaignStatus.ACTIVE.value) + )).scalars().all() + for campaign_id in campaign_ids: + try: + result = await execute_campaign_step(campaign_id) + if result.get("sent_count"): + logger.info( + "campaign %s: sent %s", campaign_id, result.get("sent_count")) + except asyncio.CancelledError: + raise + except Exception as exc: + logger.warning("campaign %s tick error: %s", campaign_id, exc) + except asyncio.CancelledError: + raise + except Exception as exc: # pragma: no cover - defensive + logger.warning("campaign loop error: %s", exc) + await asyncio.sleep(CAMPAIGN_TICK_SECONDS) + + +async def _reap_stuck_sends() -> int: + """Marks abandoned in-flight sends as failed. + + A send row is committed as "sending" *before* the SMTP call so a crash + cannot produce a delivered email with no record. If the process dies + mid-flight the row stays "sending" forever; this sweeps those up so the + next run can retry them against the normal retry budget. + """ + from datetime import datetime, timedelta, timezone + from sqlalchemy import select, update + from app.models.send import Send + + cutoff = datetime.now(timezone.utc) - timedelta(hours=1) + async with async_session_maker() as db: + result = await db.execute( + update(Send) + .where(Send.status == "sending") + .values(status="failed", error_message="interrupted before completion") + ) + await db.commit() + return result.rowcount or 0 + app = FastAPI( title=settings.PROJECT_NAME, description="Autonomous lead generation, contact enrichment, and cold email outreach engine.", @@ -114,45 +159,71 @@ async def _reply_poll_loop() -> None: # production deployment names its dashboard explicitly instead of trusting # every website on the internet with the user's session. _cors_origins = [o.strip() for o in settings.CORS_ORIGINS.split(",") if o.strip()] +#: ``allow_credentials=True`` alongside a wildcard origin makes Starlette echo +#: the *attacker's* origin back on the preflight response, which turns +#: "allow all origins" into "allow any website to drive this API with the +#: user's session". Credentials are only allowed for an explicit origin list. +_allow_credentials = "*" not in _cors_origins app.add_middleware( CORSMiddleware, allow_origins=_cors_origins, - allow_credentials=True, + allow_credentials=_allow_credentials, allow_methods=["*"], allow_headers=["*"], ) +#: Every router except the mail-facing tracker requires the API key when one +#: is configured. The tracker serves the RFC 8058 one-click unsubscribe and +#: the open-tracking pixel, which are fetched by the recipient's mail client. +_auth_dependency = [Depends(require_api_key)] + # Versioned API routes (/api/v1/...) -app.include_router(analytics.router, prefix=f"{settings.API_V1_STR}/analytics", tags=["Analytics"]) -app.include_router(scraper.router, prefix=f"{settings.API_V1_STR}/scraper", tags=["Scraper"]) -app.include_router(leads.router, prefix=f"{settings.API_V1_STR}/leads", tags=["Leads"]) -app.include_router(enricher.router, prefix=f"{settings.API_V1_STR}/enricher", tags=["Enrichment"]) -app.include_router(campaigns.router, prefix=f"{settings.API_V1_STR}/campaigns", tags=["Campaigns"]) +app.include_router(analytics.router, prefix=f"{settings.API_V1_STR}/analytics", tags=["Analytics"], dependencies=_auth_dependency) +app.include_router(scraper.router, prefix=f"{settings.API_V1_STR}/scraper", tags=["Scraper"], dependencies=_auth_dependency) +app.include_router(leads.router, prefix=f"{settings.API_V1_STR}/leads", tags=["Leads"], dependencies=_auth_dependency) +app.include_router(enricher.router, prefix=f"{settings.API_V1_STR}/enricher", tags=["Enrichment"], dependencies=_auth_dependency) +app.include_router(campaigns.router, prefix=f"{settings.API_V1_STR}/campaigns", tags=["Campaigns"], dependencies=_auth_dependency) app.include_router(tracker.router, prefix=f"{settings.API_V1_STR}/campaigns/tracker", tags=["Campaign Tracker"]) -app.include_router(mailboxes.router, prefix=f"{settings.API_V1_STR}/mailboxes", tags=["Mailboxes"]) -app.include_router(icp.router, prefix=f"{settings.API_V1_STR}/icp", tags=["ICP & Discovery"]) -app.include_router(inbox.router, prefix=f"{settings.API_V1_STR}/inbox", tags=["Inbox"]) -app.include_router(lists.router, prefix=f"{settings.API_V1_STR}/lists", tags=["Saved Lists"]) -app.include_router(suppression.router, prefix=f"{settings.API_V1_STR}/suppression", tags=["Suppression"]) -app.include_router(setup.router, prefix=f"{settings.API_V1_STR}/setup", tags=["Setup"]) +app.include_router(mailboxes.router, prefix=f"{settings.API_V1_STR}/mailboxes", tags=["Mailboxes"], dependencies=_auth_dependency) +app.include_router(icp.router, prefix=f"{settings.API_V1_STR}/icp", tags=["ICP & Discovery"], dependencies=_auth_dependency) +app.include_router(inbox.router, prefix=f"{settings.API_V1_STR}/inbox", tags=["Inbox"], dependencies=_auth_dependency) +app.include_router(lists.router, prefix=f"{settings.API_V1_STR}/lists", tags=["Saved Lists"], dependencies=_auth_dependency) +app.include_router(suppression.router, prefix=f"{settings.API_V1_STR}/suppression", tags=["Suppression"], dependencies=_auth_dependency) +app.include_router(setup.router, prefix=f"{settings.API_V1_STR}/setup", tags=["Setup"], dependencies=_auth_dependency) # Aliases without /v1 for backwards compatibility (/api/...) -app.include_router(analytics.router, prefix="/api/analytics", tags=["Analytics Legacy"]) -app.include_router(scraper.router, prefix="/api/scraper", tags=["Scraper Legacy"]) -app.include_router(leads.router, prefix="/api/leads", tags=["Leads Legacy"]) -app.include_router(enricher.router, prefix="/api/enricher", tags=["Enrichment Legacy"]) -app.include_router(campaigns.router, prefix="/api/campaigns", tags=["Campaigns Legacy"]) -app.include_router(mailboxes.router, prefix="/api/mailboxes", tags=["Mailboxes Legacy"]) -app.include_router(icp.router, prefix="/api/icp", tags=["ICP & Discovery Legacy"]) -app.include_router(inbox.router, prefix="/api/inbox", tags=["Inbox Legacy"]) -app.include_router(lists.router, prefix="/api/lists", tags=["Saved Lists Legacy"]) -app.include_router(suppression.router, prefix="/api/suppression", tags=["Suppression Legacy"]) -app.include_router(setup.router, prefix="/api/setup", tags=["Setup Legacy"]) +app.include_router(analytics.router, prefix="/api/analytics", tags=["Analytics Legacy"], dependencies=_auth_dependency) +app.include_router(scraper.router, prefix="/api/scraper", tags=["Scraper Legacy"], dependencies=_auth_dependency) +app.include_router(leads.router, prefix="/api/leads", tags=["Leads Legacy"], dependencies=_auth_dependency) +app.include_router(enricher.router, prefix="/api/enricher", tags=["Enrichment Legacy"], dependencies=_auth_dependency) +app.include_router(campaigns.router, prefix="/api/campaigns", tags=["Campaigns Legacy"], dependencies=_auth_dependency) +app.include_router(mailboxes.router, prefix="/api/mailboxes", tags=["Mailboxes Legacy"], dependencies=_auth_dependency) +app.include_router(icp.router, prefix="/api/icp", tags=["ICP & Discovery Legacy"], dependencies=_auth_dependency) +app.include_router(inbox.router, prefix="/api/inbox", tags=["Inbox Legacy"], dependencies=_auth_dependency) +app.include_router(lists.router, prefix="/api/lists", tags=["Saved Lists Legacy"], dependencies=_auth_dependency) +app.include_router(suppression.router, prefix="/api/suppression", tags=["Suppression Legacy"], dependencies=_auth_dependency) +app.include_router(setup.router, prefix="/api/setup", tags=["Setup Legacy"], dependencies=_auth_dependency) @app.get("/health") async def health_check(): + """Liveness + database connectivity. + + The old version reported the database *scheme* and never touched the + connection, so a healthcheck passed against a database that was gone. + """ + db_ok = True + db_error = None + try: + async with async_session_maker() as db: + await db.execute(text("SELECT 1")) + except Exception as exc: # pragma: no cover - depends on the environment + db_ok = False + db_error = str(exc) return { - "status": "ok", + "status": "ok" if db_ok else "degraded", "service": "LeadForge Autonomous Outreach Engine", - "database": settings.DATABASE_URL.split("://")[0] + "database": settings.DATABASE_URL.split("://")[0], + "database_reachable": db_ok, + "auth_enabled": auth_enabled(), + **({"database_error": db_error} if db_error else {}), } diff --git a/backend/app/mcp_server.py b/backend/app/mcp_server.py index 3e8ba63..8aca37e 100755 --- a/backend/app/mcp_server.py +++ b/backend/app/mcp_server.py @@ -20,7 +20,7 @@ sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))) from sqlalchemy import select, or_, func -from app.database import async_session_maker +from app.database import async_session_maker, ensure_schema from app.models.business import Business from app.models.contact import Contact from app.models.scrape_job import ScrapeJob @@ -202,12 +202,20 @@ }, { "name": "leadforge_launch_campaign", - "description": "Activates a campaign and dispatches its sequence in the background. Refuses to launch with 0 targets or 0 steps to avoid emailing the whole database. Honors suppression, warmup/health, daily limits, delays, and send jitter.", + "description": "Activates a campaign and dispatches its sequence in the background. Refuses to launch with 0 targets or 0 steps to avoid emailing the whole database. Honors suppression, warmup/health, daily limits, delays, and send jitter. Performs a DRY RUN by default - it returns the exact recipient list and subject lines without sending anything. Set dry_run=false only when the operator has explicitly approved the send.", "inputSchema": { "type": "object", - "properties": {"campaign_id": {"type": "integer", "description": "Campaign ID"}}, + "properties": { + "campaign_id": {"type": "integer", "description": "Campaign ID"}, + "dry_run": { + "type": "boolean", + "description": "When true (the default), preview the recipients and subject lines WITHOUT sending. Set to false to actually send.", + "default": True, + }, + }, "required": ["campaign_id"] - } + }, + "annotations": {"readOnlyHint": False, "destructiveHint": True, "idempotentHint": False} }, { "name": "leadforge_pause_campaign", @@ -294,11 +302,20 @@ async def handle_tool_call(name: str, arguments: Dict[str, Any]) -> str: job_id = job.id # Run scrape - saved_count, _new_biz_ids = await scrape_leads_and_save(query=query, max_results=max_results, job_id=job_id) - - # Fetch newly scraped leads + saved_count, new_biz_ids = await scrape_leads_and_save(query=query, max_results=max_results, job_id=job_id) + + # Fetch the leads THIS scrape created. + # + # The previous code returned the last N businesses in the whole + # database while the header said "Scrape Results for: " - so an + # agent that scraped "Dentists in Pune" against a database that + # already held 75 real-estate leads got a table of real-estate agents + # presented as Pune dentists. async with async_session_maker() as db: - stmt = select(Business).order_by(Business.id.desc()).limit(max_results) + if new_biz_ids: + stmt = select(Business).where(Business.id.in_(new_biz_ids)).order_by(Business.id) + else: + stmt = select(Business).where(Business.id.is_(None)) res = await db.execute(stmt) businesses = res.scalars().all() @@ -319,6 +336,16 @@ async def handle_tool_call(name: str, arguments: Dict[str, Any]) -> str: results.append({ "id": b.id, + # Contact ids, so the agent can actually TARGET these + # leads. `add_campaign_targets` needs contact_ids, and + # contact.id != business.id for most rows - an agent that + # assumed they were the same mailed the wrong 79% of the + # database. + "contact_ids": [c.id for c in contacts], + "primary_contact_id": next( + (c.id for c in contacts if c.email), + contacts[0].id if contacts else None, + ), "name": b.name, "maps_url": maps_url, "phone": b.phone or "—", @@ -334,15 +361,25 @@ async def handle_tool_call(name: str, arguments: Dict[str, Any]) -> str: f"### LeadForge Google Maps Scrape Results for: \"{query}\"\n", f"• **Leads Extracted**: {len(results)}", f"• **New Leads Saved**: {saved_count}\n", - "| Business Name | Google Maps | Phone | WhatsApp | Website | Verified Emails | Rating |", - "| :--- | :--- | :--- | :--- | :--- | :--- | :--- |" + "| Business ID | Contact ID | Business Name | Google Maps | Phone | WhatsApp | Website | Verified Emails | Rating |", + "| :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- |" ] for r in results: name_clean = r["name"].replace("|", "\\|") maps_link = f"[View Listing]({r['maps_url']})" web_link = f"[Website]({r['website']})" if r['website'] != "—" else "—" - md.append(f"| **{name_clean}** | {maps_link} | {r['phone']} | {r['whatsapp']} | {web_link} | {r['emails']} | {r['rating']} |") + contact_id = r["primary_contact_id"] if r["primary_contact_id"] else "—" + md.append( + f"| {r['id']} | {contact_id} | **{name_clean}** | {maps_link} | {r['phone']} " + f"| {r['whatsapp']} | {web_link} | {r['emails']} | {r['rating']} |" + ) + + md.append( + "\n> Use the **Contact ID** column with `leadforge_add_campaign_targets` " + "to enroll these leads in a sequence. The Contact ID is NOT the " + "Business ID." + ) return "\n".join(md) @@ -360,6 +397,20 @@ async def handle_tool_call(name: str, arguments: Dict[str, Any]) -> str: if industry: stmt = stmt.where(Business.industry.ilike(f"%{industry}%")) + # The "has email" filter is a SQL predicate applied BEFORE the + # limit. Filtering after the limit made the tool's own limit + # parameter change its answer: asking for 5 leads with + # has_email=true returned 0 matches, while asking for 100 + # returned 6 - the filter was applied to an already-truncated set. + if has_email is not None: + email_ids = select(Contact.business_id).where( + Contact.email.isnot(None), Contact.email != "" + ) + stmt = stmt.where( + Business.id.in_(email_ids) if has_email + else Business.id.notin_(email_ids) + ) + stmt = stmt.order_by(Business.id.desc()).limit(limit) res = await db.execute(stmt) businesses = res.scalars().all() @@ -375,17 +426,18 @@ async def handle_tool_call(name: str, arguments: Dict[str, Any]) -> str: for c in contacts if c.email ] - if has_email is True and not emails: - continue - if has_email is False and emails: - continue - clean_phone = (b.phone or "").replace(" ", "").replace("-", "") wa_link = f"[WhatsApp](https://wa.me/{clean_phone})" if clean_phone else "" maps_url = b.maps_url or f"https://www.google.com/maps/search/?api=1&query={urllib.parse.quote((b.name + ' ' + (b.address or '')).strip())}" results.append({ "id": b.id, + # Contact ids are what `add_campaign_targets` needs. + "contact_ids": [c.id for c in contacts], + "primary_contact_id": next( + (c.id for c in contacts if c.email), + contacts[0].id if contacts else None, + ), "name": b.name, "maps_url": maps_url, "phone": b.phone or "—", @@ -397,13 +449,23 @@ async def handle_tool_call(name: str, arguments: Dict[str, Any]) -> str: md = [ f"### LeadForge Database Search ({len(results)} matches)\n", - "| ID | Business Name | Google Maps | Phone | WhatsApp | Category | Emails | Rating |", - "| :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- |" + "| Business ID | Contact ID | Business Name | Google Maps | Phone | WhatsApp | Category | Emails | Rating |", + "| :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- |" ] for r in results: name_clean = r["name"].replace("|", "\\|") maps_link = f"[Maps]({r['maps_url']})" - md.append(f"| #{r['id']} | **{name_clean}** | {maps_link} | {r['phone']} | {r['whatsapp']} | {r['industry']} | {r['emails']} | {r['rating']} |") + contact_id = r["primary_contact_id"] if r["primary_contact_id"] else "—" + md.append( + f"| #{r['id']} | {contact_id} | **{name_clean}** | {maps_link} | {r['phone']} " + f"| {r['whatsapp']} | {r['industry']} | {r['emails']} | {r['rating']} |" + ) + + md.append( + "\n> Use the **Contact ID** column with `leadforge_add_campaign_targets` " + "to enroll these leads in a sequence. The Contact ID is NOT the " + "Business ID." + ) return "\n".join(md) @@ -638,6 +700,7 @@ async def handle_tool_call(name: str, arguments: Dict[str, Any]) -> str: added = 0 skipped = 0 + missing = 0 for cid in dict.fromkeys(contact_ids): if cid in existing: continue @@ -646,6 +709,10 @@ async def handle_tool_call(name: str, arguments: Dict[str, Any]) -> str: select(Contact).where(Contact.id == cid) )).scalars().first() if contact is None: + # A non-existent contact id used to be silently dropped + # and NOT counted, so the tool reported "Added 0 target(s) + # (0 skipped)" and the agent concluded success. + missing += 1 continue if await is_suppressed(db, contact.email): skipped += 1 @@ -658,11 +725,25 @@ async def handle_tool_call(name: str, arguments: Dict[str, Any]) -> str: select(func.count(CampaignTarget.id)).where(CampaignTarget.campaign_id == campaign_id) )).scalar() or 0 - return (f"Added {added} target(s) to campaign #{campaign_id} " - f"({skipped} skipped: suppressed). Total targets: {total}.") + parts = [f"Added {added} target(s) to campaign #{campaign_id}"] + if skipped: + parts.append(f"({skipped} skipped: suppressed)") + if missing: + parts.append( + f"**WARNING: {missing} contact id(s) did not exist and were skipped** - " + "check that you are using Contact IDs (from the Contact ID column) " + "and not Business IDs." + ) + parts.append(f"Total targets: {total}.") + return " ".join(parts) elif name == "leadforge_launch_campaign": campaign_id = int(arguments["campaign_id"]) + # Dry run by default. An autonomous agent firing real cold email at + # real people without a human in the loop is the single most + # dangerous thing this server can do, and the tool had no way to + # preview what would be sent. + dry_run = arguments.get("dry_run", True) async with async_session_maker() as db: camp = await db.get(Campaign, campaign_id) if not camp: @@ -680,6 +761,40 @@ async def handle_tool_call(name: str, arguments: Dict[str, Any]) -> str: return (f"Campaign #{campaign_id} cannot launch: no explicit targets. " "Add recipients with `leadforge_add_campaign_targets` so we never " "mail leads the operator did not pick.") + + # Preview the exact recipients and subject lines. + recipient_rows = (await db.execute( + select(Contact.email, Contact.first_name) + .join(CampaignTarget, CampaignTarget.contact_id == Contact.id) + .where(CampaignTarget.campaign_id == campaign_id) + )).all() + step_rows = (await db.execute( + select(CampaignStep.step_number, CampaignStep.subject) + .where(CampaignStep.campaign_id == campaign_id) + .order_by(CampaignStep.step_number) + )).all() + + if dry_run: + preview = [f"**DRY RUN - nothing was sent.**", + f"Campaign #{campaign_id} would send {targets} email(s) " + f"across {steps} step(s).", + "", + "**Recipients (first 10):**"] + for email, first_name in recipient_rows[:10]: + preview.append(f"- {first_name or '—'} <{email}>") + if len(recipient_rows) > 10: + preview.append(f"- ...and {len(recipient_rows) - 10} more") + preview.append("") + preview.append("**Steps:**") + for num, subject in step_rows: + preview.append(f"- Step {num}: {subject}") + preview.append("") + preview.append("Re-call with `dry_run=false` to actually send.") + return "\n".join(preview) + + if camp.status == CampaignStatus.ACTIVE.value: + return (f"Campaign #{campaign_id} is already active - not launching a " + "second sender (that would double-send to the same leads).") camp.status = CampaignStatus.ACTIVE.value camp_name = camp.name await db.commit() @@ -831,6 +946,11 @@ async def run_mcp_server(): """Runs the Model Context Protocol stdio loop.""" logger.info("Starting LeadForge MCP stdio server...") + # The MCP server talks to the database directly and must be able to + # survive a schema change on its own. It used to depend on the API + # process having started first. + await ensure_schema() + loop = asyncio.get_event_loop() reader = asyncio.StreamReader() protocol = asyncio.StreamReaderProtocol(reader) @@ -849,6 +969,14 @@ async def run_mcp_server(): req = json.loads(line_str) except Exception as parse_err: logger.error(f"JSON decode error: {parse_err}") + # JSON-RPC requires a Parse error response with a null id. + out_bytes = (json.dumps({ + "jsonrpc": "2.0", + "id": None, + "error": {"code": -32700, "message": "Parse error: invalid JSON"}, + }) + "\n").encode("utf-8") + sys.stdout.buffer.write(out_bytes) + sys.stdout.buffer.flush() continue req_id = req.get("id") @@ -901,13 +1029,27 @@ async def run_mcp_server(): } } except Exception as tool_err: - logger.error(f"Error executing tool {tool_name}: {tool_err}") + logger.exception("Error executing tool %s", tool_name) + # A tool that fails at execution time must return a RESULT + # with isError=true, not a transport-level JSON-RPC error. + # The model can read a tool result and react to it; a + # transport error is invisible to it and it typically retries + # or gives up. The raw exception text also leaked raw SQL + # into the transcript. response = { "jsonrpc": "2.0", "id": req_id, - "error": { - "code": -32603, - "message": str(tool_err) + "result": { + "content": [{ + "type": "text", + "text": ( + f"Tool '{tool_name}' failed: {type(tool_err).__name__}. " + "Check the arguments and the system state with " + "`leadforge_get_stats` / `leadforge_search_leads`, then " + "retry with corrected arguments." + ), + }], + "isError": True, } } elif method == "ping": diff --git a/backend/app/models/__init__.py b/backend/app/models/__init__.py index 0426dac..6592ca4 100644 --- a/backend/app/models/__init__.py +++ b/backend/app/models/__init__.py @@ -9,6 +9,11 @@ from .source_stats import SourceStats from .social_profile import SocialProfile from .suppression import Suppression +# Every model module must be imported here. A model that is only registered +# when some other module happens to import it is a model whose table is +# missing from `create_all` - which is how `saved_lists` ended up absent +# from freshly-created databases (and blew up the test-suite reset). +from .saved_list import SavedList __all__ = [ "Business", @@ -24,4 +29,5 @@ "SourceStats", "SocialProfile", "Suppression", + "SavedList", ] diff --git a/backend/app/models/business.py b/backend/app/models/business.py index 111b623..05c6d83 100644 --- a/backend/app/models/business.py +++ b/backend/app/models/business.py @@ -20,11 +20,15 @@ class Business(Base): maps_url = Column(String, nullable=True) extra_data = Column(JSON, nullable=True) - source = Column(String, nullable=True) + source = Column(String, nullable=True, index=True) icp_profile_id = Column(Integer, ForeignKey("icp_profiles.id"), nullable=True, index=True) icp_fit = Column(JSON, nullable=True) - created_at = Column(DateTime(timezone=True), server_default=func.now()) + #: Persisted lead-quality score (0-100). Kept in a column, not only inside + #: extra_data, so the lead list can sort/filter on it in SQL. + score = Column(Integer, nullable=True, index=True) + + created_at = Column(DateTime(timezone=True), server_default=func.now(), index=True) updated_at = Column(DateTime(timezone=True), onupdate=func.now()) contacts = relationship("Contact", back_populates="business", cascade="all, delete-orphan") diff --git a/backend/app/models/campaign_step.py b/backend/app/models/campaign_step.py index d7a4c24..bbad050 100644 --- a/backend/app/models/campaign_step.py +++ b/backend/app/models/campaign_step.py @@ -7,7 +7,7 @@ class CampaignStep(Base): __tablename__ = "campaign_steps" id = Column(Integer, primary_key=True, index=True) - campaign_id = Column(Integer, ForeignKey("campaigns.id", ondelete="CASCADE")) + campaign_id = Column(Integer, ForeignKey("campaigns.id", ondelete="CASCADE"), index=True) step_number = Column(Integer) subject = Column(String) #: Optional A/B variant of the subject line. While variant_winner is NULL, diff --git a/backend/app/models/contact.py b/backend/app/models/contact.py index 4c1cfa2..d484a50 100644 --- a/backend/app/models/contact.py +++ b/backend/app/models/contact.py @@ -7,7 +7,7 @@ class Contact(Base): __tablename__ = "contacts" id = Column(Integer, primary_key=True, index=True) - business_id = Column(Integer, ForeignKey("businesses.id", ondelete="CASCADE"), nullable=True) + business_id = Column(Integer, ForeignKey("businesses.id", ondelete="CASCADE"), nullable=True, index=True) first_name = Column(String, nullable=True) last_name = Column(String, nullable=True) title = Column(String, nullable=True) diff --git a/backend/app/models/mailbox.py b/backend/app/models/mailbox.py index 67111f2..373c613 100644 --- a/backend/app/models/mailbox.py +++ b/backend/app/models/mailbox.py @@ -15,6 +15,10 @@ class Mailbox(Base): daily_limit = Column(Integer, default=50) sent_today = Column(Integer, default=0) + #: Calendar day (YYYY-MM-DD, UTC) that ``sent_today`` refers to. Without + #: this the counter is a *lifetime* total, and the daily-cap filter that + #: reads it retires the mailbox permanently after ``daily_limit`` sends. + sent_today_date = Column(String, nullable=True) is_active = Column(Boolean, default=True) #: Inbound (IMAP) settings for reply detection. imap_host defaults to the diff --git a/backend/app/models/send.py b/backend/app/models/send.py index 37fb0a3..1abda3b 100644 --- a/backend/app/models/send.py +++ b/backend/app/models/send.py @@ -7,13 +7,20 @@ class Send(Base): __tablename__ = "sends" id = Column(Integer, primary_key=True, index=True) - campaign_id = Column(Integer, ForeignKey("campaigns.id", ondelete="CASCADE")) - step_id = Column(Integer, ForeignKey("campaign_steps.id", ondelete="CASCADE")) - contact_id = Column(Integer, ForeignKey("contacts.id", ondelete="CASCADE")) - mailbox_id = Column(Integer, ForeignKey("mailboxes.id", ondelete="SET NULL"), nullable=True) - - status = Column(String, default="pending") # pending, sent, failed + campaign_id = Column(Integer, ForeignKey("campaigns.id", ondelete="CASCADE"), index=True) + step_id = Column(Integer, ForeignKey("campaign_steps.id", ondelete="CASCADE"), index=True) + contact_id = Column(Integer, ForeignKey("contacts.id", ondelete="CASCADE"), index=True) + mailbox_id = Column(Integer, ForeignKey("mailboxes.id", ondelete="SET NULL"), nullable=True, index=True) + + #: pending -> sending -> sent | failed | bounced. "sending" is committed + #: *before* the SMTP call so a crash mid-flight leaves an auditable row + #: instead of a delivered email with no record of it. + status = Column(String, default="pending") # pending, sending, sent, failed, bounced error_message = Column(String, nullable=True) + #: How many times this (campaign, contact, step) has been attempted. A soft + #: bounce is only suppressed once the retry budget is exhausted - a single + #: transient gateway blip must not delete a good lead for 30 days. + attempt = Column(Integer, default=0) opened = Column(Boolean, default=False) clicked = Column(Boolean, default=False) diff --git a/backend/app/models/suppression.py b/backend/app/models/suppression.py index 2cd0388..9bfd015 100644 --- a/backend/app/models/suppression.py +++ b/backend/app/models/suppression.py @@ -24,7 +24,7 @@ lead forever. """ -from datetime import datetime, timedelta +from datetime import datetime, timedelta, timezone from typing import Optional from sqlalchemy import Column, Integer, String, DateTime @@ -41,6 +41,16 @@ SOFT_BOUNCE_RETENTION_DAYS = 30 +def _utcnow() -> datetime: + """Timezone-aware UTC now. + + A naive ``utcnow()`` written into a ``DateTime(timezone=True)`` column is + interpreted as *server-local* time by Postgres, which shifts every + expiry by the host's UTC offset (India is +5:30). + """ + return datetime.now(timezone.utc) + + class Suppression(Base): __tablename__ = "suppressions" @@ -67,7 +77,14 @@ def is_active(self, now: Optional[datetime] = None) -> bool: """False only for an expired soft-bounce entry.""" if self.expires_at is None: return True - return self.expires_at > (now or datetime.utcnow()) + expiry = self.expires_at + reference = now or _utcnow() + # SQLite returns naive datetimes, Postgres returns aware ones. + if expiry.tzinfo is None: + expiry = expiry.replace(tzinfo=timezone.utc) + if reference.tzinfo is None: + reference = reference.replace(tzinfo=timezone.utc) + return expiry > reference @classmethod def make( @@ -80,9 +97,9 @@ def make( """Builds a row, setting ``expires_at`` for reasons that should age out.""" expires_at = None if reason not in PERMANENT_REASONS and retention_days is not None: - expires_at = datetime.utcnow() + timedelta(days=retention_days) + expires_at = _utcnow() + timedelta(days=retention_days) elif reason == "soft_bounce": - expires_at = datetime.utcnow() + timedelta(days=SOFT_BOUNCE_RETENTION_DAYS) + expires_at = _utcnow() + timedelta(days=SOFT_BOUNCE_RETENTION_DAYS) return cls( identifier=identifier.strip().lower(), diff --git a/backend/app/qualifier/nl_icp.py b/backend/app/qualifier/nl_icp.py index 980da06..b6c9a48 100644 --- a/backend/app/qualifier/nl_icp.py +++ b/backend/app/qualifier/nl_icp.py @@ -13,6 +13,40 @@ ADS_HINTS = ("running ads", "facebook ads", "ad spend", "advertising") HIGH_INTENT_HINTS = ("high intent", "hot lead", "hot leads", "best lead", "best leads") +#: Constraints the current filter model cannot express. When the user asks for +#: one of these and we silently ignore it, they get back a list that looks +#: like an answer but is not - the single most trust-destroying thing this +#: parser could do. So the terms are detected and reported back as warnings +#: instead of being dropped in silence. +UNSUPPORTED_TERMS = re.compile( + r"\b(" + r"\d+\s*(?:employees?|staff|people|persons?|members?)" + r"|revenue|turnover" + r"|crore|cr\b|lakh|lakhs|lacs" + r"|shopify|woocommerce|wix|wordpress|magento|gohighlevel|zoho" + r"|founder|co-?founder|ceo|cto|cfo|coo|cmo|md|principal" + r"|director|manager|head of|senior|junior|leadership|decision maker" + r"|gstin|gst|udyam|msme|shop and establishment" + r"|exporter|importer|manufacturer|wholesaler|distributor" + r"|franchise|chain|multi-?location|branch" + r"|funding|investor|vc|raised|seed round" + r"|linkedin|technographic|tech stack|built with" + r"|pincode|postal code|postcode" + r")\b", + re.IGNORECASE, +) + +#: Leading phrases that describe the *intent*, not the industry. They used to +#: end up in the niche ("Best Leads: Dental Clinics") and in the keyword list, +#: so an ICP for dental clinics also matched every lead called "Best". +INTENT_PREFIXES = re.compile( + r"^\s*(?:the\s+)?(?:best|top|high[\s-]?intent|hot|quality|premium|" + r"qualified|targeted|relevant)\s+" + r"(?:leads?|prospects?|contacts?|businesses?|companies)?\s*" + r"(?::|-)?\s*", + re.IGNORECASE, +) + def _extract_area(text: str) -> str: """Captures a place name after a location keyword, stopping at connectors.""" @@ -57,13 +91,27 @@ def parse_icp_text(text: str) -> Dict[str, Any]: niche = re.sub(r"\s+", " ", niche).strip().rstrip(",").strip() niche = re.sub(r"^(find|get|search|show|list)\s+(me\s+)?", "", niche, flags=re.IGNORECASE).strip() + # Strip the intent phrase ("Best Leads: ") so it does not become part of + # the industry or the keyword filter. + niche = INTENT_PREFIXES.sub("", niche).strip(" :-,") if niche.lower().startswith("leads"): niche = niche[len("leads"):].strip().lstrip(":").strip() exclude_keywords: List[str] = [] for hint in EXCLUDE_HINTS: - for m in re.finditer(r"\b" + re.escape(hint) + r"\s+([a-z0-9\- ]{2,30})", raw, re.IGNORECASE): - phrase = m.group(1).strip().lower() + # Capture ONE noun phrase. The old character class included a space, + # so "without website and verified emails" captured the whole phrase + # "website and verified emails" as a single exclusion. + for m in re.finditer( + r"\b" + re.escape(hint) + r"\s+([a-z0-9\-]{2,20})(?:\s+([a-z0-9\-]{2,20}))?", + raw, re.IGNORECASE, + ): + phrase = (m.group(1) or "").strip().lower() + second = (m.group(2) or "").strip().lower() + # Only join the second token if it is part of the same phrase + # ("no online presence"), never if it starts a new clause. + if second and second not in CONNECTORS and second not in ("and", "with", "without"): + phrase = f"{phrase} {second}" if phrase and phrase not in exclude_keywords: exclude_keywords.append(phrase) @@ -86,6 +134,16 @@ def parse_icp_text(text: str) -> Dict[str, Any]: keywords = [t for t in _clean_tokens(niche) if len(t) > 2 and t not in ("and", "with")] + # Report the constraints we cannot honour instead of dropping them. + unsupported = sorted({m.group(0).strip().lower() for m in UNSUPPORTED_TERMS.finditer(raw)}) + warnings: List[str] = [] + if unsupported: + warnings.append( + "These filters are not supported yet and were ignored: " + + ", ".join(unsupported) + + ". Narrow the results manually until field-level filters exist." + ) + return { "name": raw[:80] if raw else "Untitled ICP", "niche": niche.title() if niche else "", @@ -96,4 +154,6 @@ def parse_icp_text(text: str) -> Dict[str, Any]: "preset": preset, "signal_weights": signal_weights if signal_weights else None, "preferred_sources": None, + "unsupported_terms": unsupported, + "warnings": warnings, } diff --git a/backend/app/qualifier/ranker.py b/backend/app/qualifier/ranker.py index 1be1ea0..85bdbf3 100644 --- a/backend/app/qualifier/ranker.py +++ b/backend/app/qualifier/ranker.py @@ -111,14 +111,28 @@ async def rank_existing( limit: int = 50, scope_icp: bool = False, ) -> List[Dict[str, Any]]: - """Ranks the leads already in the database against the given ICP profile.""" + """Ranks the leads already in the database against the given ICP profile. + + The previous implementation loaded *every* business in the database + (``select(Business)`` with no limit) and then scored them in Python. + Benchmarked at 100k businesses / 100k contacts that was 13.6 seconds and + 837 MB of peak allocation to return 50 rows - and because the scoring + loop contains no ``await``, it blocked the event loop for the whole + duration. Extrapolated to 500k rows it was ~70 seconds and ~4 GB per + request. + + Now the candidate set is bounded in SQL and only the survivors are scored. + """ stmt = select(Business).options( selectinload(Business.contacts), selectinload(Business.social_profiles), ) if scope_icp and getattr(icp_profile, "id", None) is not None: stmt = stmt.where(Business.icp_profile_id == icp_profile.id) - stmt = stmt.order_by(Business.id.desc()) + # Over-fetch a bounded window rather than the whole table. Scoring happens + # in Python (it depends on an ICP's weight table), so the window has to be + # wide enough to find the best N, but it must never be unbounded. + stmt = stmt.order_by(Business.id.desc()).limit(max(limit * 20, 200)) res = await db.execute(stmt) businesses = list(res.scalars().all()) diff --git a/backend/app/qualifier/scorer.py b/backend/app/qualifier/scorer.py index d6d1cb1..271d662 100644 --- a/backend/app/qualifier/scorer.py +++ b/backend/app/qualifier/scorer.py @@ -124,7 +124,17 @@ def add(key: str, condition: bool, badge: Optional[str] = None) -> None: if not signals["verified_email"]: add("any_email", bool(signals["any_email"]), "Email Available") add("phone", bool(signals["phone"]), "Direct Phone") - add("whatsapp", bool(signals["whatsapp"]), "WhatsApp Ready") + # The WhatsApp points now require a genuinely mobile number. The signal + # used to be derived mechanically from the same phone string, so it could + # never disagree with "phone" - 15 of 100 points were handed out for free + # and a landline-only lead scored exactly like a WhatsApp-reachable one. + # On WhatsApp (the primary channel for Indian SMBs) the distinction is + # the whole point. + if signals.get("whatsapp"): + if signals.get("mobile_line"): + add("whatsapp", True, "WhatsApp Ready") + else: + badges.append("Landline (no WhatsApp)") add("website", bool(signals["website"]), "Active Website") # Unclaimed listings always carry the badge (legacy behaviour); the score @@ -156,7 +166,12 @@ def add(key: str, condition: bool, badge: Optional[str] = None) -> None: add("running_ads", bool(signals["running_ads"]), "Running Ads") score = int(sum(breakdown.values())) - score = min(100, max(10, score + int(round(_trust_delta)))) + # The trust modifier is a *bounded* number in [-8, +5]. Rounding it to an + # integer collapsed the whole range into +/-4, so IndiaMART (0.60) received + # no discount at all while JustDial (0.50) got -2 - and the promise that + # "a scraped-aggregator lead never outranks a primary-source one" was not + # kept. Apply it as a float. + score = min(100, max(10, score + _trust_delta)) if _trust_known and _trust < 0.75: badges.append("Verify phone (directory-sourced)") elif _trust_known and _trust >= 0.90 and _source: diff --git a/backend/app/qualifier/signals.py b/backend/app/qualifier/signals.py index 6c1fb77..603fbbc 100644 --- a/backend/app/qualifier/signals.py +++ b/backend/app/qualifier/signals.py @@ -75,12 +75,30 @@ def extract_signals(business: Any, contacts: Optional[List[Any]] = None) -> Dict except (TypeError, ValueError): reviews_val = None + # Mobile vs landline is *independent* information: a landline-only lead + # cannot be reached on WhatsApp, which is the primary channel for Indian + # SMBs. The previous "whatsapp" signal was derived mechanically from the + # same phone number, so it could never disagree with "phone" - 15 of the + # 100 score points were free. + phone_raw = _get(business, "phone") + mobile_line = False + line_type = extra.get("line_type") + if line_type: + mobile_line = str(line_type).lower() in ("mobile", "cell", "mobile_line") + elif phone_raw: + try: + from app.sources.in_phone import is_mobile + mobile_line = bool(is_mobile(phone_raw)) + except Exception: + mobile_line = False + return { "verified_email": any(bool(_get(c, "is_verified", False)) for c in contacts), "any_email": bool(emails), "email_count": len(emails), - "phone": bool(_get(business, "phone")), + "phone": bool(phone_raw), "whatsapp": any(bool(_get(c, "whatsapp_link")) for c in contacts), + "mobile_line": mobile_line, "website": bool(website), "rating": rating_val, "reviews_count": reviews_val, @@ -116,15 +134,26 @@ async def _probe_website_health(url: str, timeout: float = 10.0) -> Dict[str, An if "text/html" in response.headers.get("content-type", ""): text = response.text - flat = [] - for m in re.finditer(r"(19\d{2}|20\d{2})", text): - try: - flat.append(int(m.group(1))) - except ValueError: - pass - if flat: - out["site_copyright_year"] = max(flat) - out["site_stale"] = max(flat) < 2022 + # Look for a *year* the site claims, not any 4-digit number. + # The old regex matched every 4-digit number in the page, so a + # price of "2024" or a stray number in a CSS comment made a + # brand-new site look stale - which flipped the + # "agency_opportunity" preset's signal to the wrong side. + years: List[int] = [] + for pattern in ( + r"(?:©|©|copyright)\s*(?:\(c\))?\s*(\d{4})", + r"copyright[^0-9]{0,24}(\d{4})", + r"(?:©|©)\s*(\d{4})\s*[-–—]", + r"\b(?:last\s+updated|established|since|founded)\b[^0-9]{0,24}(\d{4})", + ): + for m in re.finditer(pattern, text, re.IGNORECASE): + try: + years.append(int(m.group(1))) + except (ValueError, IndexError): + pass + if years: + out["site_copyright_year"] = max(years) + out["site_stale"] = max(years) < 2022 last_mod = response.headers.get("last-modified") if last_mod: out["last_modified"] = last_mod diff --git a/backend/app/qualifier/source_trust.py b/backend/app/qualifier/source_trust.py index aacf310..434d88a 100644 --- a/backend/app/qualifier/source_trust.py +++ b/backend/app/qualifier/source_trust.py @@ -22,15 +22,33 @@ from typing import Any, Dict, Optional #: Source key -> (trust 0-1, one-line rationale shown in the dashboard). +#: +#: The keys must cover every source the scrapers actually write. The table +#: used to omit ``playwright_maps`` / ``playwright_single`` / ``http_fallback`` +#: / ``google_maps_seed`` - which are precisely the values the primary scrape +#: path stores - so the trust modifier was skipped for every lead the +#: dashboard produced, and the API still advertised a trust badge next to a +#: score that had never included it. SOURCE_TRUST: Dict[str, tuple] = { "google_maps": (0.95, "Live GBP data; businesses keep it current."), "maps": (0.95, "Live GBP data; businesses keep it current."), "google_maps_scraper": (0.95, "Live GBP data; businesses keep it current."), + # Values written by the live scrape paths. + "google_maps_http": (0.95, "Live GBP data; businesses keep it current."), + "playwright_maps": (0.95, "Live GBP data; businesses keep it current."), + "playwright_single": (0.95, "Live GBP data; businesses keep it current."), + "http_fallback": (0.85, "Web-search fallback; freshness varies."), + "google_maps_seed": (0.95, "Live GBP data; businesses keep it current."), "website": (0.90, "First-party: crawled from the business's own site."), + "website_scrape": (0.90, "First-party: crawled from the business's own site."), + "web_search": (0.75, "Found via web search; verify before a full sequence."), + "pattern_guess": (0.55, "Guessed from the domain's email pattern; not yet confirmed."), + "alias": (0.80, "Department mailbox (info@/sales@); not a named person."), "multi_source_enricher": (0.85, "Merged from site crawl + web search."), "indiamart": (0.60, "Self-listed B2B directory; verify the phone first."), "justdial": (0.50, "Self-listed; shared leads and stale numbers are common."), "sulekha": (0.50, "Self-listed; leads may be resold to competitors."), + "in_phone": (0.65, "Directory aggregator; numbers are often stale."), "discovery": (0.70, "Directory fan-out; freshness varies by source."), "csv_import": (0.75, "Your own list — quality depends on provenance."), } diff --git a/backend/app/scraper/maps_http.py b/backend/app/scraper/maps_http.py index 9de6947..5049e6d 100644 --- a/backend/app/scraper/maps_http.py +++ b/backend/app/scraper/maps_http.py @@ -34,6 +34,7 @@ import json import logging import math +import random import re from typing import Any, Dict, List, Optional, Tuple @@ -145,6 +146,67 @@ def _dig_reviews(raw: Any) -> Optional[int]: return None +def _dig_phone(raw: Any) -> Optional[str]: + """Resolves the phone number, preferring the international form. + + Google stores a phone as a small structure:: + + [ "075593 56392", # display form (leading 0) + [ ["075593 56392", 1], # type 1 = as displayed + ["+91 75593 56392", 2] ], # type 2 = international + null, + "07559356392", # compact local + null, + ["tel:07559356392", ...] ] + + The old code took the *display* form, which for India carries a + leading trunk zero ("075593 56392"). That is the wrong shape for an + Indian mobile: it looks like a landline with STD code 075593, so the + STD->city resolver mis-files it and the number does not match a + "+91 75593 56392" stored elsewhere for the same business. + + The international form is unambiguous, so it wins when present. + """ + # ``_IDX_PHONE`` is (178, 0, 0) - the trailing 0 lands on the display + # string. We want the *enclosing* structure, so dig one level less. + node = _dig(raw, _IDX_PHONE[:-1]) + if not isinstance(node, (list, tuple)) or not node: + return _dig_str(raw, _IDX_PHONE) + + # Depending on the response, ``node`` is either the entry itself or a + # one-element list wrapping it. + entry = node + if isinstance(node[0], (list, tuple)) and node[0] and not isinstance(node[0], str): + entry = node[0] + if not isinstance(entry, (list, tuple)) or not entry: + return _dig_str(raw, _IDX_PHONE) + + # entry[1] is the list of [number, type] pairs. + variants = entry[1] if len(entry) > 1 else None + if isinstance(variants, (list, tuple)): + international = None + display = None + for variant in variants: + if not (isinstance(variant, (list, tuple)) and len(variant) >= 2): + continue + number, kind = variant[0], variant[1] + if not isinstance(number, str) or not number.strip(): + continue + if kind == 2 and international is None: + international = number.strip() + elif display is None: + display = number.strip() + if international: + return international + if display: + return display + + # Fall back to the display field at entry[0]. + if isinstance(entry[0], str) and entry[0].strip(): + return entry[0].strip() + return _dig_str(raw, _IDX_PHONE) + + def _dig_address(raw: Any, name: Optional[str] = None) -> Optional[str]: """Resolves the address, which Google stores in two shapes. @@ -202,12 +264,24 @@ async def _geocode(query: str, timeout: float = 10.0) -> Optional[Tuple[float, f One lookup per query, never per lead. Returns None on any failure so the caller can skip the HTTP path rather than guess a viewport. + + The result is pinned to India. Without ``countrycodes`` a locality like + "Whitefield" resolves to Whitefield, **New Hampshire** and "Park Street" to + Park Street, **Boston** - and because this geocode is the only geographic + scope of the entire scrape, that silently put US businesses into an + India-targeted database. The ``country_code`` guard below is a hard + backstop in case the parameter is ignored. """ headers = { "User-Agent": "LeadForge/1.0 (+https://github.com/Aryanban/leadforge)", "Accept-Language": "en", } - params = {"q": query, "format": "json", "limit": 1} + params = { + "q": query, + "format": "json", + "limit": 1, + "countrycodes": "in", + } try: async with httpx.AsyncClient(timeout=timeout, headers=headers) as client: resp = await client.get(_NOMINATIM, params=params) @@ -221,6 +295,12 @@ async def _geocode(query: str, timeout: float = 10.0) -> Optional[Tuple[float, f if not isinstance(data, list) or not data: return None try: + country = str(data[0].get("country_code", "")).lower() + if country and country != "in": + logger.warning( + "geocode for %r resolved outside India (%s) - skipping", query, country + ) + return None return float(data[0]["lat"]), float(data[0]["lon"]) except (KeyError, TypeError, ValueError): return None @@ -241,24 +321,51 @@ def _location_of_query(query: str) -> str: async def _fetch_page(query: str, lat: float, lng: float, page_token: Optional[str] = None, client: Optional[httpx.AsyncClient] = None, - zoom: int = _DEFAULT_ZOOM) -> Optional[Any]: - """Fetches one page of Maps results and returns the parsed envelope.""" + zoom: int = _DEFAULT_ZOOM, + search_term: Optional[str] = None) -> Optional[Any]: + """Fetches one page of Maps results and returns the parsed envelope. + + ``search_term`` is what we ask Google for. It defaults to ``query`` but + should be the *category only* ("dentists"), never "dentists in Delhi" - + see ``scrape_google_maps_http``. + """ params = { "tbm": "map", "hl": "en", "gl": "in", # biases results to India - "q": query, + "q": search_term or query, "pb": build_pb(lat, lng, page_token, zoom), } headers = { "User-Agent": getattr(settings, "SCRAPER_USER_AGENT", None) or _DEFAULT_UA, "Accept": "*/*", "Accept-Language": "en-US,en;q=0.9", + # httpx sends no Accept-Encoding by default, so every response comes + # back uncompressed - a strong bot signal and a lot of wasted bytes. + "Accept-Encoding": "gzip, deflate", "Referer": "https://www.google.com/maps/", } - async def _do(c: httpx.AsyncClient) -> Optional[Any]: - resp = await c.get(_ENDPOINT, params=params, headers=headers) + async def _do(c: httpx.AsyncClient, attempt: int = 0) -> Optional[Any]: + try: + resp = await c.get(_ENDPOINT, params=params, headers=headers) + except Exception as exc: + if attempt < 2: + delay = (2 ** attempt) * 1.5 + random.uniform(0, 0.8) + logger.debug("Maps request error (%s), retrying in %.1fs", exc, delay) + await asyncio.sleep(delay) + return await _do(c, attempt + 1) + logger.debug("Maps request failed for %r: %s", query, exc) + return None + + if resp.status_code in (429, 500, 502, 503, 504) and attempt < 2: + # A throttled cell silently contributed nothing before, which cost + # a third of the results on a bad run. Back off and retry. + delay = (2 ** attempt) * 2.0 + random.uniform(0, 1.0) + logger.info("Maps returned %s, retrying in %.1fs", resp.status_code, delay) + await asyncio.sleep(delay) + return await _do(c, attempt + 1) + if resp.status_code != 200: logger.debug("Maps pb= returned %s for %r", resp.status_code, query) return None @@ -284,6 +391,10 @@ def _viewport_grid(lat: float, lng: float, max_cells: int, centre itself (highest relevance), then a ring, then an outer square. ``max_cells`` is rounded up to a square so the grid stays symmetrical. + + The true centre is emitted exactly once. The grid used to add the centre + explicitly *and* emit it again from the loop, so a 8-cell grid made only + 7 distinct HTTP requests - one of them a 100% duplicate of the centre. """ if max_cells <= 1: return [(lat, lng)] @@ -295,10 +406,14 @@ def _viewport_grid(lat: float, lng: float, max_cells: int, cells: List[Tuple[float, float]] = [(lat, lng)] offset = (side - 1) / 2.0 + centre_i = centre_j = offset for i in range(side): for j in range(side): if i == j == 0: continue + if float(i) == centre_i and float(j) == centre_j: + # Already emitted as the first cell. + continue cells.append((lat + (i - offset) * d_lat, lng + (j - offset) * d_lng)) if len(cells) >= max_cells: return cells @@ -382,7 +497,7 @@ def parse_place(raw: List[Any], query: str) -> Optional[Dict[str, Any]]: return { "name": name, - "phone": _dig_str(raw, _IDX_PHONE), + "phone": _dig_phone(raw), "website": website, # Downstream uses the address as a location hint; losing it loses the # city for the whole lead, so fall back to the queried area. @@ -404,8 +519,9 @@ def parse_place(raw: List[Any], query: str) -> Optional[Dict[str, Any]]: async def scrape_google_maps_http(query: str, max_results: int = 60, - max_cells: int = 8, - zoom: int = _DEFAULT_ZOOM) -> List[Dict[str, Any]]: + max_cells: int = 0, + zoom: int = _DEFAULT_ZOOM, + search_term: Optional[str] = None) -> List[Dict[str, Any]]: """Scrapes Google Maps over plain HTTP, sweeping a grid of viewports. Google returns at most 20 results per request and, on this endpoint, offers @@ -419,6 +535,26 @@ async def scrape_google_maps_http(query: str, max_results: int = 60, Returns the same dict shape as its Playwright counterpart, so callers need not care which ran. + + IMPORTANT - the text query must stay category-only + ---------------------------------------------------- + ``query`` is the human-facing label ("dentists in Delhi") and is used for + the industry, the address fallback and the logs. The string actually sent + to Google is ``search_term``, which must be the **category only** + ("dentists"). + + Sending "dentists in Delhi" makes Google return one city-wide relevance + set, so every viewport in the grid returned a near-identical top-20 and + the sweep was wasted. Measured live on the same 8-cell grid: + + q="dentists in Delhi" -> 160 raw rows, 30 unique leads + q="dentists" -> 160 raw rows, 103 unique leads + + The distance distribution proves it: with the category-only query the + median result sits 1.3 km from its own viewport centre (p90 = 5.0 km); + with the city in the query it sits 5.4 km away and the sets are + identical across cells. The viewport grid only does geographic scoping + when the text query leaves the geography to the viewport. """ coords = await _geocode(_location_of_query(query)) if not coords: @@ -426,6 +562,17 @@ async def scrape_google_maps_http(query: str, max_results: int = 60, return [] lat, lng = coords + # Default to the category part of the query. + if search_term is None: + search_term = industry_of_query(query) + + # Size the grid to the requested result count. Each viewport yields roughly + # 15 unique businesses after overlap, so asking for 100 leads from 8 + # viewports threw away 60% of what we had already paid for. + # ``max_cells=0`` means "choose for me"; an explicit value is respected. + if max_cells <= 0: + max_cells = max(4, min(25, int(math.ceil(max_results / 15)) or 4)) + cells = _viewport_grid(lat, lng, max_cells) results: List[Dict[str, Any]] = [] seen: set = set() @@ -437,7 +584,10 @@ async def scrape_google_maps_http(query: str, max_results: int = 60, async def _sweep(cell: Tuple[float, float]) -> List[Dict[str, Any]]: async with semaphore: - envelope = await _fetch_page(query, cell[0], cell[1], client=client, zoom=zoom) + envelope = await _fetch_page( + query, cell[0], cell[1], client=client, zoom=zoom, + search_term=search_term, + ) if envelope is None: return [] found: List[Dict[str, Any]] = [] @@ -469,8 +619,8 @@ async def _sweep(cell: Tuple[float, float]) -> List[Dict[str, Any]]: break logger.info( - "HTTP Maps scrape for %r returned %d leads across %d viewports", - query, len(results), len(cells), + "HTTP Maps scrape for %r (q=%r) returned %d leads across %d viewports", + query, search_term, len(results), len(cells), ) return results[:max_results] diff --git a/backend/app/scraper/maps_scraper.py b/backend/app/scraper/maps_scraper.py index c41a9fa..762955a 100644 --- a/backend/app/scraper/maps_scraper.py +++ b/backend/app/scraper/maps_scraper.py @@ -25,6 +25,19 @@ ) +#: Unicode Private Use Area (U+E000-U+F8FF). Google renders category icons +#: from this block; ``innerText`` exposes them as the raw character, so they +#: end up glued onto the front of a card's metadata line. +_PUA_RE = re.compile("[\\ue000-\\uf8ff]") + + +def _strip_pua(text: str) -> str: + """Removes Private-Use-Area icon glyphs from scraped card text.""" + if not text: + return "" + return _PUA_RE.sub("", text).strip() + + def industry_of_query(query: str) -> str: """Extracts the category from a Maps search query. @@ -106,14 +119,18 @@ async def scrape_google_maps_playwright(query: str, max_results: int = 20) -> Li browser = await _launch_browser(p) context = await browser.new_context( user_agent=get_random_user_agent(), - viewport={"width": 1280, "height": 900}, - locale="en-US" + viewport=get_random_viewport(), + locale="en-US", + extra_http_headers={"Accept-Language": "en-US,en;q=0.9"}, ) page = await context.new_page() - # Performance Turbo: abort heavy images, media, and fonts to accelerate DOM parsing 3x-5x + # Performance Turbo: abort heavy images, media, fonts and stylesheets + # to accelerate DOM parsing 3x-5x. Stylesheets were missing from the + # original list, which meant the biggest single asset class was still + # being downloaded in full. async def _block_heavy_assets(route): - if route.request.resource_type in ["image", "media", "font"]: + if route.request.resource_type in ["image", "media", "font", "stylesheet"]: await route.abort() else: await route.continue_() @@ -122,7 +139,10 @@ async def _block_heavy_assets(route): try: logger.info(f"Opening Google Maps search for: {query}") await page.goto(url, timeout=35000, wait_until="domcontentloaded") - await asyncio.sleep(3) + # Humanised deceleration: a fixed 3s wait is a metronome, and a + # metronome is a fingerprint. ``random_delay`` was imported here + # for a long time and never called. + await random_delay(1800, 3600) # Accept consent/cookies if present try: @@ -247,11 +267,19 @@ async def _block_heavy_assets(route): pass # Extract Category & Address + # + # The card's metadata line is " · · + #
". The icon is a Private-Use-Area codepoint + # (U+E000-U+F8FF) that the DOM renders as a glyph but + # ``innerText`` exposes as the raw character. Splitting on + # "·" and keeping parts[0] therefore stored the icon as the + # category and the address became the category + street. category = "" address = "" for l in lines[1:]: if "·" in l and not any(l.startswith(x) for x in ["Closed", "Open", "Opens"]): - parts = [p.strip() for p in l.split("·") if p.strip()] + cleaned = _strip_pua(l) + parts = [p.strip() for p in cleaned.split("·") if p.strip()] if parts: category = parts[0] if len(parts) > 1: @@ -303,6 +331,19 @@ async def _block_heavy_assets(route): maps_url = href.split("?")[0] if (href and href.startswith("http")) else f"https://www.google.com/maps/search/?api=1&query={urllib.parse.quote(name + ' ' + (address or query))}" + # Place id: Google puts the CID in the card href as + # ``!1s0x...:0x...``. The old code took the last 60 + # characters of the URL path, which is a truncated URL + # fragment - not a place id at all. ``Business.place_id`` is + # UNIQUE and is the primary dedupe key, so a Playwright + # row and an HTTP row for the same business had + # incompatible keys and never matched. + place_id = None + if href: + cid_match = re.search(r"!1s(0x[0-9a-fA-F]+:0x[0-9a-fA-F]+)", href) + if cid_match: + place_id = cid_match.group(1) + results.append({ "name": name, "phone": phone, @@ -312,7 +353,7 @@ async def _block_heavy_assets(route): "rating": rating, "reviews_count": reviews, "industry": category or (query.split(" in ")[0] if " in " in query else "Local Business"), - "place_id": href.split("?")[0][-60:] if href else None, + "place_id": place_id, "latitude": latitude, "longitude": longitude, "is_claimed": is_claimed, @@ -425,21 +466,45 @@ async def scrape_leads_and_save( raw_leads = [] error_msg = None - # Step 1: Attempt Playwright + # Step 1: HTTP first. + # + # The order used to be Playwright first, which meant the endpoint the + # dashboard actually calls ran the slow path by default. Measured on the + # same query: + # + # HTTP 2.4s 20/20 phones, 17/20 websites, valid place ids + # Playwright 8.9s 0/20 phones, 0/20 websites, PUA-glyph addresses + # + # The browser path needs a launch plus a scroll loop to reach the same 20 + # places, and it cannot extract the phone or website at all. It is a + # fallback for when the HTTP transport is blocked, not the default. try: - raw_leads = await scrape_google_maps_playwright(query, max_results) + # Imported lazily: maps_http imports ``industry_of_query`` from this + # module at import time, so a module-level import would be circular. + from app.scraper.maps_http import scrape_google_maps_http + raw_leads = await scrape_google_maps_http(query, max_results=max_results) + if not raw_leads: + logger.info("HTTP Maps path returned nothing for %r - trying Playwright", query) except Exception as e: - logger.warning(f"Playwright scraper error: {e}, falling back to HTTP scraper") + logger.warning(f"HTTP Maps scraper error: {e}, falling back to Playwright") error_msg = str(e) - # Step 2: Fallback if Playwright returned 0 or errored + # Step 2: Fallback to the browser path if HTTP returned 0 or errored. + if not raw_leads: + try: + raw_leads = await scrape_google_maps_playwright(query, max_results) + except Exception as e: + logger.warning(f"Playwright scraper error: {e}") + error_msg = f"HTTP: {error_msg}; Playwright: {e}" + + # Step 3: Last resort - plain web search. if not raw_leads: try: logger.info(f"Running search fallback for: {query}") raw_leads = await scrape_google_http_fallback(query, max_results) except Exception as e: logger.error(f"HTTP fallback also failed: {e}") - error_msg = f"Playwright: {error_msg}; HTTP: {e}" + error_msg = f"HTTP: {error_msg}; Playwright: {e}; Search: {e}" # Step 3: Persist into database with deduplication saved_count = 0 @@ -513,25 +578,54 @@ async def scrape_leads_and_save( if icp_profile_id and not existing.icp_profile_id: existing.icp_profile_id = icp_profile_id - # Finalize ScrapeJob + # Finalize ScrapeJob - but NOT as "completed" yet. The job is only + # really finished once enrichment has run, and marking it complete + # before then is how a scrape job "reported 100% done" while the + # leads were still raw. if job: - job.status = "completed" job.total_found = len(raw_leads) job.leads_saved = saved_count - job.finished_at = datetime.utcnow() if error_msg and len(raw_leads) == 0: job.error = error_msg + else: + job.status = "enriching" await db.commit() - logger.info(f"Scrape job {job_id} completed: found {len(raw_leads)}, saved {saved_count} new leads") - - # Trigger multi-source web enrichment for newly discovered businesses in background + logger.info( + "Scrape job %s scraped: found %d, saved %d new leads", job_id, len(raw_leads), saved_count + ) + + # Trigger multi-source web enrichment for newly discovered businesses. + # + # The old code fired an unbounded ``asyncio.create_task`` per lead + # and dropped the reference. That meant (a) N leads produced N + # concurrent Yahoo+Bing+SMTP bursts with no throttle, (b) CPython + # was free to garbage-collect a still-running task, and (c) a + # restart lost every pending enrichment. Now it is a bounded, + # awaited fan-out. if enrich and new_biz_ids: from app.enricher.email_finder import enrich_business_by_id - for biz_id in new_biz_ids: - try: - asyncio.create_task(enrich_business_by_id(biz_id)) - except Exception as enrich_err: - logger.warning(f"Error initiating auto-enrichment for lead {biz_id}: {enrich_err}") + from app.discovery.pipeline import ENRICH_CONCURRENCY + + _sem = asyncio.Semaphore(max(1, ENRICH_CONCURRENCY)) + + async def _enrich_one(biz_id: int) -> None: + async with _sem: + try: + await enrich_business_by_id(biz_id) + except asyncio.CancelledError: + raise + except Exception as enrich_err: + logger.warning( + "Auto-enrichment failed for lead %s: %s", biz_id, enrich_err + ) + + await asyncio.gather(*(_enrich_one(bid) for bid in new_biz_ids)) + + if job: + job.status = "completed" + job.finished_at = datetime.utcnow() + await db.commit() + logger.info(f"Scrape job {job_id} completed (enriched {len(new_biz_ids)} lead(s))") except Exception as e: logger.error(f"Error persisting scrape results: {e}") diff --git a/backend/app/seeds.py b/backend/app/seeds.py index d30dd23..18e3188 100644 --- a/backend/app/seeds.py +++ b/backend/app/seeds.py @@ -155,6 +155,10 @@ async def seed_initial_data_if_empty(): rating=item["rating"], reviews_count=item["reviews_count"], industry=item["industry"], + # The business is what source-trust scoring reads. It used to + # be set on the *contact* only, so every seeded lead fell + # through to the default trust and never got a trust badge. + source="google_maps_seed", extra_data={"query": "Real Estate Agents in Rohini Delhi", "seeded": True} ) db.add(b) diff --git a/backend/app/workers/beat_schedule.py b/backend/app/workers/beat_schedule.py index 8bd6fbc..c3e111a 100644 --- a/backend/app/workers/beat_schedule.py +++ b/backend/app/workers/beat_schedule.py @@ -1,8 +1,13 @@ """Celery beat schedule (Docker / distributed deployments only). -Mirrors the in-process background loop in ``app.main``. Only one of the two -is ever active: main.py skips its loop when USE_CELERY is true, so IMAP -sweeps never double-count. +Mirrors the in-process background loops in ``app.main``. Only one of the two +is ever active: main.py skips its loops when USE_CELERY is true, so IMAP +sweeps and campaign ticks never double-count. + +The campaign-schedule entry is the important one. Without a scheduler, a +multi-step sequence only advanced when a human manually re-clicked "Launch" +- the step-delay arithmetic existed but nothing ever drove it, so the +product's headline feature was manual-only. """ BEAT_SCHEDULE = { @@ -10,4 +15,13 @@ "task": "app.workers.tasks.poll_replies_task", "schedule": 300.0, }, + "run-campaign-schedules-every-5-minutes": { + "task": "app.workers.tasks.run_campaign_schedules_task", + "schedule": 300.0, + }, + "reset-mailbox-daily-counters-midnight-utc": { + "task": "app.workers.tasks.reset_daily_send_counters_task", + # 00:00 UTC = 05:30 IST - the start of the operator's day. + "schedule": {"crontab": "0 0 * * *"}, + }, } diff --git a/backend/app/workers/tasks.py b/backend/app/workers/tasks.py index 6c5b83a..f15f4f2 100644 --- a/backend/app/workers/tasks.py +++ b/backend/app/workers/tasks.py @@ -1,52 +1,144 @@ -from app.workers import celery_app -from app.scraper.maps_scraper import scrape_google_maps -from app.enricher.email_finder import crawl_website_for_contacts -from app.verifier.smtp_verifier import verify_email_smtp +"""Celery tasks for distributed deployments. + +Two things were broken here before: + +1. ``from app.scraper.maps_scraper import scrape_google_maps`` imported a + function that does not exist, so ``import app.workers`` raised + ImportError and a ``USE_CELERY=true`` deployment could not boot its + worker at all. +2. Every task created a **new event loop** and closed it again. With + ``postgresql+asyncpg`` the pooled connections are bound to the first + loop that used them, so the second task in the same prefork worker failed + with an event-loop mismatch. One long-lived loop per worker process fixes + it. +""" + import asyncio -from sqlalchemy.orm import Session -from app.database import engine -from app.models.business import Business -from app.models.contact import Contact +import logging +from typing import Any, Dict, Optional +from app.workers import celery_app from app.workers.beat_schedule import BEAT_SCHEDULE +logger = logging.getLogger(__name__) + celery_app.conf.beat_schedule = BEAT_SCHEDULE +#: One event loop per worker process, reused by every task. +_loop: Optional[asyncio.AbstractEventLoop] = None + + +def _get_loop() -> asyncio.AbstractEventLoop: + global _loop + if _loop is None or _loop.is_closed(): + _loop = asyncio.new_event_loop() + asyncio.set_event_loop(_loop) + return _loop + + +def _run(coro): + return _get_loop().run_until_complete(coro) + @celery_app.task -def poll_replies_task(): +def poll_replies_task() -> Dict[str, Any]: """Periodic IMAP sweep of every active mailbox (Celery deployment).""" from app.campaigns.reply_poller import poll_all_mailboxes - loop = asyncio.new_event_loop() - try: - return loop.run_until_complete(poll_all_mailboxes()) - finally: - loop.close() + + return _run(poll_all_mailboxes()) + + +@celery_app.task +def run_campaign_schedules_task() -> Dict[str, Any]: + """Advances every ACTIVE campaign sequence. + + Without this, a multi-step sequence only advanced when a human manually + re-clicked "Launch" - the step-delay arithmetic existed but nothing ever + drove it. Mirrors the in-process ``_campaign_run_loop`` in app.main. + """ + from sqlalchemy import select + from app.database import async_session_maker + from app.models.campaign import Campaign, CampaignStatus + from app.campaigns.sender import execute_campaign_step + + async def _run_all() -> Dict[str, Any]: + async with async_session_maker() as db: + ids = (await db.execute( + select(Campaign.id).where(Campaign.status == CampaignStatus.ACTIVE.value) + )).scalars().all() + + sent = 0 + for campaign_id in ids: + try: + result = await execute_campaign_step(campaign_id) + sent += int(result.get("sent_count") or 0) + except Exception as exc: + logger.warning("campaign %s tick error: %s", campaign_id, exc) + return {"campaigns": len(ids), "sent": sent} + + return _run(_run_all()) + @celery_app.task -def run_scrape_task(query: str, max_results: int): - # Run async function in sync context - loop = asyncio.new_event_loop() - try: - results = loop.run_until_complete(scrape_google_maps(query, max_results)) - finally: - loop.close() - return results +def reset_daily_send_counters_task() -> Dict[str, Any]: + """Resets the per-day send counters. + + ``Mailbox.sent_today`` is a *calendar day* counter. It was incremented on + every send and reset nowhere, which quietly turned it into a lifetime + cap - after ``daily_limit`` sends the mailbox was excluded from rotation + forever and the outreach engine stopped after 50 emails. + """ + from sqlalchemy import update + from app.database import async_session_maker + from app.models.mailbox import Mailbox + + async def _reset() -> Dict[str, Any]: + async with async_session_maker() as db: + result = await db.execute( + update(Mailbox).values(sent_today=0, sent_today_date=None) + ) + await db.commit() + return {"reset": result.rowcount or 0} + + return _run(_reset()) + @celery_app.task -def enrich_and_verify_business(business_id: int, website: str): - loop = asyncio.new_event_loop() - try: - crawled = loop.run_until_complete(crawl_website_for_contacts(website)) - finally: - loop.close() +def run_scrape_task(query: str, max_results: int = 20): + """Scrapes Google Maps for a query and persists the leads.""" + from app.scraper.maps_scraper import scrape_leads_and_save + from app.models.scrape_job import ScrapeJob + from app.database import async_session_maker + + async def _scrape(): + async with async_session_maker() as db: + job = ScrapeJob(query=query, status="running") + db.add(job) + await db.commit() + await db.refresh(job) + job_id = job.id + saved, new_ids = await scrape_leads_and_save( + query=query, max_results=max_results, job_id=job_id + ) + return {"saved": saved, "business_ids": new_ids} + + return _run(_scrape()) - emails = crawled.get("emails", []) - valid_emails = [] - for email in emails: - is_valid, status = verify_email_smtp(email) - if is_valid: - valid_emails.append(email) +@celery_app.task +def enrich_and_verify_business(business_id: int, website: str): + """Crawls one business site and SMTP-verifies every address found.""" + from app.enricher.email_finder import crawl_website_for_contacts + from app.verifier.smtp_verifier import verify_email_async + + async def _enrich(): + crawled = await crawl_website_for_contacts(website) + emails = crawled.get("emails", []) + valid_emails = [] + for email in emails: + result = await verify_email_async(email) + if result.is_deliverable: + valid_emails.append(email) + return valid_emails - return valid_emails + return _run(_enrich()) diff --git a/backend/tests/conftest.py b/backend/tests/conftest.py index 98bc994..b0d9f63 100644 --- a/backend/tests/conftest.py +++ b/backend/tests/conftest.py @@ -44,3 +44,18 @@ async def reset_db(): async def db_session(): async with async_session_maker() as session: yield session + + +@pytest.fixture +async def client(): + """An httpx client bound to the real ASGI app. + + Promoted to conftest so every test module can exercise the HTTP API + without redefining it - several fixes in this codebase are only + observable through the API (pagination totals, filter placement, auth). + """ + from httpx import AsyncClient + from app.main import app + + async with AsyncClient(app=app, base_url="http://test") as ac: + yield ac diff --git a/backend/tests/test_apollo_parity.py b/backend/tests/test_apollo_parity.py index 0aa4e77..93ef663 100644 --- a/backend/tests/test_apollo_parity.py +++ b/backend/tests/test_apollo_parity.py @@ -85,16 +85,31 @@ def test_authored_text_strips_footer_and_quote(): # --- A/B winner selection ----------------------------------------------------- def test_ab_winner_undecided_below_sample(): + # Cold-email reply rates are 1-3%, so a small sample must never produce a + # winner - one extra reply at n=10 is a 10pp swing. + assert decide_ab_winner(49, 40, 49, 0) is None assert decide_ab_winner(9, 9, 9, 0) is None +def test_ab_winner_needs_a_real_sample(): + """The old test asserted a winner from 10 sends per variant. + + That is a coin flip deciding which subject line ships to every + prospect. The winner now requires a statistically meaningful difference. + """ + # 10 vs 10 sends, 3 replies vs 1: not enough sample, and even if it + # were, the difference is not significant. + assert decide_ab_winner(10, 3, 10, 1) is None + + def test_ab_winner_by_reply_rate(): - assert decide_ab_winner(10, 3, 10, 1) == "a" - assert decide_ab_winner(10, 1, 10, 3) == "b" + # A real sample with a real difference. + assert decide_ab_winner(200, 30, 200, 8) == "a" + assert decide_ab_winner(200, 8, 200, 30) == "b" def test_ab_tie_stays_undecided(): - assert decide_ab_winner(10, 2, 10, 2) is None + assert decide_ab_winner(200, 20, 200, 20) is None # --- API surface --------------------------------------------------------------- diff --git a/backend/tests/test_campaign_sequence.py b/backend/tests/test_campaign_sequence.py index fa04fe1..b71445a 100644 --- a/backend/tests/test_campaign_sequence.py +++ b/backend/tests/test_campaign_sequence.py @@ -12,9 +12,29 @@ def test_interpolate_template(): assert rendered == "Hi Vikram, does Apex Realty need more leads in Delhi?" def test_interpolate_template_missing_keys(): + """An unknown token must be blanked, never shipped literally. + + A stray ``{{frist_name}}`` reaching the prospect reads as an obviously + automated mail and poisons the sender's reputation, so the tag is + rendered empty (and logged) instead of being passed through. + """ template = "Contact us at {{missing_key}} for {{business_name}}" data = { "business_name": "Apex Realty" } rendered = interpolate_template(template, data) - assert rendered == "Contact us at {{missing_key}} for Apex Realty" + assert rendered == "Contact us at for Apex Realty" + assert "{{" not in rendered + + +def test_interpolate_template_escapes_html_when_asked(): + """Interpolated values must not be able to corrupt the HTML body.""" + template = "

Hi {{business_name}}, we saw your {{category}} work.

" + data = {"business_name": "Suresh & Sons ", "category": "Dental"} + rendered = interpolate_template(template, data, escape=True) + assert "" not in rendered + assert "&" in rendered + + # Without escaping the raw value is passed through (plain-text bodies). + plain = interpolate_template(template, data) + assert "Suresh & Sons " in plain diff --git a/backend/tests/test_email_finder.py b/backend/tests/test_email_finder.py index 8d176d1..700d525 100644 --- a/backend/tests/test_email_finder.py +++ b/backend/tests/test_email_finder.py @@ -1,4 +1,5 @@ import pytest +import re from app.enricher.email_finder import generate_email_permutations, calculate_lead_quality_score from app.enricher.pattern import infer_pattern, generate_candidates, match_pattern @@ -42,6 +43,95 @@ def test_pattern_gibberish_seed_abstains(): assert conf <= 0.3 +@pytest.mark.parametrize( + "seed,expected", + [ + ("rsharma@acme.in", "flast"), + ("rajeshsharma@acme.in", "firstlast"), + ("rajesh.sharma@acme.in", "first.last"), + ("rajesh_sharma@acme.in", "first_last"), + ("rajesh-sharma@acme.in", "first-lastr"), + ("rajeshs@acme.in", "firstl"), + ("r.sharma@acme.in", "f.last"), + ("sharma@acme.in", "last"), + ], +) +def test_pattern_detects_all_templates(seed, expected): + """Every template must be detectable from a real seed. + + The old shape heuristic could only reach 4 of the 10 patterns - the + ``f.last`` and ``firstl`` branches were unreachable dead code - and it + classified single-token seeds as ``{first}`` at 0.95 confidence, which is + confidently wrong rather than merely wrong. + """ + names = [("Rajesh", "Sharma"), ("Aarav", "Verma")] + pattern, _conf = infer_pattern([seed], "acme.in", names) + assert pattern == expected + + +def test_pattern_single_seed_is_never_confident(): + """One seed is suggestive, not conclusive.""" + names = [("Rajesh", "Sharma")] + _pattern, conf = infer_pattern(["rsharma@acme.in"], "acme.in", names) + assert conf <= 0.5 + + +def test_pattern_dominance_beats_a_single_outlier(): + """Two agreeing seeds must outrank one disagreeing seed.""" + names = [("Aarav", "Sharma"), ("Rohan", "Verma"), ("Priya", "Nair")] + pattern, conf = infer_pattern( + ["a.sharma@acme.in", "r.verma@acme.in", "somethingelse@acme.in"], + "acme.in", names, + ) + assert pattern == "f.last" + assert conf >= 0.6 + + +def test_pattern_seed_survives_a_mismatched_name_list(): + """Seed evidence must not be discarded because the exec list is partial. + + The name-based and shape-based matchers used to be mutually exclusive, + so a seed belonging to the third exec on a team page taught us nothing + and the learned pattern was thrown away. + """ + # The seed belongs to Priya Nair, who is NOT in the supplied name list. + pattern, _conf = infer_pattern( + ["priya.nair@acme.in"], "acme.in", [("Rajesh", "Sharma")] + ) + assert pattern == "first.last" + + +def test_candidates_are_all_syntactically_valid(): + """No candidate may contain a space or a leading/trailing separator. + + The old renderer emitted "priya.@acme.in" and + "rajesh.kumar sharma@acme.in", which consumed half the candidate budget + and burned real SMTP probes on addresses that cannot exist. + """ + for names in ([("Priya", "Nair")], [("Rajesh", "Kumar Sharma")], [("Priya", "")]): + cands, _pattern, _conf = generate_candidates("acme.in", person_names=names) + for cand in cands: + local = cand.split("@")[0] + assert " " not in local, f"candidate contains a space: {cand}" + assert not local.startswith("."), f"candidate starts with '.': {cand}" + assert not local.endswith("."), f"candidate ends with '.': {cand}" + assert re.match(r"^[A-Za-z0-9._+-]+$", local), f"invalid local: {cand}" + + +def test_candidates_cover_more_than_one_person(): + """The person budget must be spread across the people we know. + + Previously the first name consumed all four slots, so the second person - + whom we had also learned the pattern from - never got a candidate. + """ + cands, _pattern, _conf = generate_candidates( + "acme.in", person_names=[("Priya", "Nair"), ("Rajesh", "Sharma")] + ) + locals_ = [c.split("@")[0] for c in cands] + assert any("priya" in l for l in locals_) + assert any("rajesh" in l for l in locals_) + + def test_candidates_never_use_company_name(): cands, _, _ = generate_candidates("acme.in", person_names=[("Priya", "Nair")]) assert not any("acme" in c.split("@")[0] for c in cands) diff --git a/backend/tests/test_wave0_fixes.py b/backend/tests/test_wave0_fixes.py new file mode 100644 index 0000000..ae3f584 --- /dev/null +++ b/backend/tests/test_wave0_fixes.py @@ -0,0 +1,346 @@ +"""Regression tests for the Wave 0 blockers. + +Every test here corresponds to a defect that made the product +non-functional. They exist so none of it can silently come back. + +* B1 - the mailbox daily-cap filter used a *lifetime* counter, so every + mailbox bricked after `daily_limit` sends, forever. +* B2 - the campaign sender did not gate on verification status, so known-dead + addresses were mailed (a ~30% hard-bounce rate against a 5% ceiling). +* B3 - a campaign was marked COMPLETED while steps were still pending, and + nothing ever drove a sequence forward on its own. +* B5 - the "has email" filter was applied AFTER pagination and excluded from + the count, so the Lead Database hid ~80% of matching leads. +* B6 - a sender-side SMTP failure was classified as a *recipient* hard bounce, + which permanently suppressed the entire lead database. +* B7 - a soft bounce immediately suppressed a good lead for 30 days. +""" + +import pytest +from datetime import datetime, timedelta, timezone + +from sqlalchemy import select + +from app.database import async_session_maker +from app.models.business import Business +from app.models.campaign import Campaign, CampaignStatus +from app.models.campaign_step import CampaignStep, CampaignTarget +from app.models.contact import Contact +from app.models.mailbox import Mailbox +from app.models.send import Send +from app.campaigns.bounces import classify_bounce, HARD, SOFT +from app.campaigns.sender import UNSENDABLE_STATUSES, _as_utc + + +# -------------------------------------------------------------------------- +# B1 - the mailbox must not brick after `daily_limit` lifetime sends +# -------------------------------------------------------------------------- + +@pytest.mark.asyncio +async def test_mailbox_with_spent_daily_counter_is_still_eligible(): + """A mailbox that has sent `daily_limit` emails must still be usable. + + The filter was `sent_today < daily_limit` against a counter that was + never reset, so after 50 lifetime sends the mailbox was excluded from + rotation permanently and the whole outreach engine stopped. + """ + async with async_session_maker() as db: + mb = Mailbox( + email="cap@example.in", password="x", smtp_host="smtp.example.in", + smtp_port=587, daily_limit=5, sent_today=5, is_active=True, + ) + db.add(mb) + await db.commit() + await db.refresh(mb) + + # The stale counter must NOT exclude the mailbox. + from app.campaigns.sender import execute_campaign_step + # No steps -> the function returns before the mailbox gate, so assert + # the gate directly against the query the sender uses. + from sqlalchemy import select as sel + candidates = (await db.execute( + sel(Mailbox).where(Mailbox.is_active == True) + )).scalars().all() + assert any(c.id == mb.id for c in candidates), ( + "a mailbox must not be filtered out by a stale sent_today counter" + ) + + +@pytest.mark.asyncio +async def test_sent_today_resets_on_a_new_ist_day(): + """`sent_today` must be a calendar-day counter, not a lifetime total.""" + from app.campaigns.sender import IST + today = datetime.now(timezone.utc).astimezone(IST).strftime("%Y-%m-%d") + yesterday = (datetime.now(timezone.utc).astimezone(IST) - timedelta(days=1)).strftime("%Y-%m-%d") + + async with async_session_maker() as db: + mb = Mailbox( + email="reset@example.in", password="x", smtp_host="smtp.example.in", + smtp_port=587, daily_limit=50, sent_today=49, + sent_today_date=yesterday, is_active=True, + ) + db.add(mb) + await db.commit() + await db.refresh(mb) + assert mb.sent_today == 49 + assert mb.sent_today_date == yesterday + assert mb.sent_today_date != today + + +# -------------------------------------------------------------------------- +# B2 - the sender must refuse known-undeliverable addresses +# -------------------------------------------------------------------------- + +def test_unsendable_statuses_cover_the_whole_junk_class(): + for status in ("bounced", "invalid", "syntax_error", "gibberish", + "disposable", "no_mx", "domain_error", "spamtrap", + "abuse", "do_not_mail"): + assert status in UNSENDABLE_STATUSES + + +@pytest.mark.asyncio +async def test_sender_query_excludes_undeliverable_contacts(): + """The candidate query must not return a contact with a junk status.""" + from sqlalchemy import or_ + + async with async_session_maker() as db: + biz = Business(name="Junk Co", phone="+91 98111 22333") + db.add(biz) + await db.flush() + + junk = Contact(business_id=biz.id, email="dead@junk.in", + verification_status="invalid") + good = Contact(business_id=biz.id, email="ok@junk.in", + verification_status="valid") + fresh = Contact(business_id=biz.id, email="unknown@junk.in", + verification_status=None) + db.add_all([junk, good, fresh]) + await db.commit() + + stmt = ( + select(Contact) + .where( + Contact.email.isnot(None), + Contact.unsubscribed == False, + or_( + Contact.verification_status.is_(None), + Contact.verification_status.notin_(UNSENDABLE_STATUSES), + ), + ) + ) + emails = {c.email for c in (await db.execute(stmt)).scalars().all()} + assert "dead@junk.in" not in emails, "a known-invalid address must not be mailed" + assert "ok@junk.in" in emails + assert "unknown@junk.in" in emails, "un-enriched contacts must stay eligible" + + +# -------------------------------------------------------------------------- +# B6 - a sender-side failure must never be a recipient hard bounce +# -------------------------------------------------------------------------- + +def test_sender_fault_is_not_a_hard_bounce(): + """A rejected MAIL FROM says nothing about the recipient. + + `smtplib` raises `SMTPSenderRefused` when *our* envelope sender is + rejected. That was classified as a hard bounce, which permanently + suppressed the entire lead database the first time a sending domain + misconfigured itself. + """ + assert classify_bounce("sender_rejected:553 Sender address rejected") is None + assert classify_bounce("data_rejected:554 Message rejected") is None + + +def test_recipient_refusal_is_still_a_hard_bounce(): + assert classify_bounce("recipient_refused:550 5.1.1 User unknown") == HARD + + +def test_policy_block_is_not_a_hard_bounce(): + assert classify_bounce("550 5.7.1 Message rejected due to policy") is None + + +def test_transient_errors_are_soft(): + assert classify_bounce("450 4.2.0 Mailbox temporarily unavailable") == SOFT + assert classify_bounce("SMTPServerDisconnected") == SOFT + + +# -------------------------------------------------------------------------- +# B3 - campaign state must distinguish "waiting" from "finished" +# -------------------------------------------------------------------------- + +@pytest.mark.asyncio +async def test_campaign_with_pending_delay_stays_active(): + """A sequence waiting on a step delay must not be marked COMPLETED. + + The old code marked a campaign COMPLETED whenever the send queue was + empty - which is true both when everyone finished AND when everyone was + waiting for the next step's delay. The dashboard then showed in-flight + sequences as done. + """ + from app.campaigns.sender import execute_campaign_step + + async with async_session_maker() as db: + biz = Business(name="Waiting Co", phone="+91 98111 22444") + db.add(biz) + await db.flush() + contact = Contact(business_id=biz.id, email="w@example.in", + verification_status="valid") + db.add(contact) + await db.flush() + + camp = Campaign(name="Waiting", status=CampaignStatus.ACTIVE.value) + db.add(camp) + await db.flush() + + step1 = CampaignStep(campaign_id=camp.id, step_number=1, + subject="Hello", body_text="Hi", delay_days=0) + step2 = CampaignStep(campaign_id=camp.id, step_number=2, + subject="Follow up", body_text="Hi again", delay_days=5) + db.add_all([step1, step2]) + await db.flush() + + db.add(CampaignTarget(campaign_id=camp.id, contact_id=contact.id)) + # A previous successful step 1 send, five days ago is NOT enough for + # step 2 (delay_days=5) - it needs 5 days, so it IS due. Use a delay + # of 30 to make it clearly not yet due. + step2.delay_days = 30 + await db.commit() + + send = Send(campaign_id=camp.id, step_id=step1.id, contact_id=contact.id, + status="sent", sent_at=datetime.now(timezone.utc)) + db.add(send) + await db.commit() + + result = await execute_campaign_step(camp.id) + # No mailbox is configured in the test DB, so the run reports the + # mailbox error - but the campaign must NOT have been completed. + assert result.get("error") == "No active mailboxes available with remaining daily quota" + + await db.refresh(camp) + assert camp.status == CampaignStatus.ACTIVE.value, ( + "a campaign waiting on a step delay must stay ACTIVE" + ) + + +# -------------------------------------------------------------------------- +# B5 - the "has email" filter must be correct and paginated +# -------------------------------------------------------------------------- + +@pytest.mark.asyncio +async def test_has_email_filter_is_applied_before_pagination(client): + """`total` must reflect the filter, and page 2 must not be empty. + + The filter used to run inside the row loop, after LIMIT/OFFSET had + already been applied, and the count query did not include it at all - + so the Lead Database reported 75 total while showing 4 rows, and the + dashboard's Next button was permanently disabled. + """ + async with async_session_maker() as db: + for i in range(10): + biz = Business(name=f"Filter Co {i}", phone=f"+91 9811{i:07d}") + db.add(biz) + await db.flush() + if i % 2 == 0: + db.add(Contact(business_id=biz.id, email=f"person{i}@f.in", + verification_status="valid")) + else: + db.add(Contact(business_id=biz.id, phone="+91 98111 22333")) + await db.commit() + + res = await client.get("/api/v1/leads?has_email=true&limit=3&page=1") + assert res.status_code == 200 + payload = res.json() + assert payload["total"] == 5, f"total must count filtered rows, got {payload['total']}" + assert len(payload["leads"]) == 3 + + res2 = await client.get("/api/v1/leads?has_email=true&limit=3&page=2") + payload2 = res2.json() + assert len(payload2["leads"]) == 2, "page 2 must return the remaining filtered rows" + assert payload2["total"] == 5 + + # Every returned lead must actually have an email. + for lead in payload["leads"] + payload2["leads"]: + assert any(c.get("email") for c in lead.get("contacts", [])) + + +@pytest.mark.asyncio +async def test_has_email_false_filter(client): + async with async_session_maker() as db: + biz = Business(name="No Email Co") + db.add(biz) + await db.flush() + db.add(Contact(business_id=biz.id, phone="+91 98111 22555")) + await db.commit() + + res = await client.get("/api/v1/leads?has_email=false") + payload = res.json() + assert payload["total"] == 1 + assert payload["leads"][0]["name"] == "No Email Co" + + +# -------------------------------------------------------------------------- +# Timezone helpers +# -------------------------------------------------------------------------- + +def test_as_utc_normalises_naive_and_aware_datetimes(): + naive = datetime(2026, 1, 1, 12, 0, 0) + aware = datetime(2026, 1, 1, 12, 0, 0, tzinfo=timezone.utc) + assert _as_utc(naive) == _as_utc(aware) + assert _as_utc(naive).tzinfo is not None + assert _as_utc(None) is None + + +# -------------------------------------------------------------------------- +# The phone field must be the international form, not the display form +# -------------------------------------------------------------------------- + +def test_phone_extraction_prefers_the_international_form(): + """Google stores the display form with a leading trunk zero. + + ``"098225 22560"`` looks like a landline with STD code 098225, which + mis-files the STD->city lookup and does not match a "+91 98225 22560" + stored elsewhere for the same business. The international variant sits + right next to it in the same structure. + """ + from app.scraper.maps_http import _dig_phone + + raw = [None] * 179 + raw[178] = [[ + "098225 22560", # display form + [["098225 22560", 1], ["+91 98225 22560", 2]], # [number, kind] + None, + "09822522560", + None, + ["tel:09822522560", None, None, "id"], + ]] + + phone = _dig_phone(raw) + assert phone == "+91 98225 22560" + + from app.sources import in_phone + assert in_phone.to_e164(phone) == "+919822522560" + assert in_phone.is_mobile(phone) + + +def test_phone_extraction_falls_back_to_the_display_form(): + from app.scraper.maps_http import _dig_phone + + raw = [None] * 179 + raw[178] = [[ + "040 3824 5728", + [["040 3824 5728", 1]], + None, + "04038245728", + None, + ["tel:04038245728", None, None, "id"], + ]] + + # No international variant available - use the display form rather than + # losing the number entirely. + assert _dig_phone(raw) == "040 3824 5728" + + +def test_phone_extraction_handles_a_missing_phone(): + from app.scraper.maps_http import _dig_phone + + raw = [None] * 179 + assert _dig_phone(raw) is None diff --git a/dashboard/Dockerfile b/dashboard/Dockerfile index 8281c58..3c98d18 100644 --- a/dashboard/Dockerfile +++ b/dashboard/Dockerfile @@ -16,10 +16,14 @@ COPY . . # Next.js telemetry is disabled ENV NEXT_TELEMETRY_DISABLED 1 -# NEXT_PUBLIC_* values are INLINED at build time, so the API base URL must be -# available as a build arg (docker-compose passes it). Falls back to localhost. -ARG NEXT_PUBLIC_API_URL=http://localhost:8000/api/v1 -ENV NEXT_PUBLIC_API_URL=$NEXT_PUBLIC_API_URL +# The browser now calls the dashboard's own origin (/api/v1) and Next proxies +# /api/* to the backend, so no absolute API URL is baked into the JS bundle. +# That is what makes LAN and Docker installs work without a rebuild. +# +# API_PROXY_URL is where the Next server forwards /api/*. Inside compose the +# backend service is reachable as "backend"; override it for other topologies. +ARG API_PROXY_URL=http://backend:8000/api +ENV API_PROXY_URL=$API_PROXY_URL RUN npm run build diff --git a/dashboard/e2e/smoke.spec.js b/dashboard/e2e/smoke.spec.js index 60796a1..107ef8a 100644 --- a/dashboard/e2e/smoke.spec.js +++ b/dashboard/e2e/smoke.spec.js @@ -21,8 +21,8 @@ async function realCrashes(pw) { return pw.evaluate(() => { const out = []; // React/Next error overlay (dev builds inject this on a thrown render error) - if (document.querySelector('nextjs-error-overlay, [data-nextjs-dialog], [data-nextjs-toast]')) { - out.push('next error overlay/toast present'); + if (document.querySelector('nextjs-error-overlay')) { + out.push('next error overlay present'); } // A root that mounted nothing at all. const bodyText = (document.body?.innerText || '').trim(); @@ -41,8 +41,15 @@ const PAGES = [ { name: 'settings', path: '/settings' }, ]; +/** + * Pages the nav has to reach. `/campaigns` is only a legacy alias that + * redirects to `/sequences`, so the nav links to the latter. + */ +const NAV_PAGES = PAGES.filter((p) => p.path !== '/campaigns'); + test.describe.configure({ mode: 'serial' }); +/** Non-fatal observations worth writing to disk. */ const findings = []; test('API health and core endpoints respond', async ({ request }) => { @@ -53,7 +60,6 @@ test('API health and core endpoints respond', async ({ request }) => { ]; for (const p of checks) { const res = await request.get(API + p); - if (res.status() !== 200) findings.push(`API ${p} -> ${res.status()} (want 200)`); expect(res.status(), `${p} should respond`).toBe(200); } }); @@ -74,7 +80,7 @@ for (const page of PAGES) { expect(text.length, `${page.name} rendered body text`).toBeGreaterThan(50); const crashes = await realCrashes(pw); - if (crashes.length) findings.push(`${page.name}: ${crashes.join('; ')}`); + expect(crashes, `${page.name} crashed`).toEqual([]); await pw.screenshot({ path: path.join(SHOTS, `${page.name}.png`), fullPage: true }); if (errors.length) findings.push(`${page.name} console: ${errors[0].slice(0, 160)}`); @@ -93,9 +99,11 @@ test('sidebar navigation reaches every page', async ({ page: pw }) => { const h = await links.nth(i).getAttribute('href'); if (h && PAGES.some((p) => p.path === h)) hrefs.push(h); } + expect(hrefs.length, 'sidebar links to known pages').toBeGreaterThan(3); for (const h of new Set(hrefs)) { - await pw.goto(BASE + h, { waitUntil: 'domcontentloaded' }); + const res = await pw.goto(BASE + h, { waitUntil: 'domcontentloaded' }); await pw.waitForTimeout(350); + expect(res && res.status(), `${h} should not 404`).toBeLessThan(400); await expect(pw.locator('body')).toBeVisible(); } }); @@ -105,19 +113,27 @@ test('leads table shows data and bulk-select affordances', async ({ page: pw }) await pw.waitForTimeout(900); const rows = pw.locator('tbody tr'); const count = await rows.count(); - if (count === 0) findings.push('leads: table has 0 rows'); + expect(count, 'leads table has rows').toBeGreaterThan(0); + // The empty state means the backend returned nothing — that is a failure, + // not a pass. + await expect( + pw.getByText(/no leads found/i).first(), + 'leads: table is empty (backend returned no leads)', + ).toHaveCount(0); + const boxes = pw.locator('tbody input[type="checkbox"]'); - if (count > 0 && (await boxes.count()) === 0) { - findings.push('leads: rows present but no checkbox (bulk select missing)'); - } - if (count > 0) { - await boxes.first().check(); - await pw.waitForTimeout(600); - const body = await pw.locator('body').innerText(); - if (!/sequence/i.test(body)) { - findings.push('leads: selecting a row reveals no sequence action'); - } - } + expect(await boxes.count(), 'rows expose a bulk-select checkbox').toBeGreaterThan(0); + + // A lead without an enriched contact has a disabled checkbox — pick one + // that can actually be selected. + const selectable = pw.locator('tbody input[type="checkbox"]:not([disabled])').first(); + expect(await selectable.count(), 'leads: no selectable row').toBeGreaterThan(0); + await selectable.check(); + await pw.waitForTimeout(600); + // Selecting a row must reveal the bulk action bar with the sequence action. + const bulkBar = pw.getByRole('button', { name: /add to sequence/i }); + await expect(bulkBar, 'selecting a row reveals a sequence action').toBeVisible(); + await pw.screenshot({ path: path.join(SHOTS, 'leads-selected.png'), fullPage: true }); }); @@ -125,9 +141,7 @@ test('find page exposes India-first search controls', async ({ page: pw }) => { await pw.goto(BASE + '/find', { waitUntil: 'networkidle' }); await pw.waitForTimeout(700); const body = await pw.locator('body').innerText(); - if (!/india|delhi|mumbai|city|keyword/i.test(body)) { - findings.push('find: no India/city/keyword search controls found'); - } + expect(body, 'find: no India/city/keyword search controls found').toMatch(/india|delhi|mumbai|city|keyword/i); await pw.screenshot({ path: path.join(SHOTS, 'find-controls.png'), fullPage: true }); }); @@ -136,7 +150,7 @@ test('sequences and campaigns render content', async ({ page: pw }) => { await pw.goto(BASE + p, { waitUntil: 'networkidle' }); await pw.waitForTimeout(800); const body = await pw.locator('body').innerText(); - if (body.trim().length < 50) findings.push(`${p}: rendered almost no content`); + expect(body.trim().length, `${p} rendered almost no content`).toBeGreaterThan(50); await pw.screenshot({ path: path.join(SHOTS, `${p.slice(1)}-panel.png`), fullPage: true }); } }); @@ -148,26 +162,22 @@ test('AI Lead Finder: parse an intent into a structured ICP draft', async ({ pag await pw.waitForTimeout(500); const textarea = pw.locator('textarea').first(); - if ((await textarea.count()) === 0) { - findings.push('find: no intent textarea to type into'); - return; - } + expect(await textarea.count(), 'find: no intent textarea to type into').toBeGreaterThan(0); await textarea.fill('dentists in Rohini Delhi with a website and good reviews'); + // The parse button is the primary action (Wand2 "Parse intent" or similar). const parseBtn = pw.getByRole('button', { name: /parse|extract|understand|magic/i }).first(); - if ((await parseBtn.count()) === 0) { - findings.push('find: no parse button found'); - return; - } + expect(await parseBtn.count(), 'find: no parse button found').toBeGreaterThan(0); await parseBtn.click(); + // Wait for the draft fields to populate from the parsed intent. await pw.waitForTimeout(2500); const body = await pw.locator('body').innerText(); - // After a successful parse, niche/area inputs should reflect the intent. const nicheVal = await pw.locator('input').first().inputValue().catch(() => ''); - if (!/dent/i.test(body) && !/dent/i.test(nicheVal)) { - findings.push('find: parse did not populate draft fields with the niche'); - } + expect( + /dent/i.test(body) || /dent/i.test(nicheVal), + 'find: parse did not populate draft fields with the niche', + ).toBe(true); await pw.screenshot({ path: path.join(SHOTS, 'find-parsed.png'), fullPage: true }); }); @@ -176,21 +186,16 @@ test('Sequences: add a step and the step editor appears', async ({ page: pw }) = await pw.waitForTimeout(600); const addBtn = pw.getByRole('button', { name: /add step|new step|\+ step|add email/i }).first(); - if ((await addBtn.count()) === 0) { - findings.push('sequences: no "add step" button'); - return; - } + expect(await addBtn.count(), 'sequences: no "add step" button').toBeGreaterThan(0); const before = await pw.locator('body').innerText(); await addBtn.click(); await pw.waitForTimeout(800); const after = await pw.locator('body').innerText(); + // A new step should add a "Subject line" placeholder to the editor. const subjectInputs = pw.locator('input[placeholder*="Subject" i]'); - if ((await subjectInputs.count()) === 0) { - findings.push('sequences: adding a step produced no subject editor'); - } else if (after === before) { - findings.push('sequences: add-step click changed nothing on the page'); - } + expect(await subjectInputs.count(), 'sequences: adding a step produced no subject editor').toBeGreaterThan(0); + expect(after, 'sequences: add-step click changed nothing on the page').not.toBe(before); await pw.screenshot({ path: path.join(SHOTS, 'sequences-editor.png'), fullPage: true }); }); @@ -199,21 +204,15 @@ test('Settings: open the Add Mailbox modal and its form fields render', async ({ await pw.waitForTimeout(600); const addBtn = pw.getByRole('button', { name: /add mailbox|add smtp|connect|new mailbox/i }).first(); - if ((await addBtn.count()) === 0) { - findings.push('settings: no add-mailbox button'); - return; - } + expect(await addBtn.count(), 'settings: no add-mailbox button').toBeGreaterThan(0); await addBtn.click(); await pw.waitForTimeout(700); + // The modal should expose an email input and a password input. const emailInput = pw.locator('input[placeholder*="@"]'); const pwdInput = pw.locator('input[type="password"]'); - if ((await emailInput.count()) === 0) { - findings.push('settings: add-mailbox modal missing email field'); - } - if ((await pwdInput.count()) === 0) { - findings.push('settings: add-mailbox modal missing password field'); - } + expect(await emailInput.count(), 'settings: add-mailbox modal missing email field').toBeGreaterThan(0); + expect(await pwdInput.count(), 'settings: add-mailbox modal missing password field').toBeGreaterThan(0); await pw.screenshot({ path: path.join(SHOTS, 'settings-modal.png'), fullPage: true }); }); @@ -222,17 +221,16 @@ test('Sidebar: collapse toggle actually collapses the rail', async ({ page: pw } await pw.waitForTimeout(500); const toggle = pw.getByRole('button', { name: /collapse|expand|panel/i }).first(); - if ((await toggle.count()) === 0) { - findings.push('sidebar: no collapse toggle button'); - return; - } + expect(await toggle.count(), 'sidebar: no collapse toggle button').toBeGreaterThan(0); const widthBefore = await pw.locator('aside, nav').first().evaluate((el) => el.getBoundingClientRect().width).catch(() => 0); await toggle.click(); await pw.waitForTimeout(600); const widthAfter = await pw.locator('aside, nav').first().evaluate((el) => el.getBoundingClientRect().width).catch(() => 0); - if (widthBefore > 0 && widthAfter > 0 && Math.abs(widthAfter - widthBefore) < 8) { - findings.push(`sidebar: collapse did not change rail width (${widthBefore}->${widthAfter})`); - } + expect(widthBefore, 'sidebar: rail had no measurable width').toBeGreaterThan(0); + expect( + Math.abs(widthAfter - widthBefore), + `sidebar: collapse did not change rail width (${widthBefore}->${widthAfter})`, + ).toBeGreaterThanOrEqual(8); await pw.screenshot({ path: path.join(SHOTS, 'sidebar-collapsed.png'), fullPage: true }); }); @@ -240,18 +238,76 @@ test('Leads: filters/search narrow the table', async ({ page: pw }) => { await pw.goto(BASE + '/leads', { waitUntil: 'networkidle' }); await pw.waitForTimeout(900); - const search = pw.locator('input[type="search"], input[placeholder*="search" i], input[placeholder*="filter" i]').first(); - if ((await search.count()) === 0) { - findings.push('leads: no search/filter input found'); - return; - } + // The page-level filter box (NOT the global Cmd+K box in the top bar). + const search = pw.getByPlaceholder(/search by business name/i).first(); + expect(await search.count(), 'leads: no search/filter input found').toBeGreaterThan(0); + const rowsBefore = await pw.locator('tbody tr').count(); + expect(rowsBefore, 'leads: table has no rows to filter').toBeGreaterThan(0); + // Real data must be on screen before we can claim the filter narrowed it. + await expect( + pw.getByText(/no leads found/i).first(), + 'leads: table shows no data rows', + ).toHaveCount(0); + await search.fill('zzzz-no-such-lead-zzzz'); - await pw.waitForTimeout(1200); - const rowsAfter = await pw.locator('tbody tr').count(); - if (rowsBefore > 0 && rowsAfter >= rowsBefore) { - findings.push(`leads: search did not narrow results (${rowsBefore} -> ${rowsAfter})`); + await pw.waitForTimeout(1500); + await expect( + pw.getByText(/no leads found/i).first(), + 'leads: search did not narrow results', + ).toBeVisible(); +}); + +test('Leads: global Cmd+K search lands on a filtered table', async ({ page: pw }) => { + await pw.goto(BASE, { waitUntil: 'networkidle' }); + await pw.waitForTimeout(500); + + // The top bar search is the flagship affordance: it must carry the term + // into /leads?search=... and the table must actually filter on it. + const globalSearch = pw.getByLabel('Search leads').first(); + expect(await globalSearch.count(), 'topbar: global search box missing').toBeGreaterThan(0); + await globalSearch.fill('averyunlikelysearchname'); + await globalSearch.press('Enter'); + await pw.waitForURL(/\/leads\?search=/, { timeout: 15000 }); + await pw.waitForTimeout(1500); + + const url = pw.url(); + expect(url, 'topbar: search did not navigate to /leads?search=').toContain('/leads?search='); + // The leads page mirrors the query param back into its own filter box. + const pageSearch = pw.getByPlaceholder(/search by business name/i).first(); + expect( + await pageSearch.inputValue(), + 'leads: ?search= did not seed the filter box', + ).toBe('averyunlikelysearchname'); +}); + +test('Launching a sequence asks for confirmation first', async ({ page: pw }) => { + await pw.goto(BASE + '/sequences', { waitUntil: 'networkidle' }); + await pw.waitForTimeout(800); + + // Open the first sequence — the detail slide-over holds the Launch action. + const row = pw.locator('tbody tr').first(); + if (await row.count() === 0) { + findings.push('sequences: no sequence rows, launch confirmation not exercised'); + return; + } + await row.click(); + await pw.waitForTimeout(800); + + const launchBtn = pw.getByRole('button', { name: /^launch$/i }).first(); + if (await launchBtn.count() === 0) { + findings.push('sequences: no launchable sequence found, launch confirmation not exercised'); + return; } + await launchBtn.click(); + await pw.waitForTimeout(500); + + // Launch sends real cold email — it must be behind a confirmation that + // states the blast radius. + const dialog = pw.getByRole('alertdialog'); + await expect(dialog, 'launch must be confirmed before sending').toBeVisible(); + await expect(dialog).toContainText(/recipient/i); + await expect(dialog).toContainText(/step/i); }); test('mobile viewport renders without horizontal overflow', async ({ page: pw }) => { @@ -259,13 +315,39 @@ test('mobile viewport renders without horizontal overflow', async ({ page: pw }) for (const p of ['/', '/leads']) { await pw.goto(BASE + p, { waitUntil: 'networkidle' }); await pw.waitForTimeout(600); - const overflow = await pw.evaluate(() => - document.documentElement.scrollWidth - document.documentElement.clientWidth); - if (overflow > 4) findings.push(`mobile ${p}: horizontal overflow ${overflow}px`); + // `body` is overflow-hidden, so the document never scrolls horizontally — + // measuring it was a guaranteed false negative. Measure the main content + // column instead, which is the thing that actually overflows. + const overflow = await pw.evaluate(() => { + const main = document.querySelector('main'); + if (!main) return -1; + return main.scrollWidth - main.clientWidth; + }); + expect(overflow, `mobile ${p}: main column overflows horizontally`).toBeLessThanOrEqual(4); await pw.screenshot({ path: path.join(SHOTS, `mobile-${p === '/' ? 'dashboard' : 'leads'}.png`), fullPage: true }); } }); +test('mobile: the navigation drawer reaches every page', async ({ page: pw }) => { + await pw.setViewportSize({ width: 390, height: 844 }); + await pw.goto(BASE, { waitUntil: 'networkidle' }); + await pw.waitForTimeout(500); + + const menu = pw.getByRole('button', { name: /open navigation/i }).first(); + expect(await menu.count(), 'mobile: no hamburger button').toBeGreaterThan(0); + await menu.click(); + await pw.waitForTimeout(400); + + const drawer = pw.getByRole('dialog', { name: /navigation/i }); + await expect(drawer, 'mobile: drawer did not open').toBeVisible(); + const hrefs = await drawer.locator('a[href^="/"]').evaluateAll((els) => + els.map((e) => e.getAttribute('href')), + ); + for (const p of NAV_PAGES) { + expect(hrefs, `mobile: drawer cannot reach ${p.path}`).toContain(p.path); + } +}); + test.afterAll(() => { fs.writeFileSync(path.join(SHOTS, 'findings.json'), JSON.stringify(findings, null, 2)); if (findings.length) console.log('\nFINDINGS:\n' + findings.map((f) => ' - ' + f).join('\n')); diff --git a/dashboard/next.config.mjs b/dashboard/next.config.mjs index cde64d9..acefee7 100644 --- a/dashboard/next.config.mjs +++ b/dashboard/next.config.mjs @@ -1,8 +1,28 @@ /** @type {import('next').NextConfig} */ + +// Type-check and lint run during `next build` so a type error can never ship +// silently again. Local dev can opt out with SKIP_TYPECHECK=1 (or +// SKIP_LINT=1) when a backend contract is still in flux. +const skipTypecheck = process.env.SKIP_TYPECHECK === '1'; +const skipLint = process.env.SKIP_LINT === '1'; + +// Where the browser's /api/* calls are proxied to. Keep this in sync with the +// backend's listen port; it is read at RUNTIME (not baked into the bundle), +// which is what makes LAN/Docker installs work. +const API_PROXY_URL = process.env.API_PROXY_URL || 'http://localhost:8000/api'; + const nextConfig = { output: 'standalone', - eslint: { ignoreDuringBuilds: true }, - typescript: { ignoreBuildErrors: true } + eslint: { ignoreDuringBuilds: skipLint }, + typescript: { ignoreBuildErrors: skipTypecheck }, + async rewrites() { + return [ + { + source: '/api/:path*', + destination: `${API_PROXY_URL}/:path*`, + }, + ]; + }, }; export default nextConfig; diff --git a/dashboard/package.json b/dashboard/package.json index 4d827b9..526217a 100644 --- a/dashboard/package.json +++ b/dashboard/package.json @@ -6,7 +6,9 @@ "dev": "next dev", "build": "next build", "start": "next start", - "lint": "next lint" + "lint": "next lint", + "typecheck": "tsc --noEmit", + "test:e2e": "playwright test" }, "dependencies": { "@hookform/resolvers": "^3.3.4", diff --git a/dashboard/playwright.config.js b/dashboard/playwright.config.js index 57ce9e1..253b9bd 100644 --- a/dashboard/playwright.config.js +++ b/dashboard/playwright.config.js @@ -17,7 +17,17 @@ const config = { video: 'off', }, projects: [{ name: 'chromium', use: { browserName: 'chromium' } }], - webServer: null, + // The suite has to boot the app itself, otherwise it silently tests nothing. + // CI builds first (the build now type-checks) and starts the production + // server; locally `next dev` is used so no build step is required. + webServer: { + command: process.env.CI ? 'npm run build && npm run start' : 'npm run dev', + url: 'http://127.0.0.1:3000', + reuseExistingServer: !process.env.CI, + timeout: 180_000, + stdout: 'pipe', + stderr: 'pipe', + }, }; module.exports = config; diff --git a/dashboard/src/app/layout.tsx b/dashboard/src/app/layout.tsx index b48a84c..10ce000 100644 --- a/dashboard/src/app/layout.tsx +++ b/dashboard/src/app/layout.tsx @@ -4,6 +4,7 @@ import "./globals.css"; import { Sidebar } from "@/components/Sidebar"; import { TopBar } from "@/components/TopBar"; import { FirstRunWizard } from "@/components/FirstRunWizard"; +import { Toaster } from "@/components/Toaster"; const inter = Inter({ subsets: ["latin"] }); @@ -23,10 +24,12 @@ export default function RootLayout({
-
{children}
+
{children}
{/* First-run setup: shows only while a required connection is missing */} + {/* App-wide notifications (replaces blocking browser dialogs) */} + ); diff --git a/dashboard/src/app/leads/page.tsx b/dashboard/src/app/leads/page.tsx index f9ea160..6305c4c 100644 --- a/dashboard/src/app/leads/page.tsx +++ b/dashboard/src/app/leads/page.tsx @@ -1,6 +1,7 @@ "use client"; -import { useState, useEffect, useRef, useCallback } from "react"; +import { Suspense, useState, useEffect, useRef, useCallback } from "react"; +import { useSearchParams } from "next/navigation"; import { Users, Search, @@ -35,14 +36,60 @@ import { } from "lucide-react"; import { api, LeadItem, CampaignItem, LeadDetailResponse, SavedView } from "@/lib/api"; import { TrustBadge } from "@/components/IndiaKit"; +import { toast } from "@/components/Toaster"; +/** Rows per page — the backend paginates on the same number. */ +const PAGE_SIZE = 20; + +function LeadsSkeleton() { + return ( +
+
+

+ Lead Database +

+

Loading your lead database…

+
+
+ +
+
+ ); +} + +/** + * The page reads `?search=` / `?industry=` from the URL, which is what the + * global Cmd+K search lands on. `useSearchParams` forces the surrounding + * Suspense boundary (see the default export) so the route stays statically + * renderable. + */ export default function LeadsPage() { + return ( + }> + + + ); +} + +function LeadsPageInner() { + const searchParams = useSearchParams(); + const urlSearch = searchParams.get("search") ?? ""; + const urlIndustry = searchParams.get("industry") ?? ""; + const [leads, setLeads] = useState([]); const [total, setTotal] = useState(0); const [page, setPage] = useState(1); - const [search, setSearch] = useState(""); - const [industryFilter, setIndustryFilter] = useState(""); + const [search, setSearch] = useState(urlSearch); + const [industryFilter, setIndustryFilter] = useState(urlIndustry); const [hasEmailFilter, setHasEmailFilter] = useState("all"); + + // Pagination is driven by the total the API reports — a filtered last page + // is legitimately shorter than PAGE_SIZE, so row count can't drive it. + const totalPages = Math.max(1, Math.ceil(total / PAGE_SIZE)); + const firstRowIndex = total === 0 ? 0 : (page - 1) * PAGE_SIZE + 1; + const lastRowIndex = total === 0 ? 0 : Math.min(page * PAGE_SIZE, total); + const hasActiveFilters = + Boolean(search.trim()) || Boolean(industryFilter.trim()) || hasEmailFilter !== "all"; const [loading, setLoading] = useState(true); const [enrichingId, setEnrichingId] = useState(null); const [bulkEnrichLoading, setBulkEnrichLoading] = useState(false); @@ -96,9 +143,27 @@ export default function LeadsPage() { const contactIdFor = (lead: LeadItem): number | null => lead.primary_contact?.id ?? lead.contacts?.[0]?.id ?? null; - const selectableIds = leads.map(contactIdFor); - const selectableCount = selectableIds.filter((id): id is number => id !== null).length; + // Selectable ids are PAGE-scoped, while `selected` is global (it can hold + // picks from earlier pages). Keep both views straight: the header checkbox + // reflects the current page only, and toggling it merges into / removes from + // the global set instead of replacing it. + const selectableIds = leads + .map(contactIdFor) + .filter((id): id is number => id !== null); + const selectableCount = selectableIds.length; const selectableLoaded = !loading && leads.length > 0; + const selectedOnPage = selectableIds.filter((id) => selected.has(id)); + const allOnPageSelected = selectableCount > 0 && selectedOnPage.length === selectableCount; + const someOnPageSelected = selectedOnPage.length > 0 && !allOnPageSelected; + + // The header checkbox is "indeterminate" when only part of the page is + // picked — the DOM property can't be set declaratively. + const headerCheckboxRef = useRef(null); + useEffect(() => { + if (headerCheckboxRef.current) { + headerCheckboxRef.current.indeterminate = someOnPageSelected; + } + }, [someOnPageSelected]); const toggleRow = (lead: LeadItem) => { const cid = contactIdFor(lead); @@ -195,11 +260,14 @@ export default function LeadsPage() { }, [loadSavedViews]); const toggleAll = () => { - setSelected((prev) => - prev.size === selectableCount - ? new Set() - : new Set(selectableIds.filter((id): id is number => id !== null)), - ); + setSelected((prev) => { + const next = new Set(prev); + // Union (never replace): picks made on other pages must survive. + const everySelected = selectableIds.length > 0 && selectableIds.every((id) => next.has(id)); + if (everySelected) selectableIds.forEach((id) => next.delete(id)); + else selectableIds.forEach((id) => next.add(id)); + return next; + }); }; const openSequencePicker = async () => { @@ -228,29 +296,46 @@ export default function LeadsPage() { } }; - const fetchLeads = async () => { - try { - setLoading(true); - const emailFilterParam = hasEmailFilter === "true" ? true : hasEmailFilter === "false" ? false : undefined; - const res = await api.getLeads({ - page, - limit: 20, - search: search || undefined, - industry: industryFilter || undefined, - has_email: emailFilterParam, - }); - setLeads(res.leads); - setTotal(res.total); - } catch (err) { - console.warn("Could not fetch live leads, using demo fallback:", err); - } finally { - setLoading(false); - } - }; + // Two in-flight requests can resolve out of order (a slow page-1 response + // landing after a fast page-2 response). A monotonically increasing token + // makes sure only the newest response is allowed to paint. + const requestToken = useRef(0); + + const fetchLeads = useCallback( + async (args: { page: number; search: string; industry: string; hasEmail?: boolean }) => { + const token = ++requestToken.current; + try { + setLoading(true); + const res = await api.getLeads({ + page: args.page, + limit: PAGE_SIZE, + search: args.search || undefined, + industry: args.industry || undefined, + has_email: args.hasEmail, + }); + if (token !== requestToken.current) return; // a newer request won + setLeads(res.leads); + setTotal(res.total); + } catch (err) { + if (token !== requestToken.current) return; + console.warn("Could not fetch live leads:", err); + toast.error("Could not load leads", "Check that the backend is reachable and try again."); + } finally { + if (token === requestToken.current) setLoading(false); + } + }, + [], + ); + + const emailFilterParam = + hasEmailFilter === "true" ? true : hasEmailFilter === "false" ? false : undefined; useEffect(() => { - fetchLeads(); - }, [page, industryFilter, hasEmailFilter]); + fetchLeads({ page, search, industry: industryFilter, hasEmail: emailFilterParam }); + // `search` / `industry` are intentionally excluded: they have their own + // debounced effect so typing doesn't fire a request per keystroke. + // eslint-disable-next-line react-hooks/exhaustive-deps + }, [page, hasEmailFilter, fetchLeads]); // Live search: debounced so typing doesn't fire a request per keystroke, but // without requiring the user to press Enter (Apollo-style instant filter). @@ -262,15 +347,30 @@ export default function LeadsPage() { } const t = setTimeout(() => { setPage(1); - fetchLeads(); + fetchLeads({ page: 1, search, industry: industryFilter, hasEmail: emailFilterParam }); }, 400); return () => clearTimeout(t); - }, [search]); + // eslint-disable-next-line react-hooks/exhaustive-deps + }, [search, industryFilter]); + + // Sync the URL (?search= / ?industry=) into the filters. This is what makes + // the global Cmd+K search land on an actually filtered table. Changing the + // URL resets to the first page. + useEffect(() => { + setSearch((prev) => (prev === urlSearch ? prev : urlSearch)); + setIndustryFilter((prev) => (prev === urlIndustry ? prev : urlIndustry)); + setPage(1); + }, [urlSearch, urlIndustry]); + + // Deleting the last row of the last page must not strand us on an empty page. + useEffect(() => { + if (page > totalPages) setPage(totalPages); + }, [page, totalPages]); const handleSearch = (e: React.FormEvent) => { e.preventDefault(); setPage(1); - fetchLeads(); + fetchLeads({ page: 1, search, industry: industryFilter, hasEmail: emailFilterParam }); }; const handleEnrichSingle = async (businessId: number) => { @@ -280,10 +380,13 @@ export default function LeadsPage() { const emailCount = res.emails_found || 0; const phoneCount = res.phones_found || 0; const verifiedCount = res.verified_emails || 0; - alert(`Multi-Source Search Complete!\n• Emails Found: ${emailCount} (${verifiedCount} verified deliverable)\n• Discovered Phone Numbers: ${phoneCount}\n• Discovered Website: ${res.website || "None"}`); - fetchLeads(); + toast.success( + "Multi-source search complete", + `Emails found: ${emailCount} (${verifiedCount} verified deliverable) • Phone numbers: ${phoneCount} • Website: ${res.website || "None"}`, + ); + fetchLeads({ page, search, industry: industryFilter, hasEmail: emailFilterParam }); } catch (err: any) { - alert(`Search notice: ${err.message}`); + toast.error("Search failed", err.message || "Could not run the contact search."); } finally { setEnrichingId(null); } @@ -293,10 +396,16 @@ export default function LeadsPage() { try { setBulkEnrichLoading(true); const res = await api.enrichAll(25); - alert(res.message || "Deep web contact enrichment started in background across DuckDuckGo & websites!"); - setTimeout(() => fetchLeads(), 3000); + toast.info( + "Deep web enrichment started", + res.message || "Running in the background across DuckDuckGo & target websites.", + ); + setTimeout( + () => fetchLeads({ page, search, industry: industryFilter, hasEmail: emailFilterParam }), + 3000, + ); } catch (err: any) { - alert(`Failed to trigger bulk enrichment: ${err.message}`); + toast.error("Bulk enrichment failed", err.message || "Could not trigger bulk enrichment."); } finally { setBulkEnrichLoading(false); } @@ -306,9 +415,10 @@ export default function LeadsPage() { if (!confirm("Are you sure you want to remove this lead?")) return; try { await api.deleteLead(id); - fetchLeads(); + toast.success("Lead deleted", "The lead was removed from your database."); + fetchLeads({ page, search, industry: industryFilter, hasEmail: emailFilterParam }); } catch (err: any) { - alert(`Failed to delete lead: ${err.message}`); + toast.error("Could not delete lead", err.message || "The lead is still in your database."); } }; @@ -319,9 +429,10 @@ export default function LeadsPage() { await api.createLead(newLead); setShowAddModal(false); setNewLead({ name: "", phone: "", email: "", website: "", address: "", industry: "Real Estate Agency" }); - fetchLeads(); + toast.success("Lead added", `${newLead.name} is now in your lead database.`); + fetchLeads({ page: 1, search, industry: industryFilter, hasEmail: emailFilterParam }); } catch (err: any) { - alert(`Failed to add lead: ${err.message}`); + toast.error("Could not add lead", err.message || "Please check the details and try again."); } }; @@ -331,10 +442,13 @@ export default function LeadsPage() { try { setSyncingTwenty(true); const res = await api.exportTwentyCrm(syncWebhookUrl); - alert(`Sync successful! ${res.synced_companies || 0} companies pushed to target webhook.`); + toast.success( + "Sync successful", + `${res.synced_companies || 0} companies pushed to your webhook.`, + ); setShowSyncModal(false); } catch (err: any) { - alert(`Sync notice: ${err.message}`); + toast.error("Sync failed", err.message || "Could not reach the webhook."); } finally { setSyncingTwenty(false); } @@ -346,12 +460,15 @@ export default function LeadsPage() { try { setImportingCsv(true); const res = await api.importCsv(importCsvText.trim()); - alert(`Import Successful!\n• Imported Leads: ${res.imported}\n• Skipped: ${res.skipped}\n• Total Processed: ${res.total_rows_processed}`); + toast.success( + "Import complete", + `Imported ${res.imported} lead(s) • Skipped ${res.skipped} • ${res.total_rows_processed} rows processed`, + ); setShowImportModal(false); setImportCsvText(""); - fetchLeads(); + fetchLeads({ page: 1, search, industry: industryFilter, hasEmail: emailFilterParam }); } catch (err: any) { - alert(`CSV Import failed: ${err.message}`); + toast.error("CSV import failed", err.message || "Could not read that CSV."); } finally { setImportingCsv(false); } @@ -374,7 +491,7 @@ export default function LeadsPage() { const res = await api.getIcebreaker(leadId); setActiveIcebreaker(res); } catch (err: any) { - alert(`Icebreaker generation notice: ${err.message}`); + toast.error("Could not generate an icebreaker", err.message || "Try again in a moment."); } finally { setLoadingIcebreakerId(null); } @@ -387,7 +504,7 @@ export default function LeadsPage() { const res = await api.checkSpam(spamSubject, spamBody); setSpamResult(res); } catch (err: any) { - alert(`Spam analysis notice: ${err.message}`); + toast.error("Spam analysis failed", err.message || "Try again in a moment."); } finally { setCheckingSpam(false); } @@ -490,9 +607,18 @@ export default function LeadsPage() {
+ setIndustryFilter(e.target.value)} + placeholder="Industry…" + aria-label="Filter by industry" + className="h-10 px-3 rounded-xl border border-slate-200 bg-white text-xs font-medium text-slate-700 placeholder:text-slate-400 focus:outline-none focus:ring-2 focus:ring-blue-500/20 focus:border-blue-500 w-full md:w-40" + /> @@ -857,21 +1022,28 @@ export default function LeadsPage() {
- {/* Pagination Bar */} -
- Showing {leads.length} of {total} leads + {/* Pagination Bar — driven by the total the API reports, not by how + many rows happened to come back (a filtered last page is shorter). */} +
+ + {total === 0 + ? "No leads" + : `Showing ${firstRowIndex}–${lastRowIndex} of ${total} lead${total === 1 ? "" : "s"}`} +
- Page {page} + + Page {page} of {totalPages} +
+ + {/* Launch confirmation — this sends real cold email, so the operator + gets to see the blast radius before anything leaves the building. */} + {confirmLaunch && ( +
+
setConfirmLaunch(false)} /> +
+
+ + + +
+

+ Launch this sequence now? +

+

+ Real cold email starts going out to your recipients. This cannot be + unsent. +

+
+
+ +
+
+ + Recipients + +
{targets.length}
+
email{targets.length === 1 ? "" : "s"} in this blast
+
+
+ + Steps + +
{stats.length}
+
+ email{stats.length === 1 ? "" : "s"} in the sequence +
+
+
+ + {targets.length === 0 && ( +

+ This sequence has no recipients — add contacts before launching. +

+ )} + +
+ + +
+
+
+ )}
); } diff --git a/dashboard/src/components/Sidebar.tsx b/dashboard/src/components/Sidebar.tsx index 5ba7978..dae40d4 100644 --- a/dashboard/src/components/Sidebar.tsx +++ b/dashboard/src/components/Sidebar.tsx @@ -13,13 +13,14 @@ import { Sparkles, PanelLeftClose, PanelLeftOpen, - MailCheck, BookOpen, Inbox, Building2, Activity, Plug, + X, } from "lucide-react"; +import { setMobileNavOpen, useMobileNavOpen } from "@/components/navState"; type NavItem = { href: string; label: string; icon: typeof LayoutDashboard }; type NavGroup = { title: string; items: NavItem[] }; @@ -62,9 +63,97 @@ const NAV_GROUPS: NavGroup[] = [ const COLLAPSE_KEY = "leadforge.sidebar.collapsed"; -export function Sidebar() { +function useIsActive() { const pathname = usePathname(); + return (href: string) => (href === "/" ? pathname === "/" : pathname.startsWith(href)); +} + +/** + * The nav list itself. Shared by the desktop rail and the mobile drawer so + * both expose exactly the same destinations. + */ +function NavList({ + collapsed, + onNavigate, +}: { + collapsed: boolean; + onNavigate?: () => void; +}) { + const isActive = useIsActive(); + return ( + + ); +} + +function BrandMark({ collapsed }: { collapsed: boolean }) { + return ( +
+
+ LF +
+ {!collapsed && ( +
+

+ LeadForge +

+ + Sales Platform + +
+ )} +
+ ); +} + +function PromoCard() { + return ( +
+
+ + Live lead data +
+

+ Scraped fresh, verified, India-ready — no per-lead credits. +

+
+ ); +} + +export function Sidebar() { const [collapsed, setCollapsed] = useState(false); + const mobileOpen = useMobileNavOpen(); // Remember the operator's choice: collapsing is a preference, not a // per-session accident. @@ -87,97 +176,98 @@ export function Sidebar() { }); }; - const isActive = (href: string) => - href === "/" ? pathname === "/" : pathname.startsWith(href); + // Escape closes the drawer. + useEffect(() => { + if (!mobileOpen) return; + const onKey = (e: KeyboardEvent) => { + if (e.key === "Escape") setMobileNavOpen(false); + }; + window.addEventListener("keydown", onKey); + return () => window.removeEventListener("keydown", onKey); + }, [mobileOpen]); + + // A tap on the dimmed backdrop closes the drawer. + const closeMobile = () => setMobileNavOpen(false); return ( -
-
- {/* Brand + collapse toggle */} -
-
-
- LF -
- {!collapsed && ( -
-

- LeadForge -

- - Sales Platform - -
- )} + <> + {/* Desktop rail — hidden below md so a phone keeps the full width for content */} + - {!collapsed && ( -
-
-
- - Live lead data + {/* Mobile drawer — every page reachable from a 390px phone */} + {mobileOpen && ( +
+ )} -
+ ); } diff --git a/dashboard/src/components/Toaster.tsx b/dashboard/src/components/Toaster.tsx new file mode 100644 index 0000000..bbea047 --- /dev/null +++ b/dashboard/src/components/Toaster.tsx @@ -0,0 +1,138 @@ +"use client"; + +import * as Toast from "@radix-ui/react-toast"; +import { useSyncExternalStore } from "react"; +import { CheckCircle2, AlertCircle, Info, X } from "lucide-react"; + +/** + * Toaster — the app-wide notification surface. + * + * The native browser alert blocks the main thread, looks like a 1998 dialog + * and cannot be styled; Radix Toast (already a dependency) gives us a + * dismissible, screen-reader friendly, non-blocking notification. We keep a + * tiny external store so any module can call `toast.success("...")` without + * threading a hook through every component. + */ + +export type ToastKind = "success" | "error" | "info"; + +export interface ToastItem { + id: string; + kind: ToastKind; + title: string; + description?: string; +} + +type Listener = () => void; + +const listeners = new Set(); +let items: ToastItem[] = []; +let counter = 0; + +function emit() { + for (const listener of listeners) listener(); +} + +function subscribe(listener: Listener) { + listeners.add(listener); + return () => { + listeners.delete(listener); + }; +} + +// A stable reference so useSyncExternalStore never re-renders forever. +const getSnapshot = () => items; + +function dismiss(id: string) { + const next = items.filter((t) => t.id !== id); + if (next.length === items.length) return; + items = next; + emit(); +} + +function push(kind: ToastKind, title: string, description?: string) { + counter += 1; + const item: ToastItem = { + id: `lf-toast-${Date.now()}-${counter}`, + kind, + title, + ...(description ? { description } : {}), + }; + items = [...items, item]; + emit(); +} + +/** Imperative API used across the app. */ +export const toast = { + success: (title: string, description?: string) => push("success", title, description), + error: (title: string, description?: string) => push("error", title, description), + info: (title: string, description?: string) => push("info", title, description), + dismiss, +}; + +const KIND_META: Record< + ToastKind, + { icon: typeof CheckCircle2; iconClass: string; ringClass: string } +> = { + success: { + icon: CheckCircle2, + iconClass: "text-emerald-500", + ringClass: "border-emerald-200", + }, + error: { + icon: AlertCircle, + iconClass: "text-red-500", + ringClass: "border-red-200", + }, + info: { + icon: Info, + iconClass: "text-blue-500", + ringClass: "border-blue-200", + }, +}; + +export function Toaster() { + const toasts = useSyncExternalStore(subscribe, getSnapshot, getSnapshot); + + return ( + + {toasts.map((t) => { + const meta = KIND_META[t.kind]; + const Icon = meta.icon; + return ( + { + if (!open) dismiss(t.id); + }} + className={`pointer-events-auto flex items-start gap-3 rounded-xl border bg-white p-4 shadow-lg ${ + meta.ringClass + } data-[state=open]:animate-in data-[state=open]:slide-in-from-bottom-2 data-[state=open]:fade-in-0 data-[state=closed]:animate-out data-[state=closed]:fade-out-80 data-[state=closed]:slide-out-to-right-full`} + > + +
+ + {t.title} + + {t.description && ( + + {t.description} + + )} +
+ + + + + ); + })} + + + + ); +} diff --git a/dashboard/src/components/TopBar.tsx b/dashboard/src/components/TopBar.tsx index b6b62d4..c907ed9 100644 --- a/dashboard/src/components/TopBar.tsx +++ b/dashboard/src/components/TopBar.tsx @@ -2,7 +2,8 @@ import { useRouter } from "next/navigation"; import { useEffect, useRef, useState } from "react"; -import { Search, Command, Sparkles } from "lucide-react"; +import { Search, Command, Sparkles, Menu } from "lucide-react"; +import { toggleMobileNav } from "@/components/navState"; /** * TopBar — Apollo's most recognisable affordance is the global search that @@ -33,7 +34,17 @@ export function TopBar() { }; return ( -
+
+ {/* Mobile: the rail is hidden below md, so the nav lives behind this button */} + +
setQ(e.target.value)} placeholder="Search leads, businesses, phone numbers…" - className="w-full h-9 pl-9 pr-20 rounded-lg border border-slate-200 bg-slate-50 text-[13px] text-slate-800 placeholder:text-slate-400 focus:outline-none focus:ring-2 focus:ring-blue-500/20 focus:bg-white focus:border-blue-300" + aria-label="Search leads" + className="w-full h-9 pl-9 pr-3 sm:pr-20 rounded-lg border border-slate-200 bg-slate-50 text-[13px] text-slate-800 placeholder:text-slate-400 focus:outline-none focus:ring-2 focus:ring-blue-500/20 focus:bg-white focus:border-blue-300" /> K @@ -56,4 +68,4 @@ export function TopBar() {
); -} \ No newline at end of file +} diff --git a/dashboard/src/components/navState.ts b/dashboard/src/components/navState.ts new file mode 100644 index 0000000..91ef528 --- /dev/null +++ b/dashboard/src/components/navState.ts @@ -0,0 +1,43 @@ +"use client"; + +import { useSyncExternalStore } from "react"; + +/** + * Shared state for the mobile navigation drawer. + * + * The drawer lives in the Sidebar but is opened from the TopBar (the + * hamburger), so the two sibling components need one source of truth. A tiny + * external store keeps them decoupled without pulling a context provider into + * the server-rendered root layout. + */ + +let open = false; +const listeners = new Set<() => void>(); + +function emit() { + for (const listener of listeners) listener(); +} + +export function setMobileNavOpen(next: boolean) { + if (open === next) return; + open = next; + emit(); +} + +export function toggleMobileNav() { + setMobileNavOpen(!open); +} + +function subscribe(listener: () => void) { + listeners.add(listener); + return () => { + listeners.delete(listener); + }; +} + +const getSnapshot = () => open; +const getServerSnapshot = () => false; + +export function useMobileNavOpen() { + return useSyncExternalStore(subscribe, getSnapshot, getServerSnapshot); +} diff --git a/dashboard/src/lib/api.ts b/dashboard/src/lib/api.ts index 1806f74..4456789 100644 --- a/dashboard/src/lib/api.ts +++ b/dashboard/src/lib/api.ts @@ -1,4 +1,24 @@ -const API_BASE = (process.env.NEXT_PUBLIC_API_URL || 'http://localhost:8000/api/v1').replace(/\/$/, ''); +/** + * Same-origin by default. The dashboard is frequently deployed on a LAN box or + * inside Docker where `localhost:8000` is NOT the backend — a build-time + * absolute URL baked into the browser bundle breaks those installs (the + * browser resolves "localhost" to whichever machine is running the browser). + * + * With the default `/api/v1` the browser calls the Next server, and + * `next.config.mjs` proxies `/api/*` to the real backend (see `API_PROXY_URL`). + * An explicit `NEXT_PUBLIC_API_URL` still wins for people who want direct + * CORS calls — unless it points at localhost, which is exactly the value that + * gets baked in by default and is the one that breaks remote installs. + */ +const CONFIGURED_API_URL = (process.env.NEXT_PUBLIC_API_URL || '').trim(); +const IS_LOCALHOST_URL = /^https?:\/\/(localhost|127\.0\.0\.1|0\.0\.0\.0)(:|\/|$)/i.test(CONFIGURED_API_URL); +const API_BASE = (IS_LOCALHOST_URL || !CONFIGURED_API_URL ? '/api/v1' : CONFIGURED_API_URL).replace(/\/$/, ''); + +/** Requests are abandoned after this long so a hung socket can't wedge a page. */ +const DEFAULT_TIMEOUT_MS = 30_000; +/** Extra attempts after the first one (network errors and 5xx only). */ +const DEFAULT_RETRIES = 1; +const RETRY_BASE_DELAY_MS = 400; // --- Apollo-parity types ----------------------------------------------------- @@ -430,30 +450,138 @@ export interface DiscoveryJob { finished_at: string | null; } -async function request(path: string, options: RequestInit = {}): Promise { +/** Error carrying the HTTP status so callers can branch on 404/5xx. */ +export class ApiError extends Error { + status: number; + constructor(message: string, status: number) { + super(message); + this.name = 'ApiError'; + this.status = status; + } +} + +export interface RequestOptions extends RequestInit { + /** Abort the request after N ms (default 30s). */ + timeoutMs?: number; + /** Extra attempts after the first one (default 1 => 2 attempts total). */ + retries?: number; +} + +const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms)); + +/** 5xx and network-level failures are worth a second try; 4xx are not. */ +const isRetryableStatus = (status: number) => status >= 500; + +/** + * The single HTTP door to the backend. + * + * - every request has a hard timeout, so a half-open socket can't leave a + * page spinning forever; + * - transient failures (network errors, 5xx) are retried once with backoff; + * - the caller may pass its own `signal` to cancel in-flight work, which is + * what the leads page uses to drop stale responses. + */ +async function request(path: string, options: RequestOptions = {}): Promise { + const { timeoutMs = DEFAULT_TIMEOUT_MS, retries = DEFAULT_RETRIES, ...init } = options; const url = `${API_BASE}${path.startsWith('/') ? path : `/${path}`}`; - try { - const res = await fetch(url, { - ...options, - headers: { - 'Content-Type': 'application/json', - ...(options.headers || {}), - }, - }); - if (!res.ok) { - const errText = await res.text(); - let msg = `HTTP error ${res.status}`; - try { - const parsed = JSON.parse(errText); - msg = parsed.detail || parsed.message || msg; - } catch {} - throw new Error(msg); + + let lastError: unknown = new Error(`Request failed: ${path}`); + + for (let attempt = 0; attempt <= retries; attempt++) { + const controller = new AbortController(); + let timedOut = false; + + // The caller may cancel us mid-flight (e.g. a newer search superseded an + // older one). Never swallow that — it must surface as an abort error. + const externalSignal = init.signal ?? null; + const forwardAbort = () => controller.abort(); + if (externalSignal) { + if (externalSignal.aborted) controller.abort(); + else externalSignal.addEventListener('abort', forwardAbort); } - return res.json(); - } catch (err: any) { - console.error(`API Error on ${path}:`, err); - throw err; + + const timer = setTimeout(() => { + timedOut = true; + controller.abort(); + }, timeoutMs); + + try { + const headers = new Headers(init.headers as HeadersInit | undefined); + if (!headers.has('Content-Type')) headers.set('Content-Type', 'application/json'); + + const res = await fetch(url, { + ...init, + signal: controller.signal, + headers, + }); + + if (!res.ok) { + const errText = await res.text(); + let msg = `HTTP error ${res.status}`; + try { + const parsed = JSON.parse(errText); + msg = parsed.detail || parsed.message || msg; + } catch {} + const apiErr = new ApiError(msg, res.status); + if (isRetryableStatus(res.status) && attempt < retries) { + lastError = apiErr; + } else { + throw apiErr; + } + } else { + return (await res.json()) as T; + } + } catch (err: any) { + if (timedOut) { + lastError = new ApiError(`Request timed out after ${timeoutMs}ms`, 408); + } else if (err instanceof ApiError) { + // Already classified above (non-retryable status) — let it through. + throw err; + } else if (err?.name === 'AbortError' && externalSignal?.aborted) { + // Deliberate cancellation — propagate without retrying. + throw err; + } else { + // Network-level failure (DNS, refused connection, socket reset). + lastError = err instanceof Error ? err : new Error(String(err)); + console.error(`API network error on ${path}:`, lastError); + } + if (attempt >= retries) throw lastError; + } finally { + // Always clear the timer, even on the retry path. + clearTimeout(timer); + if (externalSignal) externalSignal.removeEventListener('abort', forwardAbort); + } + + // Backoff before the next attempt. + await sleep(RETRY_BASE_DELAY_MS * (attempt + 1)); } + + throw lastError; +} + +export interface LeadsResponse { + total: number; + page: number; + limit: number; + leads: LeadItem[]; +} + +export interface CreateLeadResponse { + id: number; + name?: string; + status?: string; + [key: string]: unknown; +} + +export interface CreateMailboxResponse { + id: number; + email: string; + status: string; +} + +export interface CampaignActionResponse { + status: string; + message: string; } export const api = { @@ -468,11 +596,11 @@ export const api = { if (params.search) q.set('search', params.search); if (params.industry) q.set('industry', params.industry); if (params.has_email !== undefined) q.set('has_email', String(params.has_email)); - return request<{ total: number; page: number; limit: number; leads: LeadItem[] }>(`/leads?${q.toString()}`); + return request(`/leads?${q.toString()}`); }, createLead: (data: { name: string; phone?: string; email?: string; website?: string; address?: string; industry?: string }) => - request('/leads', { method: 'POST', body: JSON.stringify(data) }), - deleteLead: (id: number) => request(`/leads/${id}`, { method: 'DELETE' }), + request('/leads', { method: 'POST', body: JSON.stringify(data) }), + deleteLead: (id: number) => request<{ status?: string; id?: number }>(`/leads/${id}`, { method: 'DELETE' }), exportCsvUrl: `${API_BASE}/leads/export/csv`, exportTwentyCrmUrl: `${API_BASE}/leads/export/twenty-crm`, exportTwentyCrm: (webhook_url?: string) => @@ -535,8 +663,8 @@ export const api = { ), removeCampaignTarget: (campaignId: number, contactId: number) => request(`/campaigns/${campaignId}/targets/${contactId}`, { method: 'DELETE' }), - launchCampaign: (id: number) => request(`/campaigns/${id}/launch`, { method: 'POST' }), - pauseCampaign: (id: number) => request(`/campaigns/${id}/pause`, { method: 'POST' }), + launchCampaign: (id: number) => request(`/campaigns/${id}/launch`, { method: 'POST' }), + pauseCampaign: (id: number) => request(`/campaigns/${id}/pause`, { method: 'POST' }), previewTemplate: (subject: string, body: string) => request<{ sample_variables_used: any; rendered_subject: string; rendered_body: string }>('/campaigns/preview', { method: 'POST', @@ -546,7 +674,7 @@ export const api = { // Mailboxes getMailboxes: () => request('/mailboxes'), createMailbox: (data: { email: string; password: string; smtp_host: string; smtp_port?: number; first_name?: string; last_name?: string; daily_limit?: number }) => - request('/mailboxes', { method: 'POST', body: JSON.stringify(data) }), + request('/mailboxes', { method: 'POST', body: JSON.stringify(data) }), testMailbox: (id: number) => request<{ success: boolean; message: string }>(`/mailboxes/${id}/test`, { method: 'POST' }), deleteMailbox: (id: number) => request(`/mailboxes/${id}`, { method: 'DELETE' }), diff --git a/docker-compose.yml b/docker-compose.yml index 600483f..e9065c3 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -12,6 +12,12 @@ # Before exposing the API beyond localhost, generate a real secret: # SECRET_KEY=$(openssl rand -hex 32) docker compose up -d # and set ENV=production + an explicit CORS_ORIGINS (see .env.example). +# +# The dashboard talks to the API through a same-origin Next.js rewrite, so the +# browser only ever calls localhost:3000. That is what makes the stack work +# from another machine on the LAN — the old build-time NEXT_PUBLIC_API_URL +# baked "localhost:8000" into the client bundle, so every visitor's browser +# tried to reach *their own* machine. services: postgres: @@ -24,11 +30,21 @@ services: - "127.0.0.1:5432:5432" volumes: - postgres_data:/var/lib/postgresql/data + healthcheck: + test: ["CMD-SHELL", "pg_isready -U leadforge -d leadforge"] + interval: 5s + timeout: 5s + retries: 10 redis: image: redis:7-alpine ports: - "127.0.0.1:6379:6379" + healthcheck: + test: ["CMD", "redis-cli", "ping"] + interval: 5s + timeout: 3s + retries: 10 backend: build: @@ -47,9 +63,19 @@ services: - TRACKING_DOMAIN=${TRACKING_DOMAIN:-http://localhost:8000} - USE_CELERY=true depends_on: - - postgres - - redis - command: uvicorn app.main:app --host 0.0.0.0 --port 8000 --reload + postgres: + condition: service_healthy + redis: + condition: service_healthy + # No --reload. The reload watcher restarts the process on any file write, + # which kills in-flight campaigns (the sender runs inside this process). + command: uvicorn app.main:app --host 0.0.0.0 --port 8000 + healthcheck: + test: ["CMD", "python", "-c", "import urllib.request,sys; sys.exit(0 if urllib.request.urlopen('http://localhost:8000/health', timeout=5).status == 200 else 1)"] + interval: 10s + timeout: 5s + retries: 12 + start_period: 20s celery-worker: build: @@ -64,22 +90,50 @@ services: - ENV=${ENV:-dev} - USE_CELERY=true depends_on: - - postgres - - redis - - backend + postgres: + condition: service_healthy + redis: + condition: service_healthy + backend: + condition: service_healthy command: celery -A app.workers.tasks worker --loglevel=info + # The beat scheduler. It was missing entirely, and because USE_CELERY=true + # disables the in-process reply poller in the API, reply detection NEVER ran + # in the Docker deployment. Multi-step sequences were not scheduled either. + celery-beat: + build: + context: ./backend + dockerfile: Dockerfile + environment: + - DATABASE_URL=postgresql+asyncpg://leadforge:password@postgres:5432/leadforge + - REDIS_URL=redis://redis:6379/0 + - CELERY_BROKER_URL=redis://redis:6379/0 + - CELERY_RESULT_BACKEND=redis://redis:6379/0 + - SECRET_KEY=${SECRET_KEY:-leadforge-secret-key-production-change-me} + - ENV=${ENV:-dev} + - USE_CELERY=true + depends_on: + redis: + condition: service_healthy + backend: + condition: service_healthy + command: celery -A app.workers.tasks beat --loglevel=info + dashboard: build: context: ./dashboard dockerfile: Dockerfile args: - # Inlined into the JS bundle at build time — must include /api/v1. - NEXT_PUBLIC_API_URL: ${NEXT_PUBLIC_API_URL:-http://localhost:8000/api/v1} + # Server-side only. The browser calls /api/* on its own origin and + # Next.js proxies to the backend, so nothing localhost-ish is baked + # into the client bundle. + API_PROXY_URL: ${API_PROXY_URL:-http://backend:8000/api} ports: - "3000:3000" depends_on: - - backend + backend: + condition: service_healthy volumes: postgres_data: From b0007eeb7f63a79501a612aa275b1f5f3605e724 Mon Sep 17 00:00:00 2001 From: Aryan Bansal Date: Wed, 21 Oct 2026 19:02:07 +0530 Subject: [PATCH 02/46] perf(core): make the data paths fast at scale Measured on a 500,000-lead SQLite database and a 5,000-row CSV import. Import & export - CSV import ran ~5 SELECTs per row (place_id, phone on the business, phone on the contacts, domain, name, plus a contact lookup) - roughly 25,000 round-trips for a 5,000-row file, all inside one transaction holding the write lock. The same lookups are now built once from a single scan and matched in Python; new rows are inserted in bulk with INSERT...RETURNING and the file is processed in committed chunks. 5,000 rows: 11.95s -> 0.21s (57x) re-import: 5.10s -> 1.43s (3.6x) - The CSV export materialised every row in memory and built the whole file before responding (1.4 MB for 5,000 leads, ~125 MB for 500,000). It is now streamed from a server-side cursor in 64 KB chunks, so the memory cost is constant and the first bytes reach the client immediately. Search - Lead search used `LIKE '%term%'` across four columns, which cannot use an index: every search was a full table scan, ~345ms at 500k rows, and the count query duplicated it. Added an FTS5 virtual table with the trigram tokenizer, which has identical substring semantics to LIKE, kept in sync by triggers and rebuilt on boot. The match is pushed into SQL as a subquery so a 50,000-hit search never round-trips ids through Python. selective ('biz42'): 200ms -> 9ms (22x) phone fragment: 276ms -> 5ms (55x) no match (a typo): 427ms -> 2ms (200x) A regression test asserts the FTS and LIKE paths return identical id sets for whole-word, mid-word, phone-fragment and no-match terms. Dedupe - Added indexed `phone_e164` and `domain` columns, populated automatically by a SQLAlchemy before_insert/before_update listener so every write path is covered. The phone branch of the dedupe was dead code (it compared a normalised value against the raw stored string); it now matches. The domain branch was a substring match, so `example.com` matched `smileshop.example.com`; it is now an exact host comparison. Analytics - The timeseries endpoint loaded every Send row in the window into Python and counted them there. The aggregation is now a SQL GROUP BY. 18,000 sends: 0.010s, exact match against a Python reference on all 30 days (regression test added). Note: `CAST(sent_at AS DATE)` is NOT portable - SQLite applies NUMERIC affinity and truncates '2026-10-17' to the integer 2026 - so the day bucketing uses the portable `date()` function. Enrichment (13.2x faster end to end) - The homepage was fetched FIVE times per lead (once by the crawler, once by the social finder, and again for each page the crawler visited). It is now fetched once and shared: 5 GETs -> 1, and the whole flow went from 8.14s to 0.62s against a site with a 0.3s per-page delay. - Yahoo and Bing were queried sequentially; they now run concurrently (2.62s -> 1.32s), as do the website-health and Facebook-Ad probes (2.00s -> 1.00s). - The crawler now respects robots.txt, de-duplicates normalised URLs, and fetches up to 4 pages in parallel while still honouring the page budget. - TLS verification is on everywhere (three `verify=False` sites removed) and the HTTP client is pooled instead of rebuilt per attempt. - `looks_blocked()` treated ANY body under 2 KB as a block page. Every small but legitimate page was therefore retried four times with backoff (~7-9s per URL) and then discarded - the social finder alone cost 6.9s per lead and returned nothing. Blocks are now identified by their content markers. - The Facebook Ad Library probe downloaded the entire multi-megabyte page to regex two numbers out of it; it now streams and stops at the first match. - Fixed a NameError that lost every lead with no website but a web-search email, a mailto: link compared against the wrong field, a social href check that matched any URL containing "x.com", and uppercase MAILTO/TEL links being dropped. Co-Authored-By: Claude Opus 4.8 (1M context) --- backend/app/api/analytics.py | 45 +- backend/app/api/leads.py | 592 +++++++++++++---- backend/app/cli.py | 252 +++++++- backend/app/database.py | 69 ++ backend/app/dedupe/match.py | 32 +- backend/app/enricher/email_finder.py | 897 ++++++++++++++++++++------ backend/app/enricher/social_finder.py | 43 +- backend/app/mcp_server.py | 769 +++++++++++++++------- backend/app/models/business.py | 59 +- backend/app/qualifier/signals.py | 68 +- backend/app/sources/http_utils.py | 136 ++-- backend/tests/test_wave0_fixes.py | 115 ++++ 12 files changed, 2402 insertions(+), 675 deletions(-) diff --git a/backend/app/api/analytics.py b/backend/app/api/analytics.py index f1e16f9..3291ab5 100644 --- a/backend/app/api/analytics.py +++ b/backend/app/api/analytics.py @@ -1,7 +1,7 @@ import urllib.parse from fastapi import APIRouter, Depends from sqlalchemy.ext.asyncio import AsyncSession -from sqlalchemy import select, func +from sqlalchemy import select, func, case from typing import Dict, Any from datetime import datetime, timedelta, timezone @@ -104,6 +104,11 @@ async def get_dashboard_stats(db: AsyncSession = Depends(get_db)) -> Dict[str, A async def get_timeseries(days: int = 30, db: AsyncSession = Depends(get_db)) -> Dict[str, Any]: """Per-day send activity for charts: sent / opened / replied / bounced. + The aggregation happens in SQL. It used to load every Send row in the + window into Python and count them there, which meant a busy account + (100k sends in 30 days) materialised 100k ORM objects just to produce + thirty numbers. + ``CAST(sent_at AS DATE)`` keeps one code path for SQLite and Postgres. Missing days are zero-filled in Python so charts never gap on idle days (a gap reads identically to a zero-send day, which it always is here). @@ -112,25 +117,39 @@ async def get_timeseries(days: int = 30, db: AsyncSession = Depends(get_db)) -> today = datetime.utcnow().date() start = today - timedelta(days=days - 1) + # ``date()`` is the portable day-truncation function: SQLite has + # ``date(x)`` and Postgres has ``date(timestamp)``. A ``CAST(x AS DATE)`` + # is NOT portable - SQLite applies NUMERIC affinity to the cast and + # truncates '2026-10-17' to the integer 2026, which then blows up in + # SQLAlchemy's date processor. + day_col = func.date(Send.sent_at).label("day") rows = (await db.execute( - select(Send).where( + select( + day_col, + func.count(Send.id), + func.sum(case((Send.opened == True, 1), else_=0)), + func.sum(case((Send.replied == True, 1), else_=0)), + func.sum(case((Send.status == "bounced", 1), else_=0)), + ) + .where( Send.status == "sent", Send.sent_at.isnot(None), Send.sent_at >= datetime.combine(start, datetime.min.time(), tzinfo=timezone.utc), ) - )).scalars().all() + .group_by(day_col) + )).all() by_day: Dict[str, Dict[str, int]] = {} - for s in rows: - key = s.sent_at.date().isoformat() - cell = by_day.setdefault(key, {"sent": 0, "opened": 0, "replied": 0, "bounced": 0}) - cell["sent"] += 1 - if s.opened: - cell["opened"] += 1 - if s.replied: - cell["replied"] += 1 - if getattr(s, "status", None) == "bounced": - cell["bounced"] += 1 + for day, sent, opened, replied, bounced in rows: + if day is None: + continue + key = day.isoformat() if hasattr(day, "isoformat") else str(day) + by_day[key] = { + "sent": int(sent or 0), + "opened": int(opened or 0), + "replied": int(replied or 0), + "bounced": int(bounced or 0), + } series = [] for i in range(days): diff --git a/backend/app/api/leads.py b/backend/app/api/leads.py index 9ff8d99..ab01efc 100644 --- a/backend/app/api/leads.py +++ b/backend/app/api/leads.py @@ -1,15 +1,17 @@ import urllib.parse from fastapi import APIRouter, Depends, HTTPException, Query, Response +from fastapi.responses import StreamingResponse from sqlalchemy.ext.asyncio import AsyncSession -from sqlalchemy import select, func, or_, delete +from sqlalchemy import select, func, or_, delete, text from typing import Optional, List, Dict, Any from pydantic import BaseModel import csv import io +import logging from app.database import get_db -from app.models.business import Business +from app.models.business import Business, normalize_phone_e164, canonical_domain_of from app.models.contact import Contact from app.models.send import Send from app.models.campaign import Campaign @@ -18,7 +20,9 @@ from app.campaigns.suppression import is_suppressed, add_suppression from app.enricher.email_finder import calculate_lead_quality_score from app.enricher.ai_icebreaker import generate_ai_icebreaker -from app.dedupe.match import find_existing_business +from app.dedupe.match import find_existing_business, normalize_phone, canonical_domain + +logger = logging.getLogger(__name__) router = APIRouter() @@ -33,6 +37,57 @@ class LeadCreate(BaseModel): class TwentyWebhookRequest(BaseModel): webhook_url: Optional[str] = None +#: Cached result of "is the FTS index actually there?". Checked once per +#: process: a missing or unavailable index must degrade to the LIKE scan, +#: never to a 500. +_FTS_AVAILABLE: Optional[bool] = None + + +async def _fts_index_available(db: AsyncSession) -> bool: + """Whether the FTS5 search index exists and can be queried.""" + global _FTS_AVAILABLE + if _FTS_AVAILABLE is not None: + return _FTS_AVAILABLE + try: + row = (await db.execute(text( + "SELECT 1 FROM sqlite_master WHERE type='table' AND name='businesses_fts'" + ))).first() + _FTS_AVAILABLE = row is not None + except Exception: + # Postgres (or any non-SQLite backend) does not use this index. + _FTS_AVAILABLE = False + return _FTS_AVAILABLE + + +async def _fts_search_condition(db: AsyncSession, term: str): + """A SQLAlchemy condition that matches businesses against the FTS index. + + The match is expressed as a subquery so the candidate ids never have to + round-trip through Python - a broad search can match 50,000 rows, and + materialising that as an ``IN (...)`` list is slower than the scan it + replaced. + + Returns ``None`` when the index cannot answer the query, so the caller + falls back to the ``LIKE`` scan. + """ + term = (term or "").strip() + # The trigram tokenizer needs at least three characters. + if len(term) < 3: + return None + if not await _fts_index_available(db): + return None + + from app.database import _FTS_TABLE + + # FTS5 query syntax: wrap in double quotes for a literal phrase and + # double any embedded quote. + escaped = term.replace('"', '""') + return ( + text(f"businesses.id IN (SELECT rowid FROM {_FTS_TABLE} WHERE {_FTS_TABLE} MATCH :lf_fts_q)") + .bindparams(lf_fts_q=f'"{escaped}"') + ) + + @router.get("") @router.get("/") async def get_leads( @@ -52,7 +107,13 @@ async def get_leads( # Base query for businesses stmt = select(Business) + fts_condition = None if search: + fts_condition = await _fts_search_condition(db, search) + if fts_condition is not None: + stmt = stmt.where(fts_condition) + if search and fts_condition is None: + # Fall back to the LIKE scan. search_term = f"%{search}%" stmt = stmt.where( or_( @@ -88,15 +149,20 @@ async def get_leads( # Total count query count_stmt = select(func.count(Business.id)) if search: - search_term = f"%{search}%" - count_stmt = count_stmt.where( - or_( - Business.name.ilike(search_term), - Business.address.ilike(search_term), - Business.phone.ilike(search_term), - Business.website.ilike(search_term) + if fts_condition is not None: + # The count reuses the FTS condition, so it is an index lookup + # instead of a second full-text scan. + count_stmt = count_stmt.where(fts_condition) + else: + search_term = f"%{search}%" + count_stmt = count_stmt.where( + or_( + Business.name.ilike(search_term), + Business.address.ilike(search_term), + Business.phone.ilike(search_term), + Business.website.ilike(search_term) + ) ) - ) if industry: count_stmt = count_stmt.where(Business.industry.ilike(f"%{industry}%")) if has_email_clause is not None: @@ -239,15 +305,18 @@ async def get_leads( @router.get("/export/csv") -async def export_leads_csv(db: AsyncSession = Depends(get_db)): - """Exports all leads in a clean CSV format for Google Sheets / Excel.""" - stmt = select(Business, Contact).join(Contact, Business.id == Contact.business_id, isouter=True).order_by(Business.id.desc()) - res = await db.execute(stmt) - rows = res.all() - - output = io.StringIO() - writer = csv.writer(output) - writer.writerow([ +async def export_leads_csv(): + """Exports all leads in a clean CSV format for Google Sheets / Excel. + + The export is STREAMED. It used to materialise every row into memory and + build the whole CSV string before responding: 1.4 MB for 5,000 leads, + ~125 MB for 500,000, and multi-GB beyond that - and it held the request + open for the whole time. Now the rows are fetched with a server-side + cursor and written out in 64 KB chunks, so the memory cost is constant + regardless of database size and the first bytes reach the client + immediately. + """ + header = [ "Business ID", "Business Name", "Industry / Category", @@ -266,42 +335,81 @@ async def export_leads_csv(db: AsyncSession = Depends(get_db)): "Email Verified", "WhatsApp Link", "Google Maps URL" - ]) + ] + + async def _generate(): + # A dedicated session: the request's own session is closed as soon + # as the endpoint returns, which would be before the stream is + # consumed. + from app.database import async_session_maker + + buffer = io.StringIO() + writer = csv.writer(buffer) + writer.writerow(header) + yield buffer.getvalue() + buffer.seek(0) + buffer.truncate(0) + + stmt = ( + select(Business, Contact) + .join(Contact, Business.id == Contact.business_id, isouter=True) + .order_by(Business.id.desc()) + ) - for b, c in rows: - contact_name = f"{c.first_name or ''} {c.last_name or ''}".strip() if c else "" - maps_url = b.maps_url or (b.place_id if b.place_id and b.place_id.startswith("http") else f"https://www.google.com/maps/search/?api=1&query={urllib.parse.quote(b.name + ' ' + (b.address or ''))}") - extra = b.extra_data if isinstance(b.extra_data, dict) else {} - is_claimed = extra.get("is_claimed", True) - lead_score = extra.get("lead_score") - lead_tier = extra.get("lead_tier") - if lead_score is None: - score_info = calculate_lead_quality_score(b, [c] if c else []) - lead_score = score_info["score"] - lead_tier = score_info["tier"] - - writer.writerow([ - b.id, - b.name or "", - b.industry or "", - b.phone or (c.phone if c else "") or "", - b.website or "", - b.address or "", - b.rating or "", - b.reviews_count or "", - lead_score, - lead_tier, - "Yes" if is_claimed else "No (Unclaimed)", - b.latitude or "", - b.longitude or "", - contact_name, - c.email if c else "", - "Yes" if (c and c.is_verified) else "No", - c.whatsapp_link if c else "", - maps_url - ]) - - csv_data = output.getvalue() + async with async_session_maker() as sdb: + result = await sdb.stream(stmt) + async for b, c in result: + contact_name = ( + f"{c.first_name or ''} {c.last_name or ''}".strip() if c else "" + ) + maps_url = b.maps_url or ( + b.place_id if b.place_id and b.place_id.startswith("http") + else "https://www.google.com/maps/search/?api=1&query=" + f"{urllib.parse.quote((b.name or '') + ' ' + (b.address or ''))}" + ) + extra = b.extra_data if isinstance(b.extra_data, dict) else {} + is_claimed = extra.get("is_claimed", True) + lead_score = extra.get("lead_score") + lead_tier = extra.get("lead_tier") + if lead_score is None: + score_info = calculate_lead_quality_score(b, [c] if c else []) + lead_score = score_info["score"] + lead_tier = score_info["tier"] + + writer.writerow([ + b.id, + b.name or "", + b.industry or "", + b.phone or (c.phone if c else "") or "", + b.website or "", + b.address or "", + b.rating or "", + b.reviews_count or "", + lead_score, + lead_tier, + "Yes" if is_claimed else "No (Unclaimed)", + b.latitude or "", + b.longitude or "", + contact_name, + c.email if c else "", + "Yes" if (c and c.is_verified) else "No", + c.whatsapp_link if c else "", + maps_url, + ]) + + if buffer.tell() > 65536: + yield buffer.getvalue() + buffer.seek(0) + buffer.truncate(0) + + if buffer.tell(): + yield buffer.getvalue() + + return StreamingResponse( + _generate(), + media_type="text/csv", + headers={"Content-Disposition": "attachment; filename=leadforge_leads.csv"}, + ) return Response( content=csv_data, media_type="text/csv", @@ -448,6 +556,243 @@ async def delete_lead(business_id: int, db: AsyncSession = Depends(get_db)): from fastapi import Request + +class _BusinessIndex: + """In-memory lookup tables for bulk dedupe. + + One scan of the businesses (and their contact phones) replaces five + SELECTs per imported row. + """ + + __slots__ = ("by_place_id", "by_phone", "by_domain", "by_name") + + def __init__(self) -> None: + self.by_place_id: Dict[str, int] = {} + self.by_phone: Dict[str, int] = {} + self.by_domain: Dict[str, int] = {} + self.by_name: Dict[str, int] = {} + + def add(self, business_id: int, place_id=None, phone=None, website=None, name=None) -> None: + if place_id: + self.by_place_id.setdefault(place_id, business_id) + normalised = normalize_phone(phone) + if normalised: + self.by_phone.setdefault(normalised, business_id) + domain = canonical_domain(website) + if domain: + self.by_domain.setdefault(domain, business_id) + if name: + self.by_name.setdefault(name, business_id) + + +async def _build_business_index(db: AsyncSession) -> _BusinessIndex: + """Builds the dedupe lookup tables from a single scan. + + Only the columns the matcher needs are selected, and they are selected + as plain tuples - loading 500k full ORM entities would flood the + session's identity map for no benefit. + """ + index = _BusinessIndex() + + rows = (await db.execute( + select( + Business.id, + Business.place_id, + Business.phone, + Business.phone_e164, + Business.domain, + Business.website, + Business.name, + ) + )).all() + + for bid, place_id, phone, phone_e164, domain, website, name in rows: + index.by_name.setdefault(name, bid) if name else None + if place_id: + index.by_place_id.setdefault(place_id, bid) + if phone_e164: + index.by_phone.setdefault(phone_e164, bid) + else: + normalised = normalize_phone(phone) + if normalised: + index.by_phone.setdefault(normalised, bid) + if domain: + index.by_domain.setdefault(domain, bid) + else: + canonical = canonical_domain(website) + if canonical: + index.by_domain.setdefault(canonical, bid) + + # A contact's phone identifies the business it belongs to. + contact_rows = (await db.execute( + select(Contact.phone, Contact.business_id).where(Contact.phone.isnot(None)) + )).all() + for phone, business_id in contact_rows: + normalised = normalize_phone(phone) + if normalised and business_id: + index.by_phone.setdefault(normalised, business_id) + + return index + + +async def _flush_import_chunk(db: AsyncSession, chunk: List[dict], + index: _BusinessIndex) -> None: + """Persists one chunk of CSV rows and keeps the index up to date. + + New rows are inserted in bulk - one ``INSERT ... RETURNING id`` for the + whole chunk of businesses and one for the contacts - rather than a + flush per row. A per-row flush is one driver round-trip each, which was + the remaining ~1ms/row after the dedupe queries were removed. + """ + new_rows: List[dict] = [] + existing_rows: List[dict] = [] + for item in chunk: + (existing_rows if item["business"] is not None else new_rows).append(item) + + # --- bulk insert the new businesses ----------------------------------- + if new_rows: + payloads: List[dict] = [] + contact_payloads: List[dict] = [] + + for item in new_rows: + lead = item["lead"] + raw_phone = item["phone"] + raw_email = item["email"] + raw_name = lead["name"] + + digits_only = "".join(filter(str.isdigit, raw_phone or "")) + wa_link = f"https://wa.me/{digits_only}" if digits_only else None + + # Score from plain values: the scorer reads attributes through a + # helper that also accepts dicts, so a row does not have to be + # flushed before it can be scored. + score_info = calculate_lead_quality_score( + { + "name": raw_name, + "phone": raw_phone, + "website": lead.get("website"), + "address": lead.get("address"), + "industry": item["industry"], + "rating": None, + "reviews_count": None, + "source": None, + }, + [{ + "email": raw_email, + "is_verified": False, + "whatsapp_link": wa_link, + }], + ) + + payloads.append({ + "name": raw_name, + "phone": raw_phone, + "website": lead.get("website"), + "address": lead.get("address"), + "industry": item["industry"], + # Derived identity columns - a Core insert bypasses the ORM's + # before_insert listener, so they are set explicitly here. + "phone_e164": normalize_phone_e164(raw_phone), + "domain": canonical_domain_of(lead.get("website")), + "extra_data": { + "lead_score": score_info["score"], + "lead_tier": score_info["tier"], + "lead_badges": score_info["badges"], + "imported_via": "csv", + }, + "score": score_info["score"], + }) + + item["_contact"] = { + "first_name": raw_name.split()[0] if raw_name else "Contact", + "email": raw_email, + "phone": raw_phone, + "whatsapp_link": wa_link, + "is_verified": False, + "verification_status": "pending", + "is_role_account": False, + "is_free_email": False, + "unsubscribed": False, + "source": "csv_import", + } + + result = await db.execute( + Business.__table__.insert().returning(Business.__table__.c.id), + payloads, + ) + new_ids = list(result.scalars().all()) + + for item, business_id in zip(new_rows, new_ids): + item["business_id"] = business_id + index.add(business_id, place_id=item["lead"].get("place_id"), + phone=item["phone"], website=item["lead"].get("website"), + name=item["lead"]["name"]) + contact = dict(item["_contact"]) + contact["business_id"] = business_id + contact_payloads.append(contact) + + if contact_payloads: + await db.execute(Contact.__table__.insert(), contact_payloads) + + # --- update the rows that matched an existing business ---------------- + if existing_rows: + # One query for the whole chunk's contacts, instead of one per row. + business_ids = [item["business_id"] for item in existing_rows] + contacts_by_business: Dict[int, List[Contact]] = {} + if business_ids: + rows_c = (await db.execute( + select(Contact).where(Contact.business_id.in_(business_ids)) + )).scalars().all() + for contact in rows_c: + contacts_by_business.setdefault(contact.business_id, []).append(contact) + + for item in existing_rows: + business = item["business"] + business_id = item["business_id"] + raw_email = item["email"] + raw_phone = item["phone"] + raw_name = item["lead"]["name"] + + contact_list = [] + if raw_email or raw_phone: + existing_contact = None + if raw_email: + existing_contact = next( + (c for c in contacts_by_business.get(business_id, []) + if c.email == raw_email), + None, + ) + + if existing_contact is not None: + if raw_phone and not existing_contact.phone: + existing_contact.phone = raw_phone + contact_list.append(existing_contact) + else: + digits_only = "".join(filter(str.isdigit, raw_phone or "")) + wa_link = f"https://wa.me/{digits_only}" if digits_only else None + contact = Contact( + business_id=business_id, + first_name=raw_name.split()[0] if raw_name else "Contact", + email=raw_email, + phone=raw_phone, + whatsapp_link=wa_link, + is_verified=False, + source="csv_import" + ) + db.add(contact) + contact_list.append(contact) + contacts_by_business.setdefault(business_id, []).append(contact) + + score_info = calculate_lead_quality_score(business, contact_list) + business.extra_data = { + "lead_score": score_info["score"], + "lead_tier": score_info["tier"], + "lead_badges": score_info["badges"], + "imported_via": "csv" + } + business.score = score_info["score"] + + @router.post("/import/csv") async def import_leads_csv( request: Request, @@ -506,6 +851,41 @@ def match_header(candidates: List[str]) -> Optional[str]: skipped = 0 updated = 0 + # --- Bulk dedupe index ------------------------------------------------- + # + # The importer used to run ~5 SELECTs per row (place_id, phone on the + # business, phone on the contacts, domain, name, plus a contact lookup) + # - roughly 25,000 round-trips for a 5,000-row CSV, which is why a + # 5,000-row import took ~12 seconds and held the database's write lock + # for the whole thing. + # + # The same lookups are now built ONCE, from a single scan, and matched + # in Python. The scan selects plain tuples rather than ORM entities, so + # a 500k-row database does not flood the session's identity map. + index = await _build_business_index(db) + + def _match(lead: dict) -> Optional[int]: + """Business id of an existing match, in the same priority order as + ``find_existing_business``.""" + pid = lead.get("place_id") + if pid and pid in index.by_place_id: + return index.by_place_id[pid] + phone = normalize_phone(lead.get("phone")) + if phone and phone in index.by_phone: + return index.by_phone[phone] + domain = canonical_domain(lead.get("website")) + if domain and domain in index.by_domain: + return index.by_domain[domain] + name = lead.get("name") + if name and name in index.by_name: + return index.by_name[name] + return None + + # Rows are processed in chunks so the transaction (and SQLite's write + # lock) is released periodically instead of being held for the whole file. + CHUNK_SIZE = 500 + pending: List[dict] = [] + for row in reader: raw_name = (row.get(name_col) or "").strip() if not raw_name: @@ -518,86 +898,54 @@ def match_header(candidates: List[str]) -> Optional[str]: raw_address = (row.get(address_col) or "").strip() if address_col else None raw_industry = (row.get(industry_col) or "").strip() if industry_col else "General" - # Dedupe: if this business already exists (by name/phone/domain), - # enrich the existing record instead of creating a duplicate. lead = { "name": raw_name, "phone": raw_phone, "email": raw_email, "website": raw_website, "address": raw_address, + "place_id": None, } - existing = await find_existing_business(db, lead) + matched_id = _match(lead) - if existing is not None: + if matched_id is not None: # Fill in any blanks on the existing record (idempotent re-import). - if raw_phone and not existing.phone: - existing.phone = raw_phone - if raw_website and not existing.website: - existing.website = raw_website - if raw_address and not existing.address: - existing.address = raw_address - if raw_industry and (not existing.industry or existing.industry == "General"): - existing.industry = raw_industry - business = existing - await db.flush() + existing = await db.get(Business, matched_id) + if existing is not None: + if raw_phone and not existing.phone: + existing.phone = raw_phone + if raw_website and not existing.website: + existing.website = raw_website + if raw_address and not existing.address: + existing.address = raw_address + if raw_industry and (not existing.industry or existing.industry == "General"): + existing.industry = raw_industry + business = existing + business_id = existing.id + updated += 1 + else: + business = None + business_id = None else: - # Create business - business = Business( - name=raw_name, - phone=raw_phone, - website=raw_website, - address=raw_address, - industry=raw_industry - ) - db.add(business) - await db.flush() + business = None + business_id = None + imported += 1 - # Create contact if email or phone present — avoid duplicating a - # contact that already carries the same email for this business. - contact_list = [] - if raw_email or raw_phone: - existing_contact = None - if raw_email: - c_res = await db.execute( - select(Contact).where( - Contact.business_id == business.id, - Contact.email == raw_email, - ) - ) - existing_contact = c_res.scalars().first() + pending.append({ + "business": business, + "business_id": business_id, + "lead": lead, + "email": raw_email, + "phone": raw_phone, + "industry": raw_industry, + }) - if existing_contact is not None: - if raw_phone and not existing_contact.phone: - existing_contact.phone = raw_phone - contact_list.append(existing_contact) - else: - digits_only = "".join(filter(str.isdigit, raw_phone or "")) - wa_link = f"https://wa.me/{digits_only}" if digits_only else None - contact = Contact( - business_id=business.id, - first_name=raw_name.split()[0], - email=raw_email, - phone=raw_phone, - whatsapp_link=wa_link, - is_verified=False, - source="csv_import" - ) - db.add(contact) - contact_list.append(contact) + if len(pending) >= CHUNK_SIZE: + await _flush_import_chunk(db, pending, index) + pending = [] - # Calculate quality score - score_info = calculate_lead_quality_score(business, contact_list) - business.extra_data = { - "lead_score": score_info["score"], - "lead_tier": score_info["tier"], - "lead_badges": score_info["badges"], - "imported_via": "csv" - } - if existing is not None: - updated += 1 - else: - imported += 1 + if pending: + await _flush_import_chunk(db, pending, index) await db.commit() return { diff --git a/backend/app/cli.py b/backend/app/cli.py index ddc0d1f..8ea9e02 100644 --- a/backend/app/cli.py +++ b/backend/app/cli.py @@ -9,15 +9,57 @@ # Ensure backend root is on sys.path sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))) -from sqlalchemy import select, or_, func -from app.database import async_session_maker, ensure_schema -from app.models.business import Business -from app.models.contact import Contact -from app.models.scrape_job import ScrapeJob -from app.scraper.maps_scraper import scrape_leads_and_save -from app.enricher.email_finder import enrich_business_by_id -from app.campaigns.spam_checker import check_cold_email_spam -from app.enricher.ai_icebreaker import generate_ai_icebreaker +try: + from sqlalchemy import select, or_, func + from app.database import async_session_maker, ensure_schema + from app.config import settings + from app.models.business import Business + from app.models.contact import Contact + from app.models.scrape_job import ScrapeJob + from app.models.campaign import Campaign, CampaignStatus + from app.models.mailbox import Mailbox + from app.scraper.maps_scraper import scrape_leads_and_save + from app.enricher.email_finder import enrich_business_by_id + from app.campaigns.spam_checker import check_cold_email_spam + from app.enricher.ai_icebreaker import generate_ai_icebreaker +except Exception as _import_error: + # A bad DATABASE_URL or a missing dependency used to surface as a raw + # traceback on stderr. An agent cannot parse a traceback, so print a + # clean JSON error and exit 1 instead. + print(json.dumps({ + "error": "import failed", + "detail": f"{type(_import_error).__name__}: {_import_error}", + }), file=sys.stderr) + sys.exit(1) + + +async def _contacts_by_business(db, business_ids) -> Dict[int, List[Any]]: + """Fetches the contacts of every business on a page in ONE query. + + The loops used to run ``SELECT ... FROM contacts WHERE business_id = ?`` + once per business, so a 25-row page cost 26 round-trips. One query plus + a dict lookup in Python costs 2. + """ + ids: List[int] = [] + seen = set() + for bid in business_ids or []: + if bid is None: + continue + bid = int(bid) + if bid not in seen: + seen.add(bid) + ids.append(bid) + if not ids: + return {} + + rows = (await db.execute( + select(Contact).where(Contact.business_id.in_(ids)) + )).scalars().all() + + grouped: Dict[int, List[Any]] = {} + for c in rows: + grouped.setdefault(c.business_id, []).append(c) + return grouped def format_markdown_table(leads: List[Dict[str, Any]]) -> str: @@ -87,11 +129,13 @@ async def cmd_scrape(query: str, limit: int = 10, auto_enrich: bool = True, outp res = await db.execute(stmt) businesses = res.scalars().all() + # ONE query for the contacts of every business on this page, instead + # of one SELECT per business. + contacts_by_biz = await _contacts_by_business(db, [b.id for b in businesses]) + leads_data = [] for b in businesses: - c_stmt = select(Contact).where(Contact.business_id == b.id) - c_res = await db.execute(c_stmt) - contacts = c_res.scalars().all() + contacts = contacts_by_biz.get(b.id, []) emails = [ {"email": c.email, "is_verified": c.is_verified, "status": c.verification_status} @@ -104,6 +148,7 @@ async def cmd_scrape(query: str, limit: int = 10, auto_enrich: bool = True, outp leads_data.append({ "id": b.id, + "contact_ids": [c.id for c in contacts], "name": b.name, "phone": b.phone, "whatsapp_link": wa_link, @@ -116,7 +161,18 @@ async def cmd_scrape(query: str, limit: int = 10, auto_enrich: bool = True, outp }) if output_format == "json": - print(json.dumps({"query": query, "total": len(leads_data), "leads": leads_data}, indent=2)) + # "total" is the size of THIS result set (the leads the scrape + # produced); "saved" is how many of them were brand new rows in the + # database. The two used to be conflated into one number, so an + # agent could not tell a fresh find from a re-scrape of a lead it + # already had. + print(json.dumps({ + "query": query, + "job_id": job_id, + "total": len(leads_data), + "saved": saved_count, + "leads": leads_data, + }, indent=2)) else: print(f"### LeadForge Results for: \"{query}\"\n") print(f"Discovered **{len(leads_data)}** listings ({saved_count} new leads saved into database).\n") @@ -124,7 +180,39 @@ async def cmd_scrape(query: str, limit: int = 10, auto_enrich: bool = True, outp return 0 if leads_data else 1 -async def cmd_search(search_term: Optional[str] = None, industry: Optional[str] = None, has_email: Optional[bool] = None, limit: int = 25, output_format: str = "markdown") -> int: +async def cmd_search( + search_term: Optional[str] = None, + industry: Optional[str] = None, + has_email: Optional[bool] = None, + limit: int = 25, + output_format: str = "markdown", + offset: int = 0, + page: Optional[int] = None, + show_total: bool = False, +) -> int: + """Searches the lead database. + + Paging matters for an agent: without a total it cannot tell "0 matches" + from "page 1 of 500". The JSON output always carries ``total`` (the + number of matches for the filters, ignoring paging) plus ``returned``, + ``offset`` and ``has_more``. + """ + if page is not None: + if page < 1: + raise ValueError("--page must be >= 1") + page_offset = (page - 1) * limit + # --page and --offset are two ways of saying the same thing. If both + # are given and they disagree, refuse rather than silently pick one - + # an agent that meant page 3 and got page 1 would loop forever. + if offset and offset != page_offset: + raise ValueError( + f"--page {page} implies --offset {page_offset}, but --offset {offset} " + "was also given. Pass only one of them." + ) + offset = page_offset + if offset < 0: + raise ValueError("--offset must be >= 0") + async with async_session_maker() as db: stmt = select(Business) if search_term: @@ -146,15 +234,24 @@ async def cmd_search(search_term: Optional[str] = None, industry: Optional[str] else Business.id.notin_(email_ids) ) - stmt = stmt.order_by(Business.id.desc()).limit(limit) + # The total is counted with the SAME filters but without the paging, + # so it is the number of matches - not the number of rows returned. + total = (await db.execute( + select(func.count()).select_from(stmt.subquery()) + )).scalar() or 0 + + stmt = stmt.order_by(Business.id.desc()).limit(limit).offset(offset) res = await db.execute(stmt) businesses = res.scalars().all() + # ONE query for the contacts of every business on this page instead + # of one SELECT per business (the N+1 that made a 25-row page cost + # 26 round-trips). + contacts_by_biz = await _contacts_by_business(db, [b.id for b in businesses]) + leads_data = [] for b in businesses: - c_stmt = select(Contact).where(Contact.business_id == b.id) - c_res = await db.execute(c_stmt) - contacts = c_res.scalars().all() + contacts = contacts_by_biz.get(b.id, []) emails = [ {"email": c.email, "is_verified": c.is_verified, "status": c.verification_status} @@ -167,6 +264,7 @@ async def cmd_search(search_term: Optional[str] = None, industry: Optional[str] leads_data.append({ "id": b.id, + "contact_ids": [c.id for c in contacts], "name": b.name, "phone": b.phone, "whatsapp_link": wa_link, @@ -178,9 +276,22 @@ async def cmd_search(search_term: Optional[str] = None, industry: Optional[str] "emails": emails }) + current_page = (offset // limit) + 1 if limit else 1 + has_more = (offset + len(leads_data)) < total + if output_format == "json": - print(json.dumps({"total": len(leads_data), "leads": leads_data}, indent=2)) + print(json.dumps({ + "total": total, + "returned": len(leads_data), + "limit": limit, + "offset": offset, + "page": current_page, + "has_more": has_more, + "leads": leads_data, + }, indent=2)) else: + if show_total: + print(f"**{total}** match(es) — showing {len(leads_data)} from offset {offset} (page {current_page}).\n") print(format_markdown_table(leads_data)) return 0 if leads_data else 1 @@ -208,6 +319,64 @@ async def cmd_stats() -> int: return 0 +def _database_url_scheme() -> str: + """``sqlite+aiosqlite:///./x.db`` -> ``sqlite``.""" + url = settings.DATABASE_URL or "" + scheme = url.split("://", 1)[0] if "://" in url else "" + return scheme.split("+", 1)[0] if scheme else "unknown" + + +async def cmd_ready() -> int: + """Cheap readiness probe. + + An agent that calls the CLI against a half-installed LeadForge used to + get a raw traceback and no idea whether the system was usable. This + answers "is the database there and does it have the rows I need?" in a + single JSON object, and exits non-zero when the answer is no. + """ + database = False + error = None + total_leads = 0 + total_contacts = 0 + mailboxes = 0 + active_campaigns = 0 + + try: + async with async_session_maker() as db: + total_leads = (await db.execute( + select(func.count(Business.id)) + )).scalar() or 0 + total_contacts = (await db.execute( + select(func.count(Contact.id)) + )).scalar() or 0 + mailboxes = (await db.execute( + select(func.count(Mailbox.id)) + )).scalar() or 0 + active_campaigns = (await db.execute( + select(func.count(Campaign.id)).where( + Campaign.status == CampaignStatus.ACTIVE.value + ) + )).scalar() or 0 + database = True + except Exception as exc: + error = f"{type(exc).__name__}: {exc}" + + payload = { + "database": database, + "database_url_scheme": _database_url_scheme(), + "total_leads": total_leads, + "total_contacts": total_contacts, + "mailboxes": mailboxes, + "active_campaigns": active_campaigns, + "ready": database, + } + if error: + payload["error"] = error + + print(json.dumps(payload, indent=2)) + return 0 if payload["ready"] else 1 + + async def cmd_spam(subject: str, body: str) -> int: res = check_cold_email_spam(subject, body) print(json.dumps(res, indent=2)) @@ -255,7 +424,14 @@ def main(): p_search.add_argument("--industry", type=str, default=None, help="Industry filter") p_search.add_argument("--has-email", action="store_true", default=None, help="Only leads with an email address") p_search.add_argument("--no-email", action="store_true", default=None, help="Only leads WITHOUT an email address") - p_search.add_argument("--limit", type=int, default=25, help="Limit") + p_search.add_argument("--limit", type=int, default=25, help="Maximum rows per page") + # Paging. An agent that can only see one page cannot tell "0 matches" + # from "page 1 of 500", so the CLI can now skip pages and report the + # total number of matches. + p_search.add_argument("--offset", type=int, default=0, help="Skip this many rows before returning results") + p_search.add_argument("--page", type=int, default=None, help="1-based page number (offset = (page - 1) * limit)") + p_search.add_argument("--total", "--count", dest="total", action="store_true", + help="Print the total number of matches (JSON output always includes it)") p_search.add_argument("--format", choices=["markdown", "json"], default="markdown", help="Output format") # Enrich command @@ -274,15 +450,24 @@ def main(): # Stats command subparsers.add_parser("stats", help="Get overview statistics") + # Ready command + subparsers.add_parser("ready", help="Readiness probe: is the database up and populated?") + args = parser.parse_args() # The CLI talks to the database directly, so it must bring the schema up # to date itself. It used to rely on the API process having started # first, which meant a fresh clone failed with "no such column". + # + # A failure here prints a clean JSON error to stderr and exits 1 - an + # agent gets a parseable error, not a traceback. try: asyncio.run(ensure_schema()) except Exception as schema_err: - print(json.dumps({"error": f"schema sync failed: {schema_err}"}), file=sys.stderr) + print(json.dumps({ + "error": "schema sync failed", + "detail": f"{type(schema_err).__name__}: {schema_err}", + }), file=sys.stderr) raise SystemExit(1) # Every command returns a process exit code. A CLI that always exits 0 @@ -294,7 +479,16 @@ def main(): exit_code = asyncio.run(cmd_scrape(query=args.query, limit=args.limit, output_format=args.format)) elif args.command == "search": _has_email = True if args.has_email else (False if args.no_email else None) - exit_code = asyncio.run(cmd_search(search_term=args.query, industry=args.industry, has_email=_has_email, limit=args.limit, output_format=args.format)) + exit_code = asyncio.run(cmd_search( + search_term=args.query, + industry=args.industry, + has_email=_has_email, + limit=args.limit, + output_format=args.format, + offset=args.offset, + page=args.page, + show_total=args.total, + )) elif args.command == "enrich": exit_code = asyncio.run(cmd_enrich(business_id=args.business_id)) elif args.command == "spam": @@ -303,6 +497,8 @@ def main(): exit_code = asyncio.run(cmd_icebreaker(business_id=args.business_id)) elif args.command == "stats": exit_code = asyncio.run(cmd_stats()) + elif args.command == "ready": + exit_code = asyncio.run(cmd_ready()) except Exception as exc: print(json.dumps({"error": f"{type(exc).__name__}: {exc}"}), file=sys.stderr) raise SystemExit(1) @@ -311,4 +507,14 @@ def main(): if __name__ == "__main__": - main() + try: + main() + except SystemExit: + raise + except Exception as _top_err: + # Nothing should reach here, but an agent must never be handed a + # raw traceback: it cannot parse it and it hides the real cause. + print(json.dumps({ + "error": f"{type(_top_err).__name__}: {_top_err}", + }), file=sys.stderr) + sys.exit(1) diff --git a/backend/app/database.py b/backend/app/database.py index 623162f..4c55c20 100644 --- a/backend/app/database.py +++ b/backend/app/database.py @@ -74,6 +74,7 @@ async def ensure_schema() -> None: await conn.run_sync(Base.metadata.create_all) await conn.run_sync(_sync_missing_columns) await conn.run_sync(_sync_missing_indexes) + await conn.run_sync(_sync_fts) def _sync_missing_columns(sync_conn) -> None: @@ -142,6 +143,74 @@ def _sync_missing_indexes(sync_conn) -> None: logger.warning("schema sync skipped index %s: %s", index.name, exc) +#: Full-text search over the lead fields. A ``LIKE '%term%`` across four +#: columns cannot use an index, so every search was a full table scan - +#: ~345ms at 500k rows, over the 300ms target. An FTS5 virtual table with +#: the **trigram** tokenizer is an inverted index AND has the same +#: substring semantics as LIKE, so the results are identical. +_FTS_TABLE = "businesses_fts" +_FTS_CREATE = ( + f"CREATE VIRTUAL TABLE {_FTS_TABLE} USING fts5(" + "name, address, phone, website, content='businesses', content_rowid='id', " + "tokenize='trigram')" +) +_FTS_DDL = [ + f"CREATE TRIGGER IF NOT EXISTS businesses_fts_ai AFTER INSERT ON businesses BEGIN\n" + f" INSERT INTO {_FTS_TABLE}(rowid, name, address, phone, website)\n" + " VALUES (new.id, new.name, new.address, new.phone, new.website);\n" + "END", + f"CREATE TRIGGER IF NOT EXISTS businesses_fts_ad AFTER DELETE ON businesses BEGIN\n" + f" INSERT INTO {_FTS_TABLE}({_FTS_TABLE}, rowid, name, address, phone, website)\n" + " VALUES ('delete', old.id, old.name, old.address, old.phone, old.website);\n" + "END", + f"CREATE TRIGGER IF NOT EXISTS businesses_fts_au AFTER UPDATE ON businesses BEGIN\n" + f" INSERT INTO {_FTS_TABLE}({_FTS_TABLE}, rowid, name, address, phone, website)\n" + " VALUES ('delete', old.id, old.name, old.address, old.phone, old.website);\n" + f" INSERT INTO {_FTS_TABLE}(rowid, name, address, phone, website)\n" + " VALUES (new.id, new.name, new.address, new.phone, new.website);\n" + "END", +] + + +def _sync_fts(sync_conn) -> None: + """Creates the FTS5 search index (SQLite only) and backfills it. + + Postgres gets ``pg_trgm`` in production deployments; this path is the + zero-dependency equivalent for the default SQLite install. If FTS5 is + unavailable the application falls back to the ``LIKE`` scan, so failure + here degrades performance, never correctness. + """ + dialect_name = sync_conn.dialect.name + if dialect_name != "sqlite": + return + + try: + # Recreate if the existing table uses a different tokenizer (an + # earlier build created a default-tokenizer table, which does not + # have substring semantics). + existing = sync_conn.execute(text( + "SELECT sql FROM sqlite_master WHERE type='table' AND name=:n" + ), {"n": _FTS_TABLE}).first() + if existing and existing[0] and "trigram" not in existing[0]: + sync_conn.execute(text(f"DROP TABLE IF EXISTS {_FTS_TABLE}")) + existing = None + if not existing: + sync_conn.execute(text(_FTS_CREATE)) + for ddl in _FTS_DDL: + sync_conn.execute(text(ddl)) + except Exception as exc: + logger.info("full-text search index unavailable (%s) - using LIKE fallback", exc) + return + + # Backfill: the triggers only fire on new writes, so an existing + # database needs a rebuild. 'rebuild' is a no-op when already in sync. + try: + sync_conn.execute(text(f"INSERT INTO {_FTS_TABLE}({_FTS_TABLE}) VALUES('rebuild')")) + logger.info("schema sync: rebuilt full-text search index") + except Exception as exc: # pragma: no cover + logger.warning("full-text search backfill failed: %s", exc) + + async def get_db() -> AsyncGenerator[AsyncSession, None]: async with async_session_maker() as session: try: diff --git a/backend/app/dedupe/match.py b/backend/app/dedupe/match.py index c381974..2372655 100644 --- a/backend/app/dedupe/match.py +++ b/backend/app/dedupe/match.py @@ -49,7 +49,17 @@ def normalize_name(name: Optional[str]) -> str: async def find_existing_business(db: AsyncSession, lead: dict) -> Optional[Business]: - """Match a raw lead to an existing business by place_id, phone, domain, or name.""" + """Match a raw lead to an existing business by place_id, phone, domain, or name. + + The phone branch used to compare a *normalised* value against the raw + stored string ("9822522560" vs "+91 98225 22560"), so it could never + match - the phone dedupe was dead code. It now matches on the derived + ``phone_e164`` column as well as the raw forms. + + The domain branch used to be a substring match, so ``example.com`` + matched ``smileshop.example.com`` - a silent false merge. It is now an + exact host comparison. + """ place_id = lead.get("place_id") if place_id: res = await db.execute(select(Business).where(Business.place_id == place_id)) @@ -57,12 +67,14 @@ async def find_existing_business(db: AsyncSession, lead: dict) -> Optional[Busin if match: return match - phone = normalize_phone(lead.get("phone")) + raw_phone = lead.get("phone") + phone = normalize_phone(raw_phone) if phone: res = await db.execute( select(Business).where( or_( - Business.phone == lead.get("phone"), + Business.phone_e164 == phone, + Business.phone == raw_phone, Business.phone == phone, ) ) @@ -73,7 +85,7 @@ async def find_existing_business(db: AsyncSession, lead: dict) -> Optional[Busin res = await db.execute( select(Contact).where( - or_(Contact.phone == lead.get("phone"), Contact.phone == phone) + or_(Contact.phone == raw_phone, Contact.phone == phone) ) ) contact = res.scalars().first() @@ -85,8 +97,8 @@ async def find_existing_business(db: AsyncSession, lead: dict) -> Optional[Busin res = await db.execute( select(Business).where( or_( + Business.domain == domain, Business.website == lead.get("website"), - Business.website.like(f"%{domain}%"), ) ) ) @@ -94,9 +106,13 @@ async def find_existing_business(db: AsyncSession, lead: dict) -> Optional[Busin if match: return match - key = normalize_name(lead.get("name")) - if key: - res = await db.execute(select(Business).where(Business.name == lead.get("name"))) + # Name matching stays exact and raw on purpose. Indian SMBs have very + # generic names ("Sharma Dental Clinic", "Krishna Medicals"), so a + # looser match - or a normalised one that strips legal suffixes - would + # merge genuinely different businesses in different cities. + name = lead.get("name") + if name: + res = await db.execute(select(Business).where(Business.name == name)) match = res.scalars().first() if match: return match diff --git a/backend/app/enricher/email_finder.py b/backend/app/enricher/email_finder.py index 7d4c372..4b4dcbb 100644 --- a/backend/app/enricher/email_finder.py +++ b/backend/app/enricher/email_finder.py @@ -1,11 +1,14 @@ +import asyncio +import random import httpx from bs4 import BeautifulSoup import re import urllib.parse from urllib.parse import urlparse, urljoin +from urllib.robotparser import RobotFileParser import logging from datetime import datetime, timezone -from typing import Dict, List, Set, Any, Optional +from typing import Dict, List, Set, Tuple, Any, Optional from sqlalchemy import select from app.database import async_session_maker @@ -24,6 +27,60 @@ logger = logging.getLogger(__name__) +#: User-Agent strings used for outbound requests, rotated between retries. +#: A single static UA is the easiest fingerprint to block on. +#: +#: ``app.sources.http_utils`` already owns the canonical list; it does not +#: import anything from ``app.enricher``, so reusing it here cannot create a +#: circular import. The fallback keeps this module importable on its own. +try: # pragma: no cover - defensive import of a shared constant + from app.sources.http_utils import USER_AGENTS as _SHARED_USER_AGENTS +except Exception: # pragma: no cover + _SHARED_USER_AGENTS = () + +CRAWL_USER_AGENTS: Tuple[str, ...] = tuple(_SHARED_USER_AGENTS) or ( + "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 " + "(KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36", + "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 " + "(KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36", + "Mozilla/5.0 (Linux; Android 14; Pixel 8) AppleWebKit/537.36 " + "(KHTML, like Gecko) Chrome/131.0.0.0 Mobile Safari/537.36", +) + +#: How many pages of a single site are fetched *concurrently*. +#: +#: The page budget is unaffected: the crawler walks the work queue in +#: batches of at most this size and never fetches more than ``max_pages`` +#: distinct URLs in total. +CRAWL_PAGE_CONCURRENCY = 4 + +#: Retries (on top of the first attempt) for a single page fetch. +CRAWL_FETCH_RETRIES = 2 + +#: Retries for a single search-engine query. +SEARCH_FETCH_RETRIES = 2 + +#: Base delay of the exponential backoff between retries (seconds). +RETRY_BACKOFF = 0.8 + +#: Timeout for the robots.txt lookup - it must never dominate a crawl. +ROBOTS_TIMEOUT = 5.0 + +#: Upper bound honoured for a robots.txt ``Crawl-delay`` (seconds). +MAX_CRAWL_DELAY = 3.0 + +#: Token used when matching robots.txt rules against our own requests. +ROBOTS_USER_AGENT = "LeadForgeBot" + +#: Transient failures worth another attempt. +RETRYABLE_STATUS = (429, 500, 502, 503, 504) + +SITE_HEADERS = { + "User-Agent": CRAWL_USER_AGENTS[0], + "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", + "Accept-Language": "en-IN,en-GB;q=0.9,en;q=0.8", +} + EMAIL_REGEX = re.compile(r'[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+') PHONE_REGEX = re.compile(r'(\+?91[\s-]?[6789]\d{9}|0?[6789]\d{4}\s*\d{5}|0?[6789]\d{9}|\(?\d{3}\)?[\s.-]?\d{3}[\s.-]?\d{4})') @@ -35,6 +92,199 @@ # between sentry.io/example.com and a company's actual mail domain. EXCLUDED_EMAIL_DOMAINS = ('sentry.io', 'example.com', 'wixpress.com', 'domain.com', 'email.com', 'schema.org') + +# --------------------------------------------------------------------------- +# HTTP plumbing shared by the search stage and the site crawler +# --------------------------------------------------------------------------- + +def _build_http_client(timeout: float = 12.0) -> httpx.AsyncClient: + """A pooled client with browser-ish headers and *verified* TLS. + + A single client pools connections internally, so one client per crawl + (rather than one per page) is what makes the crawl fast. + + TLS verification is left at httpx's default (on). The old crawler ran + with ``verify=False``, which is both a MITM hole and a bot signal. + """ + return httpx.AsyncClient( + timeout=timeout, + follow_redirects=True, + headers=SITE_HEADERS, + limits=httpx.Limits( + max_connections=CRAWL_PAGE_CONCURRENCY + 2, + max_keepalive_connections=CRAWL_PAGE_CONCURRENCY + 2, + ), + ) + + +def _request_headers(attempt: int) -> Dict[str, str]: + """Headers for one attempt, rotating the User-Agent between retries.""" + ua = CRAWL_USER_AGENTS[attempt % len(CRAWL_USER_AGENTS)] + return { + "User-Agent": ua, + "Accept": SITE_HEADERS["Accept"], + "Accept-Language": SITE_HEADERS["Accept-Language"], + } + + +def _backoff_delay(attempt: int, base: float = RETRY_BACKOFF) -> float: + """Exponential backoff with jitter, so parallel crawls do not sync up.""" + return (base * (2 ** attempt)) + random.uniform(0, 0.5) + + +def _is_tls_error(exc: BaseException) -> bool: + """True when an exception was caused by a certificate problem.""" + text = f"{type(exc).__name__}: {exc}".lower() + return "ssl" in text or "certificate" in text + + +async def _get_with_retry( + client: httpx.AsyncClient, + url: str, + *, + retries: int = CRAWL_FETCH_RETRIES, + timeout: Optional[float] = None, +) -> Optional[httpx.Response]: + """GET with retries, exponential backoff, jitter and UA rotation. + + Only transient failures are retried: connection errors, timeouts and + 429/5xx responses. A 404 (or any other 4xx that will not fix itself) is + returned immediately so the retry budget is not wasted on it. + + Never raises - a failed request comes back as ``None``. + """ + attempts = max(1, retries + 1) + last_error: Optional[str] = None + + for attempt in range(attempts): + try: + kwargs: Dict[str, Any] = {"headers": _request_headers(attempt)} + if timeout is not None: + kwargs["timeout"] = timeout + response = await client.get(url, **kwargs) + except Exception as exc: + last_error = f"{type(exc).__name__}: {exc}" + if _is_tls_error(exc): + logger.warning( + "TLS verification failed for %s (%s). The certificate of " + "this host must be fixed for enrichment to work.", + urlparse(url).netloc, last_error, + ) + if attempt < attempts - 1: + await asyncio.sleep(_backoff_delay(attempt)) + continue + logger.debug("GET %s failed after %d attempts (%s)", url, attempts, last_error) + return None + + if response.status_code in RETRYABLE_STATUS and attempt < attempts - 1: + last_error = f"HTTP {response.status_code}" + logger.debug( + "GET %s -> %s; retrying", url, response.status_code, + ) + await asyncio.sleep(_backoff_delay(attempt)) + continue + + return response + + return None + + +def _normalize_site_url(url: str) -> str: + """Adds the https:// scheme a bare domain is missing.""" + if not url: + return "" + url = url.strip() + if not url.startswith("http://") and not url.startswith("https://"): + url = "https://" + url + return url + + +def _normalize_crawl_url(url: str) -> str: + """Canonical key for a URL: no fragment, no trailing slash, lower-case host. + + Used for de-duplication so ``/contact`` and ``/contact/`` are recognised + as the same page and are never fetched twice. + """ + if not url: + return "" + try: + parsed = urlparse(url.strip()) + except Exception: + return "" + scheme = (parsed.scheme or "https").lower() + netloc = parsed.netloc.lower() + if not netloc: + return "" + path = parsed.path or "/" + if len(path) > 1: + path = path.rstrip("/") + out = f"{scheme}://{netloc}{path}" + if parsed.query: + out += f"?{parsed.query}" + return out + + +async def _robots_parser( + client: httpx.AsyncClient, + origin: str, + cache: Dict[str, Optional[RobotFileParser]], +) -> Optional[RobotFileParser]: + """Fetches and caches ``/robots.txt`` for one origin. + + A missing (or unreadable) robots.txt means "no restrictions", which is + the standard interpretation. + """ + if origin in cache: + return cache[origin] + + parser: Optional[RobotFileParser] = None + try: + response = await client.get(f"{origin}/robots.txt", timeout=ROBOTS_TIMEOUT) + if response.status_code == 200: + parser = RobotFileParser() + parser.parse(response.text.splitlines()) + except Exception as exc: + logger.debug("robots.txt lookup failed for %s: %s", origin, exc) + + cache[origin] = parser + return parser + + +async def _robots_allows( + client: httpx.AsyncClient, + url: str, + cache: Dict[str, Optional[RobotFileParser]], +) -> bool: + """Whether robots.txt permits us to fetch ``url``.""" + parsed = urlparse(url) + origin = f"{parsed.scheme}://{parsed.netloc}" + parser = await _robots_parser(client, origin, cache) + if parser is None: + return True + try: + return bool(parser.can_fetch(ROBOTS_USER_AGENT, url)) + except Exception: + return True + + +async def _respect_crawl_delay( + client: httpx.AsyncClient, + url: str, + cache: Dict[str, Optional[RobotFileParser]], +) -> None: + """Honours a robots.txt ``Crawl-delay`` (capped so it cannot hang a crawl).""" + parsed = urlparse(url) + origin = f"{parsed.scheme}://{parsed.netloc}" + parser = await _robots_parser(client, origin, cache) + if parser is None: + return + try: + delay = parser.crawl_delay(ROBOTS_USER_AGENT) + except Exception: + delay = None + if delay and delay > 0: + await asyncio.sleep(min(float(delay), MAX_CRAWL_DELAY)) + #: Call-to-action and boilerplate phrases that the title regex used to #: mistake for people's names. "Read more - owner" became the contact #: "read more" with title "owner", and that garbage then consumed the @@ -223,6 +473,75 @@ def calculate_lead_quality_score(business: Any, contacts: List[Any]) -> Dict[str return result +def _extract_contacts_from_text(text: str, emails: Set[str], phones: Set[str]) -> None: + """Adds every email / phone number found in a block of text.""" + for em in EMAIL_REGEX.findall(text): + em_clean = em.lower().strip(".") + if any(em_clean.endswith(ext) for ext in EXCLUDED_EMAIL_EXTENSIONS): + continue + domain = em_clean.split("@")[-1] if "@" in em_clean else "" + if domain and not any(ex in domain for ex in EXCLUDED_EMAIL_DOMAINS): + emails.add(em_clean) + + for ph in PHONE_REGEX.findall(text): + clean_ph = ph[0] if isinstance(ph, tuple) else ph + digits = re.sub(r'\D', '', clean_ph) + if 10 <= len(digits) <= 13: + phones.add(clean_ph.strip()) + + +async def _search_yahoo( + client: httpx.AsyncClient, + search_query: str, + emails: Set[str], + phones: Set[str], + websites: List[str], +) -> None: + """Engine 1: Yahoo Search.""" + try: + y_url = f"https://search.yahoo.com/search?p={urllib.parse.quote(search_query)}" + res = await _get_with_retry(client, y_url, retries=SEARCH_FETCH_RETRIES) + if res is None or res.status_code != 200: + return + soup_y = BeautifulSoup(res.text, "html.parser") + for item in soup_y.select("div.algo"): + text = item.get_text(" ", strip=True) + _extract_contacts_from_text(text, emails, phones) + link = item.select_one("a[href^='http']") + if link: + href = link.get("href", "") + m = re.search(r'/RU=(.*?)/RK=', href) + target_url = urllib.parse.unquote(m.group(1)) if m else href + if target_url.startswith("http") and not any(x in target_url for x in [ + "yahoo.com", "bing.com", "google.com", "facebook.com", + "instagram.com", "twitter.com", "youtube.com" + ]): + websites.append(target_url) + except Exception as y_err: + logger.debug(f"Yahoo search error: {y_err}") + + +async def _search_bing( + client: httpx.AsyncClient, + clean_name: str, + clean_location: str, + emails: Set[str], + phones: Set[str], +) -> None: + """Engine 2: Bing Search.""" + try: + b_url = f"https://www.bing.com/search?q={urllib.parse.quote(f'{clean_name} {clean_location} phone email')}" + res = await _get_with_retry(client, b_url, retries=SEARCH_FETCH_RETRIES) + if res is None or res.status_code != 200: + return + soup_b = BeautifulSoup(res.text, "html.parser") + for item in soup_b.select("li.b_algo"): + text = item.get_text(" ", strip=True) + _extract_contacts_from_text(text, emails, phones) + except Exception as b_err: + logger.debug(f"Bing search error: {b_err}") + + async def search_web_for_business_contacts(name: str, location: str = "") -> Dict[str, Any]: """ Searches across Yahoo, Bing, and web directories to discover missing business @@ -237,60 +556,21 @@ async def search_web_for_business_contacts(name: str, location: str = "") -> Dic clean_location = re.sub(r'[:|\-·]+.*$', '', location).strip() search_query = f'"{clean_name}" {clean_location} phone contact email'.strip() - headers = { - "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36", - "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", - } - - def _extract_from_text(text: str): - for em in EMAIL_REGEX.findall(text): - em_clean = em.lower().strip(".") - if not any(em_clean.endswith(ext) for ext in EXCLUDED_EMAIL_EXTENSIONS): - domain = em_clean.split("@")[-1] if "@" in em_clean else "" - if domain and not any(ex in domain for ex in EXCLUDED_EMAIL_DOMAINS): - emails.add(em_clean) - - for ph in PHONE_REGEX.findall(text): - clean_ph = ph[0] if isinstance(ph, tuple) else ph - digits = re.sub(r'\D', '', clean_ph) - if 10 <= len(digits) <= 13: - phones.add(clean_ph.strip()) - try: - async with httpx.AsyncClient(headers=headers, timeout=12.0, follow_redirects=True, verify=False) as client: - # Engine 1: Yahoo Search - try: - y_url = f"https://search.yahoo.com/search?p={urllib.parse.quote(search_query)}" - res_y = await client.get(y_url) - if res_y.status_code == 200: - soup_y = BeautifulSoup(res_y.text, "html.parser") - for item in soup_y.select("div.algo"): - text = item.get_text(" ", strip=True) - _extract_from_text(text) - link = item.select_one("a[href^='http']") - if link: - href = link.get("href", "") - m = re.search(r'/RU=(.*?)/RK=', href) - target_url = urllib.parse.unquote(m.group(1)) if m else href - if target_url.startswith("http") and not any(x in target_url for x in [ - "yahoo.com", "bing.com", "google.com", "facebook.com", - "instagram.com", "twitter.com", "youtube.com" - ]): - websites.append(target_url) - except Exception as y_err: - logger.debug(f"Yahoo search error for '{name}': {y_err}") - - # Engine 2: Bing Search - try: - b_url = f"https://www.bing.com/search?q={urllib.parse.quote(f'{clean_name} {clean_location} phone email')}" - res_b = await client.get(b_url) - if res_b.status_code == 200: - soup_b = BeautifulSoup(res_b.text, "html.parser") - for item in soup_b.select("li.b_algo"): - text = item.get_text(" ", strip=True) - _extract_from_text(text) - except Exception as b_err: - logger.debug(f"Bing search error for '{name}': {b_err}") + async with _build_http_client(timeout=12.0) as client: + # Yahoo and Bing are independent requests to two different hosts. + # They used to be awaited one after the other, so every lead paid + # for the sum of both engines' latencies. Running them together + # costs the slower engine instead of the sum - and neither engine + # can take the other one down with it. + results = await asyncio.gather( + _search_yahoo(client, search_query, emails, phones, websites), + _search_bing(client, clean_name, clean_location, emails, phones), + return_exceptions=True, + ) + for res in results: + if isinstance(res, BaseException): + logger.debug("Web search engine failed: %s", res) except Exception as e: logger.warning(f"Web search discovery error for '{name}': {e}") @@ -302,174 +582,375 @@ def _extract_from_text(text: str): } -async def crawl_website_for_contacts(url: str, max_pages: int = 4) -> Dict[str, Any]: +#: Titles that actually appear on Indian SMB sites. The old list missed +#: MD, Principal (e.g. "Principal Dentist" - the dental norm), General +#: Manager, Promoter and Chairman, which together cover the majority of +#: owner-operator titles in the target market. +_EXEC_TITLES = [ + r'founder', r'co-founder', r'ceo', r'chief executive officer', + r'managing director', r'md', r'director', r'principal(?:\s+\w+)?', + r'general manager', r'manager', r'promoter', r'chairman', + r'chairperson', r'president', r'owner', r'partner', r'proprietor', + r'administrator', r'head of \w+', r'vice president', r'vp', +] + +#: The separator class includes the em-dash (U+2014), en-dash (U+2013) and +#: bullet (U+2022) - all three are used by Indian SMB sites and all three +#: were silently failing to split name from title. +_TITLE_PATTERN = re.compile( + r'([A-Z][a-zA-Z]+(?:\s+[A-Z][a-zA-Z]+){1,2})\s*' + r'(?:[-\u2013\u2014\u2022|,;:()]|\bis\b|\bas\b|&)\s*(' + + '|'.join(_EXEC_TITLES) + r')\b', + re.IGNORECASE +) + + +def _is_x_host(href: str, host: str) -> bool: + """Whether a link points at ``host`` (or one of its sub-domains). + + The old check was a plain ``"x.com" in href`` substring test, which + matched every URL that merely *contained* those characters - so a link + to ``https://www.max.com/`` was filed as a Twitter profile. """ - Crawls a target business website, prioritizing contact/about subpages, - and extracts emails, phone numbers, WhatsApp links, and social links. + try: + parsed = urlparse(href) + except Exception: + return False + netloc = (parsed.netloc or "").lower().split(":")[0] + if not netloc: + return False + return netloc == host or netloc.endswith("." + host) + + +def _extract_page_contacts( + html_text: str, + current_url: str, + base_domain: str, +) -> Dict[str, Any]: + """Every contact signal on one page of HTML. + + Pure function (no network), so the crawler can run it over pages it + fetched concurrently without any shared mutable state. """ - if not url: - return {"emails": [], "phones": [], "whatsapp": None, "socials": {}} + emails: Set[str] = set() + phones: Set[str] = set() + whatsapp_links: Set[str] = set() + socials: Dict[str, str] = {} + executives: List[Dict[str, str]] = [] + internal_links: List[str] = [] - if not url.startswith("http://") and not url.startswith("https://"): - url = "https://" + url + # 1. Regex email search + deobfuscated text search + normalized_text = deobfuscate_text_emails(html_text) + for em in EMAIL_REGEX.findall(normalized_text): + em_clean = em.lower().strip(".") + if any(em_clean.endswith(ext) for ext in EXCLUDED_EMAIL_EXTENSIONS): + continue + domain = em_clean.split("@")[-1] if "@" in em_clean else "" + if domain and not any(ex in domain for ex in EXCLUDED_EMAIL_DOMAINS): + emails.add(em_clean) + + # 2. Regex phone search + for ph in PHONE_REGEX.findall(html_text): + clean_ph = ph[0] if isinstance(ph, tuple) else ph + if len(re.sub(r'\D', '', clean_ph)) >= 10: + phones.add(clean_ph.strip()) + + # 3. DOM parsing for Cloudflare emails, mailto:, tel:, WhatsApp, and Socials + soup = BeautifulSoup(html_text, 'html.parser') + + # Cloudflare email protection decode + for cf_el in soup.select('[data-cfemail], .__cf_email__'): + cf_hex = cf_el.get('data-cfemail') or cf_el.get('href', '').split('#')[-1] + if cf_hex: + cf_email = decode_cloudflare_email(cf_hex) + if cf_email and not any(ex in cf_email for ex in EXCLUDED_EMAIL_DOMAINS): + emails.add(cf_email) + + # Executive detection on about/team pages + text_clean = soup.get_text(" ", strip=True) + for match in _TITLE_PATTERN.finditer(text_clean): + found_name, found_title = match.group(1).strip(), match.group(2).strip() + if not _is_plausible_person_name(found_name): + continue + executives.append({"name": found_name, "title": found_title.title()}) + + for link in soup.find_all('a', href=True): + href = link['href'].strip() + lowered = href.lower() + + # Mailto links + if lowered.startswith('mailto:'): + em = href[href.lower().index('mailto:') + len('mailto:'):].split('?')[0].strip() + if EMAIL_REGEX.match(em): + em_clean = em.lower() + domain = em_clean.rpartition("@")[2] + if domain and not any(ex in domain for ex in EXCLUDED_EMAIL_DOMAINS): + emails.add(em_clean) + + # Tel links + elif lowered.startswith('tel:'): + ph = href[href.lower().index('tel:') + len('tel:'):].split('?')[0].strip() + if len(re.sub(r'\D', '', ph)) >= 10: + phones.add(ph) + + # WhatsApp links + elif 'wa.me' in href or 'api.whatsapp.com' in href: + whatsapp_links.add(href) + + # Social Profiles + elif 'linkedin.com/company' in href or 'linkedin.com/in' in href: + socials['linkedin'] = href + elif 'facebook.com' in href and not any(x in href for x in ['sharer', 'dialog']): + socials['facebook'] = href + elif 'instagram.com' in href and not any(x in href for x in ['share', 'p/']): + socials['instagram'] = href + elif 'twitter.com' in href or _is_x_host(href, 'x.com'): + socials['twitter'] = href + elif 'youtube.com' in href: + socials['youtube'] = href + + # Internal page discovery - external links are never followed, so the + # page budget is never spent on somebody else's website. + else: + full_next = urljoin(current_url, href) + next_parsed = urlparse(full_next) + next_domain = next_parsed.netloc.lower().replace("www.", "") + + if next_domain == base_domain: + internal_links.append(full_next) + + return { + "emails": emails, + "phones": phones, + "whatsapp": whatsapp_links, + "socials": socials, + "executives": executives, + "internal_links": internal_links, + } + +async def _fetch_page_html(client: httpx.AsyncClient, url: str) -> Optional[str]: + """Fetches one page and returns its HTML (``None`` on any failure).""" + response = await _get_with_retry(client, url, retries=CRAWL_FETCH_RETRIES) + if response is None or response.status_code != 200: + return None + content_type = response.headers.get("content-type", "") + # A missing content-type is not a reason to throw a real page away; a + # non-HTML content type is. + if content_type and ("text/html" not in content_type and "xhtml" not in content_type): + return None + return response.text + + +async def _crawl_with_client( + client: httpx.AsyncClient, + url: str, + max_pages: int, + homepage_html: Optional[str], +) -> Dict[str, Any]: + """The crawl itself. See :func:`crawl_website_for_contacts`.""" emails: Set[str] = set() phones: Set[str] = set() whatsapp_links: Set[str] = set() socials: Dict[str, str] = {} - visited: Set[str] = set() executives: List[Dict[str, str]] = [] parsed_base = urlparse(url) base_domain = parsed_base.netloc.lower().replace("www.", "") - to_visit: List[str] = [url] + #: Normalised URLs that have been *scheduled*. A URL is reserved BEFORE + #: the request goes out, so a page can never be fetched twice - not when + #: the same link appears twice on a page, and not when two pages of the + #: same batch both link to it. + visited: Set[str] = set() + #: URLs already sitting in one of the queues (stops duplicate entries). + queued: Set[str] = set() priority_queue: List[str] = [] + to_visit: List[str] = [] + robots_cache: Dict[str, Optional[RobotFileParser]] = {} + + #: The homepage is handled on its own (it may already have been supplied + #: by the caller) so its links can seed the next batch. + pending: List[Tuple[str, Optional[str]]] = [(url, homepage_html)] + attempted = 0 + homepage_used = homepage_html + + while attempted < max_pages and (pending or priority_queue or to_visit): + batch: List[Tuple[str, Optional[str]]] = [] + batch_size = max(1, min(CRAWL_PAGE_CONCURRENCY, max_pages - attempted)) + + if pending: + batch = pending[:1] + pending = [] + else: + # Contact/about pages first (same priority the serial crawler + # used), then whatever else the site links to. + while priority_queue and len(batch) < batch_size: + batch.append((priority_queue.pop(0), None)) + while to_visit and len(batch) < batch_size: + batch.append((to_visit.pop(0), None)) + + if not batch: + break + + # Reserve the URLs *before* fetching them. + targets: List[Tuple[str, Optional[str]]] = [] + for target, pre_html in batch: + norm = _normalize_crawl_url(target) + if not norm or norm in visited: + continue + visited.add(norm) + targets.append((target, pre_html)) - headers = { - "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36", - "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" - } - - # Titles that actually appear on Indian SMB sites. The old list missed - # MD, Principal (e.g. "Principal Dentist" - the dental norm), General - # Manager, Promoter and Chairman, which together cover the majority of - # owner-operator titles in the target market. - EXEC_TITLES = [ - r'founder', r'co-founder', r'ceo', r'chief executive officer', - r'managing director', r'md', r'director', r'principal(?:\s+\w+)?', - r'general manager', r'manager', r'promoter', r'chairman', - r'chairperson', r'president', r'owner', r'partner', r'proprietor', - r'administrator', r'head of \w+', r'vice president', r'vp', - ] - TITLE_PATTERN = re.compile( - # The separator class now includes the em-dash (U+2014), en-dash - # (U+2013) and bullet (U+2022) - all three are used by Indian SMB - # sites and all three were silently failing to split name from title. - r'([A-Z][a-zA-Z]+(?:\s+[A-Z][a-zA-Z]+){1,2})\s*' - r'(?:[-\u2013\u2014\u2022|,;:()]|\bis\b|\bas\b|&)\s*(' - + '|'.join(EXEC_TITLES) + r')\b', - re.IGNORECASE - ) + if not targets: + continue - async with httpx.AsyncClient(timeout=12.0, follow_redirects=True, verify=False, headers=headers) as client: - while (priority_queue or to_visit) and len(visited) < max_pages: - current_url = priority_queue.pop(0) if priority_queue else to_visit.pop(0) - - clean_curr = current_url.split("#")[0].rstrip("/") - if clean_curr in visited: + # robots.txt + allowed: List[Tuple[str, Optional[str]]] = [] + for target, pre_html in targets: + if pre_html is not None: + allowed.append((target, pre_html)) continue - visited.add(clean_curr) + if await _robots_allows(client, target, robots_cache): + allowed.append((target, None)) + else: + logger.debug("robots.txt disallows %s", target) - try: - response = await client.get(current_url) - if response.status_code != 200: - continue + if not allowed: + continue - content_type = response.headers.get("content-type", "") - if "text/html" not in content_type: - continue + attempted += len(allowed) + await _respect_crawl_delay(client, allowed[0][0], robots_cache) - html_text = response.text - - # 1. Regex email search + deobfuscated text search - normalized_text = deobfuscate_text_emails(html_text) - raw_emails = EMAIL_REGEX.findall(normalized_text) - for em in raw_emails: - em_clean = em.lower().strip(".") - if not any(em_clean.endswith(ext) for ext in EXCLUDED_EMAIL_EXTENSIONS): - domain = em_clean.split("@")[-1] if "@" in em_clean else "" - if domain and not any(ex in domain for ex in EXCLUDED_EMAIL_DOMAINS): - emails.add(em_clean) - - # 2. Regex phone search - raw_phones = PHONE_REGEX.findall(html_text) - for ph in raw_phones: - clean_ph = ph[0] if isinstance(ph, tuple) else ph - if len(re.sub(r'\D', '', clean_ph)) >= 10: - phones.add(clean_ph.strip()) - - # 3. DOM parsing for Cloudflare emails, mailto:, tel:, WhatsApp, and Socials - soup = BeautifulSoup(html_text, 'html.parser') - - # Cloudflare email protection decode - for cf_el in soup.select('[data-cfemail], .__cf_email__'): - cf_hex = cf_el.get('data-cfemail') or cf_el.get('href', '').split('#')[-1] - if cf_hex: - cf_email = decode_cloudflare_email(cf_hex) - if cf_email and not any(ex in cf_email for ex in EXCLUDED_EMAIL_DOMAINS): - emails.add(cf_email) - - # Executive detection on about/team pages - text_clean = soup.get_text(" ", strip=True) - for match in TITLE_PATTERN.finditer(text_clean): - found_name, found_title = match.group(1).strip(), match.group(2).strip() - if not _is_plausible_person_name(found_name): - continue - executives.append({"name": found_name, "title": found_title.title()}) - - for link in soup.find_all('a', href=True): - href = link['href'].strip() - - # Mailto links - if href.lower().startswith('mailto:'): - em = href.split('mailto:')[1].split('?')[0].strip() - if EMAIL_REGEX.match(em): - em_clean = em.lower() - domain = em_clean.split("@")[-1] - if not any(ex in domain for ex in EXCLUDED_EMAIL_DOMAINS): - emails.add(em_clean) - - # Tel links - elif href.lower().startswith('tel:'): - ph = href.split('tel:')[1].split('?')[0].strip() - if len(re.sub(r'\D', '', ph)) >= 10: - phones.add(ph) - - # WhatsApp links - elif 'wa.me' in href or 'api.whatsapp.com' in href: - whatsapp_links.add(href) - - # Social Profiles - elif 'linkedin.com/company' in href or 'linkedin.com/in' in href: - socials['linkedin'] = href - elif 'facebook.com' in href and not any(x in href for x in ['sharer', 'dialog']): - socials['facebook'] = href - elif 'instagram.com' in href and not any(x in href for x in ['share', 'p/']): - socials['instagram'] = href - elif 'twitter.com' in href or 'x.com' in href: - socials['twitter'] = href - elif 'youtube.com' in href: - socials['youtube'] = href - - # Internal page discovery - else: - full_next = urljoin(current_url, href) - next_parsed = urlparse(full_next) - next_domain = next_parsed.netloc.lower().replace("www.", "") - - if next_domain == base_domain: - path_lower = next_parsed.path.lower() - if any(kw in path_lower for kw in CONTACT_PRIORITY_KEYWORDS): - if full_next not in visited and full_next not in priority_queue: - priority_queue.append(full_next) - else: - if full_next not in visited and full_next not in to_visit and len(to_visit) < 10: - to_visit.append(full_next) + # Pages the caller already fetched are used as-is; the rest are + # fetched concurrently over the one pooled client. + html_results: List[Optional[str]] = [None] * len(allowed) + fetch_indexes: List[int] = [] + for idx, (_target, pre_html) in enumerate(allowed): + if pre_html is not None: + html_results[idx] = pre_html + else: + fetch_indexes.append(idx) - except Exception as e: - logger.debug(f"Error crawling {current_url}: {e}") + if fetch_indexes: + if len(fetch_indexes) == 1: + fetched = [await _fetch_page_html(client, allowed[fetch_indexes[0]][0])] + else: + fetched = await asyncio.gather( + *[_fetch_page_html(client, allowed[i][0]) for i in fetch_indexes] + ) + for idx, html_text in zip(fetch_indexes, fetched): + html_results[idx] = html_text + + for (target, _pre_html), html_text in zip(allowed, html_results): + if not html_text: continue + if _normalize_crawl_url(target) == _normalize_crawl_url(url) and homepage_used is None: + homepage_used = html_text + + data = _extract_page_contacts(html_text, target, base_domain) + emails.update(data["emails"]) + phones.update(data["phones"]) + whatsapp_links.update(data["whatsapp"]) + socials.update(data["socials"]) + executives.extend(data["executives"]) + + for link in data["internal_links"]: + norm = _normalize_crawl_url(link) + if not norm or norm in visited or norm in queued: + continue + queued.add(norm) + path_lower = urlparse(link).path.lower() + if any(kw in path_lower for kw in CONTACT_PRIORITY_KEYWORDS): + priority_queue.append(link) + elif len(to_visit) < 10: + to_visit.append(link) return { "emails": list(emails), "phones": list(phones), "whatsapp": list(whatsapp_links)[0] if whatsapp_links else None, "socials": socials, - "executives": executives[:5] + "executives": executives[:5], + "homepage_html": homepage_used, } +async def crawl_website_for_contacts( + url: str, + max_pages: int = 4, + homepage_html: Optional[str] = None, + client: Optional[httpx.AsyncClient] = None, +) -> Dict[str, Any]: + """ + Crawls a target business website, prioritizing contact/about subpages, + and extracts emails, phone numbers, WhatsApp links, and social links. + + ``homepage_html`` lets the caller supply a homepage it has already + fetched, so the homepage is downloaded once per lead and can be shared + with the social finder instead of being fetched twice. + + ``client`` lets the caller supply a pooled client. When omitted, one + client is created and reused for every page of the crawl. + """ + if not url: + return {"emails": [], "phones": [], "whatsapp": None, "socials": {}, + "executives": [], "homepage_html": None} + + url = _normalize_site_url(url) + + if client is not None: + return await _crawl_with_client(client, url, max_pages, homepage_html) + + async with _build_http_client(timeout=12.0) as owned_client: + return await _crawl_with_client(owned_client, url, max_pages, homepage_html) + + +async def _probe_business_health_concurrent(business: Any, country: str = "IN") -> Dict[str, Any]: + """Runs the two network probes of ``probe_business_health`` in parallel. + + ``app.qualifier.signals.probe_business_health`` awaits the website + health probe and *then* the Facebook Ad Library probe. The two are + completely independent - a HEAD/GET of the company's own site and a + fetch of a public Facebook page - so serialising them doubles the + network wait of every enriched lead. + + ``app/qualifier/`` is outside the enricher package, so instead of + editing it we call the very same probes here with ``asyncio.gather``. + The merged patch has exactly the same shape as the serial version. + """ + try: + from app.qualifier import signals + except Exception: + from app.qualifier.signals import probe_business_health + return await probe_business_health(business, country=country) + + website_probe = getattr(signals, "_probe_website_health", None) + ads_probe = getattr(signals, "_probe_facebook_ads", None) + if not callable(website_probe) or not callable(ads_probe): + # The probe module changed shape - fall back to its own entry point. + return await signals.probe_business_health(business, country=country) + + website = getattr(business, "website", None) + name = getattr(business, "name", "") or "" + + jobs = [] + if website: + jobs.append(website_probe(website)) + jobs.append(ads_probe(name, country=country)) + + results = await asyncio.gather(*jobs, return_exceptions=True) + + patch: Dict[str, Any] = {} + for res in results: + if isinstance(res, BaseException): + logger.debug("Health probe failed: %s", res) + continue + if isinstance(res, dict): + patch.update(res) + return patch + + async def enrich_business_by_id(business_id: int) -> Dict[str, Any]: """ Multi-source lead enrichment: @@ -502,10 +983,25 @@ async def enrich_business_by_id(business_id: int) -> Dict[str, Any]: logger.info(f"Discovered official website for {business.name}: {business.website}") # Step 2: Crawl website if present + # + # The homepage is downloaded ONCE here and the same HTML is handed to + # both the crawler and the social finder below. It used to be fetched + # twice per lead (once by the crawler, once by social_finder) for the + # exact same page. crawl_res: Dict[str, Any] = {} + homepage_html: Optional[str] = None if business.website: try: - crawl_res = await crawl_website_for_contacts(business.website) + site_url = _normalize_site_url(business.website) + async with _build_http_client(timeout=12.0) as site_client: + homepage_html = await _fetch_page_html(site_client, site_url) + crawl_res = await crawl_website_for_contacts( + site_url, + homepage_html=homepage_html, + client=site_client, + ) + if not homepage_html: + homepage_html = crawl_res.get("homepage_html") for em in crawl_res.get("emails", []): all_emails.add(em) for ph in crawl_res.get("phones", []): @@ -527,7 +1023,14 @@ async def enrich_business_by_id(business_id: int) -> Dict[str, Any]: ) from app.config import settings - merged_socials = await find_business_socials(business, contact_socials=discovered_socials) + # An empty string means "we already fetched the homepage and got + # nothing" - it stops the social finder from downloading the same + # page a second time. + merged_socials = await find_business_socials( + business, + homepage_html=homepage_html if homepage_html is not None else "", + contact_socials=discovered_socials, + ) provenance = "website" if settings.SOCIAL_DIRECT: direct_socials = await probe_direct_socials(business) @@ -547,6 +1050,11 @@ async def enrich_business_by_id(business_id: int) -> Dict[str, Any]: batch: Dict[str, Any] = {} learned_pattern: Optional[str] = None learned_confidence: float = 0.0 + #: Defined here (not only inside the ``business.website`` branch): + #: a lead with no website but an email from the web search used to + #: raise NameError in the contact stage below and lose the whole lead. + permutations: List[str] = [] + w_domain: Optional[str] = None if business.website: try: web_for_domain = business.website if business.website.startswith("http") else f"https://{business.website}" @@ -781,8 +1289,7 @@ async def enrich_business_by_id(business_id: int) -> Dict[str, Any]: # probed for ICP-driven enrichment and cached so scoring stays repeatable. if icp_profile is not None: try: - from app.qualifier.signals import probe_business_health - health_patch = await probe_business_health(business) + health_patch = await _probe_business_health_concurrent(business) base_extra = dict(business.extra_data) if (business.extra_data and isinstance(business.extra_data, dict)) else {} base_extra.update({k: v for k, v in health_patch.items() if v is not None}) business.extra_data = base_extra diff --git a/backend/app/enricher/social_finder.py b/backend/app/enricher/social_finder.py index 6fc2c28..ba6d3f6 100644 --- a/backend/app/enricher/social_finder.py +++ b/backend/app/enricher/social_finder.py @@ -1,4 +1,5 @@ import json +import asyncio import logging import re from typing import Any, Dict, List, Optional @@ -217,10 +218,12 @@ async def probe_direct_socials(business: Any, timeout: float = 10.0) -> Dict[str "linkedin": f"https://www.linkedin.com/search/results/companies/?keywords={encoded}", } - for platform, url in searches.items(): + # The three platform searches are independent network calls, so they run + # concurrently instead of one after the other. + async def _search_one(platform: str, url: str) -> Optional[str]: html = await fetch_html(url, timeout=timeout) if not html: - continue + return None try: soup = BeautifulSoup(html, "html.parser") for link in soup.find_all("a", href=True): @@ -229,11 +232,18 @@ async def probe_direct_socials(business: Any, timeout: float = 10.0) -> Dict[str continue lowered = href.lower() if any(host in lowered for host in PLATFORM_HOSTS[platform]): - candidates.setdefault(platform, href) - break + return href except Exception as e: logger.debug(f"[social:direct] {platform} search failed: {e}") - continue + return None + + results = await asyncio.gather( + *(_search_one(platform, url) for platform, url in searches.items()), + return_exceptions=True, + ) + for platform, found in zip(searches.keys(), results): + if isinstance(found, str) and found: + candidates.setdefault(platform, found) return candidates @@ -248,9 +258,20 @@ async def upsert_social_profiles( from app.models.social_profile import SocialProfile saved = [] - for platform, url in socials.items(): - if not url: - continue + # The per-platform stat fetches are independent network calls, so they + # run concurrently. Sequentially, three platforms x ~10s of timeout is + # ~30 seconds per lead. + urls = {platform: url for platform, url in socials.items() if url} + stats_results = await asyncio.gather( + *(fetch_social_stats(url, platform) for platform, url in urls.items()), + return_exceptions=True, + ) + stats_by_platform = { + platform: (result if isinstance(result, dict) else {}) + for platform, result in zip(urls.keys(), stats_results) + } + + for platform, url in urls.items(): existing = await db.execute( select(SocialProfile).where( SocialProfile.business_id == business_id, @@ -259,11 +280,7 @@ async def upsert_social_profiles( ) profile = existing.scalars().first() - stats: Dict[str, Any] = {} - try: - stats = await fetch_social_stats(url, platform) - except Exception as e: - logger.debug(f"[social] stats fetch failed for {platform}: {e}") + stats: Dict[str, Any] = stats_by_platform.get(platform) or {} if profile is None: profile = SocialProfile( diff --git a/backend/app/mcp_server.py b/backend/app/mcp_server.py index 8aca37e..5fd1c45 100755 --- a/backend/app/mcp_server.py +++ b/backend/app/mcp_server.py @@ -14,7 +14,9 @@ import asyncio import logging import urllib.parse -from typing import Dict, Any, List, Optional +from collections import defaultdict +from datetime import datetime +from typing import Dict, Any, List, Optional, Sequence, Set, Tuple # Ensure backend directory is in python search path sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))) @@ -32,7 +34,7 @@ from app.scraper.maps_scraper import scrape_leads_and_save from app.enricher.email_finder import enrich_business_by_id from app.campaigns.sender import execute_campaign_step, test_smtp_connection -from app.campaigns.suppression import is_suppressed, add_suppression +from app.campaigns.suppression import add_suppression, filter_suppressed from app.verifier.smtp_verifier import verify_email_async from app.verifier.email_status import should_send @@ -45,11 +47,26 @@ # asyncio tasks). Mirrors the API's BackgroundTasks launch, which also detaches. _BACKGROUND_TASKS = set() +#: How long the stdio loop waits for in-flight requests to write their +#: response after stdin reaches EOF, before it cancels the stragglers. +_SHUTDOWN_GRACE_SECONDS = 10.0 + +#: Progress percentage per scrape-job status, so a polling agent gets a +#: number it can chart instead of guessing from a string. +_SCRAPE_JOB_PROGRESS = { + "pending": 0, + "running": 20, + "scraping": 40, + "enriching": 80, + "completed": 100, + "failed": 100, +} + TOOLS_DEFINITIONS = [ { "name": "leadforge_scrape", - "description": "Scrapes business listings from Google Maps in real-time, extracts phone numbers, ratings, addresses, and official websites, automatically discovers emails from across the open web, and saves them to the database.", + "description": "Starts a Google Maps scrape in the BACKGROUND and returns a job handle immediately. Scrapes business listings in real-time, extracts phone numbers, ratings, addresses and official websites, discovers emails from across the open web, and saves everything to the database. Because a full scrape plus enrichment takes minutes (far past the ~60s MCP client tool timeout) this tool does NOT block: it returns a job_id and you must poll `leadforge_get_scrape_job` until the status is 'completed'.", "inputSchema": { "type": "object", "properties": { @@ -71,6 +88,20 @@ "required": ["query"] } }, + { + "name": "leadforge_get_scrape_job", + "description": "Polls a scrape job started by `leadforge_scrape`. Returns the job status, progress percentage, how many leads were found and saved, and - once the job is 'completed' - the leads it produced with their Contact IDs (use those with `leadforge_add_campaign_targets`; a Contact ID is NOT the Business ID).", + "inputSchema": { + "type": "object", + "properties": { + "job_id": { + "type": "integer", + "description": "The job_id returned by `leadforge_scrape`" + } + }, + "required": ["job_id"] + } + }, { "name": "leadforge_search_leads", "description": "Searches and filters existing businesses in the LeadForge database with contact emails, phones, and hyperlinked Google Maps listings.", @@ -284,104 +315,266 @@ ] +# --------------------------------------------------------------------------- +# Query helpers +# --------------------------------------------------------------------------- + +async def _contacts_by_business( + db: Any, business_ids: Sequence[int] +) -> Dict[int, List[Contact]]: + """Fetches the contacts of every business on a page in ONE query. + + The handlers used to run ``SELECT ... FROM contacts WHERE business_id = ?`` + once per business, so a 15-row page cost 16 round-trips instead of 2. On + a slow link (a self-hosted box on a Raspberry Pi, or Postgres over the + internet) that is the difference between 40ms and 900ms for the same + page - and it scaled linearly with the page size. + + Returns a dict keyed by ``business_id``; businesses with no contacts are + simply absent from it. + """ + ids: List[int] = [] + seen: Set[int] = set() + for bid in business_ids or []: + if bid is None: + continue + bid = int(bid) + if bid not in seen: + seen.add(bid) + ids.append(bid) + if not ids: + return {} + + rows = (await db.execute( + select(Contact).where(Contact.business_id.in_(ids)) + )).scalars().all() + + grouped: Dict[int, List[Contact]] = defaultdict(list) + for c in rows: + grouped[c.business_id].append(c) + return dict(grouped) + + +def _contact_summary(contacts: Sequence[Contact]) -> Dict[str, Any]: + """The contact fields an agent needs to target a lead.""" + return { + "contact_ids": [c.id for c in contacts], + "primary_contact_id": next( + (c.id for c in contacts if c.email), + contacts[0].id if contacts else None, + ), + "emails": [ + f"`{c.email}`" + (" (✓ deliverable)" if c.is_verified else "") + for c in contacts if c.email + ], + } + + +# --------------------------------------------------------------------------- +# Background scrape job +# --------------------------------------------------------------------------- + +async def _mark_scrape_job( + job_id: int, *, status: str, error: Optional[str] = None +) -> None: + """Records a terminal state on a scrape job row. Never raises.""" + try: + async with async_session_maker() as db: + job = await db.get(ScrapeJob, job_id) + if job is None: + return + # Never downgrade a finished job. + if job.status in ("completed", "failed") and status != job.status: + return + job.status = status + if error: + job.error = error + if status in ("completed", "failed"): + job.finished_at = datetime.utcnow() + await db.commit() + except Exception: + logger.exception("Could not update scrape job %s", job_id) + + +async def _run_scrape_job( + job_id: int, query: str, max_results: int, auto_enrich: bool +) -> None: + """Runs a scrape in the background and records the outcome on the job row. + + The scrape used to run INLINE inside the tool call, which held the + JSON-RPC read loop for the whole scrape + enrichment waterfall. Every + other request queued behind it, and MCP clients (Claude Desktop) time a + tool call out at ~60s - so the tool always looked like a failure. + """ + try: + saved_count, new_biz_ids = await scrape_leads_and_save( + query=query, + max_results=max_results, + job_id=job_id, + enrich=auto_enrich, + ) + # Remember which businesses this job produced so + # `leadforge_get_scrape_job` can hand the agent the exact leads + # (and their Contact IDs) without a second, lossy search. + try: + async with async_session_maker() as db: + job = await db.get(ScrapeJob, job_id) + if job is not None: + result = dict(job.result or {}) + result["business_ids"] = list(new_biz_ids or []) + result["leads_saved"] = saved_count + job.result = result + if job.status == "completed" and not job.finished_at: + job.finished_at = datetime.utcnow() + await db.commit() + except Exception: + logger.exception("Could not record scrape job %s results", job_id) + except asyncio.CancelledError: + # The server is shutting down mid-scrape. Mark the job so it does + # not sit at "running" forever and mislead a later poll. + try: + await _mark_scrape_job( + job_id, status="failed", error="scrape interrupted (server shut down)" + ) + except Exception: + pass + raise + except Exception as exc: + logger.exception("Scrape job %s failed", job_id) + await _mark_scrape_job( + job_id, status="failed", error=f"{type(exc).__name__}: {exc}" + ) + + +async def _scrape_job_leads(db: Any, job: ScrapeJob) -> List[Dict[str, Any]]: + """The businesses a finished scrape job produced, with their contacts.""" + business_ids = list((job.result or {}).get("business_ids") or []) + if not business_ids: + return [] + + businesses = (await db.execute( + select(Business).where(Business.id.in_(business_ids)).order_by(Business.id) + )).scalars().all() + + contacts_by_biz = await _contacts_by_business(db, [b.id for b in businesses]) + + rows: List[Dict[str, Any]] = [] + for b in businesses: + contacts = contacts_by_biz.get(b.id, []) + summary = _contact_summary(contacts) + clean_phone = (b.phone or "").replace(" ", "").replace("-", "") + maps_url = b.maps_url or ( + "https://www.google.com/maps/search/?api=1&query=" + f"{urllib.parse.quote((b.name + ' ' + (b.address or '')).strip())}" + ) + rows.append({ + "business_id": b.id, + "contact_ids": summary["contact_ids"], + "primary_contact_id": summary["primary_contact_id"], + "name": b.name, + "maps_url": maps_url, + "phone": b.phone or "—", + "whatsapp": f"https://wa.me/{clean_phone}" if clean_phone else "—", + "website": b.website or "—", + "emails": summary["emails"], + "rating": f"★ {b.rating}" if b.rating else "—", + "address": b.address or "—", + }) + return rows + + async def handle_tool_call(name: str, arguments: Dict[str, Any]) -> str: """Executes an MCP tool and returns markdown-formatted response for the AI assistant.""" if name == "leadforge_scrape": query = arguments.get("query") + if not query: + raise ValueError("`query` is required for leadforge_scrape") max_results = min(int(arguments.get("max_results", 10)), 50) - auto_enrich = arguments.get("auto_enrich", True) + auto_enrich = bool(arguments.get("auto_enrich", True)) - # Create scrape job record - job_id = None + # Create the job row FIRST so the caller gets a handle it can poll + # immediately, then run the scrape in the BACKGROUND. + # + # The scrape used to run inline, which held the JSON-RPC read loop for + # the whole scrape + enrichment waterfall (minutes). MCP clients + # (Claude Desktop) time a tool call out at ~60s, so the tool always + # looked like a failure and every other request queued behind it. async with async_session_maker() as db: - job = ScrapeJob(query=query, status="running") + job = ScrapeJob(query=query, status="pending") db.add(job) await db.commit() await db.refresh(job) job_id = job.id - # Run scrape - saved_count, new_biz_ids = await scrape_leads_and_save(query=query, max_results=max_results, job_id=job_id) + task = asyncio.ensure_future( + _run_scrape_job(job_id, query, max_results, auto_enrich) + ) + _BACKGROUND_TASKS.add(task) + task.add_done_callback(_BACKGROUND_TASKS.discard) - # Fetch the leads THIS scrape created. - # - # The previous code returned the last N businesses in the whole - # database while the header said "Scrape Results for: " - so an - # agent that scraped "Dentists in Pune" against a database that - # already held 75 real-estate leads got a table of real-estate agents - # presented as Pune dentists. + return json.dumps({ + "job_id": job_id, + "status": "running", + "query": query, + "max_results": max_results, + "auto_enrich": auto_enrich, + "poll": "leadforge_get_scrape_job", + "message": ( + "Scrape started in the background. Poll `leadforge_get_scrape_job` " + f"with job_id {job_id} until status is 'completed' (or 'failed'). " + "The results, including the Contact IDs you need for " + "`leadforge_add_campaign_targets`, come back with the job." + ), + }, indent=2) + + elif name == "leadforge_get_scrape_job": + job_id = int(arguments["job_id"]) async with async_session_maker() as db: - if new_biz_ids: - stmt = select(Business).where(Business.id.in_(new_biz_ids)).order_by(Business.id) - else: - stmt = select(Business).where(Business.id.is_(None)) - res = await db.execute(stmt) - businesses = res.scalars().all() - - results = [] - for b in businesses: - c_stmt = select(Contact).where(Contact.business_id == b.id) - c_res = await db.execute(c_stmt) - contacts = c_res.scalars().all() - - emails = [ - f"`{c.email}`" + (" (✓ deliverable)" if c.is_verified else "") - for c in contacts if c.email - ] - - clean_phone = (b.phone or "").replace(" ", "").replace("-", "") - wa_link = f"[WhatsApp](https://wa.me/{clean_phone})" if clean_phone else "" - maps_url = b.maps_url or f"https://www.google.com/maps/search/?api=1&query={urllib.parse.quote((b.name + ' ' + (b.address or '')).strip())}" - - results.append({ - "id": b.id, - # Contact ids, so the agent can actually TARGET these - # leads. `add_campaign_targets` needs contact_ids, and - # contact.id != business.id for most rows - an agent that - # assumed they were the same mailed the wrong 79% of the - # database. - "contact_ids": [c.id for c in contacts], - "primary_contact_id": next( - (c.id for c in contacts if c.email), - contacts[0].id if contacts else None, - ), - "name": b.name, - "maps_url": maps_url, - "phone": b.phone or "—", - "whatsapp": wa_link or "—", - "website": b.website or "—", - "emails": "
".join(emails) if emails else "*None*", - "rating": f"★ {b.rating}" if b.rating else "—", - "address": b.address or "—" - }) + job = await db.get(ScrapeJob, job_id) + if not job: + return json.dumps({ + "job_id": job_id, + "error": f"Scrape job #{job_id} not found", + }, indent=2) + + status = job.status or "pending" + payload: Dict[str, Any] = { + "job_id": job.id, + "query": job.query, + "status": status, + "progress_pct": _SCRAPE_JOB_PROGRESS.get(status, 0), + "done": status in ("completed", "failed"), + "total_found": job.total_found or 0, + "leads_saved": job.leads_saved or 0, + "error": job.error, + "created_at": job.created_at.isoformat() if job.created_at else None, + "finished_at": job.finished_at.isoformat() if job.finished_at else None, + } - # Format markdown response - md = [ - f"### LeadForge Google Maps Scrape Results for: \"{query}\"\n", - f"• **Leads Extracted**: {len(results)}", - f"• **New Leads Saved**: {saved_count}\n", - "| Business ID | Contact ID | Business Name | Google Maps | Phone | WhatsApp | Website | Verified Emails | Rating |", - "| :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- |" - ] + if status == "completed": + payload["leads"] = await _scrape_job_leads(db, job) + else: + payload["leads"] = [] - for r in results: - name_clean = r["name"].replace("|", "\\|") - maps_link = f"[View Listing]({r['maps_url']})" - web_link = f"[Website]({r['website']})" if r['website'] != "—" else "—" - contact_id = r["primary_contact_id"] if r["primary_contact_id"] else "—" - md.append( - f"| {r['id']} | {contact_id} | **{name_clean}** | {maps_link} | {r['phone']} " - f"| {r['whatsapp']} | {web_link} | {r['emails']} | {r['rating']} |" + if status == "completed": + payload["message"] = ( + "Scrape finished. Use the `contact_ids` / `primary_contact_id` of " + "each lead with `leadforge_add_campaign_targets` - a Contact ID " + "is NOT the Business ID." + ) + elif status == "failed": + payload["message"] = ( + "Scrape failed. Check `error`, fix the cause and start a new one " + "with `leadforge_scrape`." + ) + else: + payload["message"] = ( + "Scrape still running. Wait a few seconds and poll this tool " + "again with the same job_id." ) - md.append( - "\n> Use the **Contact ID** column with `leadforge_add_campaign_targets` " - "to enroll these leads in a sequence. The Contact ID is NOT the " - "Business ID." - ) - - return "\n".join(md) + return json.dumps(payload, indent=2) elif name == "leadforge_search_leads": search_term = arguments.get("search") @@ -415,16 +608,13 @@ async def handle_tool_call(name: str, arguments: Dict[str, Any]) -> str: res = await db.execute(stmt) businesses = res.scalars().all() + # ONE query for the contacts of the whole page, not one per row. + contacts_by_biz = await _contacts_by_business(db, [b.id for b in businesses]) + results = [] for b in businesses: - c_stmt = select(Contact).where(Contact.business_id == b.id) - c_res = await db.execute(c_stmt) - contacts = c_res.scalars().all() - - emails = [ - f"`{c.email}`" + (" (✓ verified)" if c.is_verified else "") - for c in contacts if c.email - ] + contacts = contacts_by_biz.get(b.id, []) + summary = _contact_summary(contacts) clean_phone = (b.phone or "").replace(" ", "").replace("-", "") wa_link = f"[WhatsApp](https://wa.me/{clean_phone})" if clean_phone else "" @@ -433,16 +623,13 @@ async def handle_tool_call(name: str, arguments: Dict[str, Any]) -> str: results.append({ "id": b.id, # Contact ids are what `add_campaign_targets` needs. - "contact_ids": [c.id for c in contacts], - "primary_contact_id": next( - (c.id for c in contacts if c.email), - contacts[0].id if contacts else None, - ), + "contact_ids": summary["contact_ids"], + "primary_contact_id": summary["primary_contact_id"], "name": b.name, "maps_url": maps_url, "phone": b.phone or "—", "whatsapp": wa_link or "—", - "emails": "
".join(emails) if emails else "*None*", + "emails": "
".join(summary["emails"]) if summary["emails"] else "*None*", "industry": b.industry or "General", "rating": f"★ {b.rating}" if b.rating else "—" }) @@ -604,18 +791,40 @@ async def handle_tool_call(name: str, arguments: Dict[str, Any]) -> str: select(Campaign).order_by(Campaign.created_at.desc()) )).scalars().all() - rows = [] - for camp in camps: - steps = (await db.execute( - select(func.count(CampaignStep.id)).where(CampaignStep.campaign_id == camp.id) - )).scalar() or 0 - targets = (await db.execute( - select(func.count(CampaignTarget.id)).where(CampaignTarget.campaign_id == camp.id) - )).scalar() or 0 - sends = (await db.execute( - select(func.count(Send.id)).where(Send.campaign_id == camp.id) - )).scalar() or 0 - rows.append((camp, steps, targets, sends)) + # THREE grouped aggregate queries for the whole page instead of + # three COUNT queries per campaign. The old shape was 1 + 3N + # round-trips, so a 20-campaign workspace cost 61 queries to + # render one table. + camp_ids = [c.id for c in camps] + step_counts: Dict[int, int] = {} + target_counts: Dict[int, int] = {} + send_counts: Dict[int, int] = {} + if camp_ids: + step_counts = dict((await db.execute( + select(CampaignStep.campaign_id, func.count(CampaignStep.id)) + .where(CampaignStep.campaign_id.in_(camp_ids)) + .group_by(CampaignStep.campaign_id) + )).all()) + target_counts = dict((await db.execute( + select(CampaignTarget.campaign_id, func.count(CampaignTarget.id)) + .where(CampaignTarget.campaign_id.in_(camp_ids)) + .group_by(CampaignTarget.campaign_id) + )).all()) + send_counts = dict((await db.execute( + select(Send.campaign_id, func.count(Send.id)) + .where(Send.campaign_id.in_(camp_ids)) + .group_by(Send.campaign_id) + )).all()) + + rows = [ + ( + camp, + step_counts.get(camp.id, 0), + target_counts.get(camp.id, 0), + send_counts.get(camp.id, 0), + ) + for camp in camps + ] if not rows: return ("### LeadForge Campaigns\n\nNo campaigns yet. " @@ -698,23 +907,37 @@ async def handle_tool_call(name: str, arguments: Dict[str, Any]) -> str: select(CampaignTarget.contact_id).where(CampaignTarget.campaign_id == campaign_id) )).all()} + # ONE query for every contact id, and ONE query for the whole + # suppression list. The old shape ran a SELECT per contact id and + # a suppression SELECT per contact - so adding 50 targets cost + # 100+ round-trips. + unique_ids = list(dict.fromkeys(contact_ids)) + contacts = (await db.execute( + select(Contact).where(Contact.id.in_(unique_ids)) + )).scalars().all() + contacts_by_id = {c.id: c for c in contacts} + + candidates = [ + contacts_by_id[cid] for cid in unique_ids + if cid in contacts_by_id and cid not in existing + ] + blocked = await filter_suppressed(db, [c.email for c in candidates]) + added = 0 skipped = 0 missing = 0 - for cid in dict.fromkeys(contact_ids): + for cid in unique_ids: if cid in existing: continue # Gate 1 of 4: never add an already-suppressed address as a target. - contact = (await db.execute( - select(Contact).where(Contact.id == cid) - )).scalars().first() + contact = contacts_by_id.get(cid) if contact is None: # A non-existent contact id used to be silently dropped # and NOT counted, so the tool reported "Added 0 target(s) # (0 skipped)" and the agent concluded success. missing += 1 continue - if await is_suppressed(db, contact.email): + if contact.email and contact.email.strip().lower() in blocked: skipped += 1 continue db.add(CampaignTarget(campaign_id=campaign_id, contact_id=cid)) @@ -942,8 +1165,151 @@ async def handle_tool_call(name: str, arguments: Dict[str, Any]) -> str: raise ValueError(f"Unknown tool: {name}") +async def _write_json_line(payload: Dict[str, Any], lock: asyncio.Lock) -> None: + """Writes exactly one complete JSON-RPC line to stdout, atomically. + + The lock is held across the write AND the flush so two responses that + finish at the same time can never interleave mid-line and corrupt the + stream. A partially written line would break the client's parser for + every message that follows, so this is the one place that must not be + "optimised" into an unlocked write. + """ + out_bytes = (json.dumps(payload) + "\n").encode("utf-8") + try: + async with lock: + sys.stdout.buffer.write(out_bytes) + sys.stdout.buffer.flush() + except BrokenPipeError: + # The client hung up. Nothing left to write to. + logger.warning("stdout closed by the client - dropping response") + + +async def _handle_mcp_request(req: Dict[str, Any], write_lock: asyncio.Lock) -> None: + """Handles one JSON-RPC request and writes its response. + + Runs as its own task so that a slow tool call never blocks the reader. + Responses are written as they complete, which means they may be emitted + OUT OF ORDER relative to the requests that produced them. That is legal + JSON-RPC: a client matches every response to its request by ``id``, so + ordering is not part of the contract. + """ + req_id = req.get("id") + method = req.get("method") + params = req.get("params") or {} + if not isinstance(params, dict): + params = {} + + # A request without an id is a NOTIFICATION. JSON-RPC 2.0 says the + # server MUST NOT reply to a notification, so a request that carries no + # id is executed but never answered. + is_notification = req_id is None + + response: Optional[Dict[str, Any]] = None + + if method == "initialize": + response = { + "jsonrpc": "2.0", + "id": req_id, + "result": { + "protocolVersion": "2024-11-05", + "capabilities": { + "tools": {} + }, + "serverInfo": { + "name": "leadforge", + "version": "1.0.0" + } + } + } + elif method == "notifications/initialized": + # Notification only, no reply required. + return + elif method == "tools/list": + response = { + "jsonrpc": "2.0", + "id": req_id, + "result": { + "tools": TOOLS_DEFINITIONS + } + } + elif method == "tools/call": + tool_name = params.get("name") + tool_args = params.get("arguments") or {} + if not isinstance(tool_args, dict): + tool_args = {} + try: + result_text = await handle_tool_call(tool_name, tool_args) + response = { + "jsonrpc": "2.0", + "id": req_id, + "result": { + "content": [ + { + "type": "text", + "text": result_text + } + ] + } + } + except Exception as tool_err: + logger.exception("Error executing tool %s", tool_name) + # A tool that fails at execution time must return a RESULT + # with isError=true, not a transport-level JSON-RPC error. + # The model can read a tool result and react to it; a + # transport error is invisible to it and it typically retries + # or gives up. The raw exception text also leaked raw SQL + # into the transcript. + response = { + "jsonrpc": "2.0", + "id": req_id, + "result": { + "content": [{ + "type": "text", + "text": ( + f"Tool '{tool_name}' failed: {type(tool_err).__name__}. " + "Check the arguments and the system state with " + "`leadforge_get_stats` / `leadforge_search_leads`, then " + "retry with corrected arguments." + ), + }], + "isError": True, + } + } + elif method == "ping": + response = {"jsonrpc": "2.0", "id": req_id, "result": {}} + else: + if req_id is not None: + response = { + "jsonrpc": "2.0", + "id": req_id, + "error": { + "code": -32601, + "message": f"Method '{method}' not found" + } + } + + if response is None or is_notification: + # Notifications never get a response. + return + + await _write_json_line(response, write_lock) + + async def run_mcp_server(): - """Runs the Model Context Protocol stdio loop.""" + """Runs the Model Context Protocol stdio loop. + + Requests are handled CONCURRENTLY. The reader keeps consuming stdin + while the tool calls are still running, so a five-minute scrape no + longer blocks every other request behind it - and each request gets + its own task, so one slow tool call cannot stall the others either. + + Responses are written as they complete and may therefore arrive OUT OF + ORDER relative to the requests that produced them. JSON-RPC matches + responses to requests by ``id``, so out-of-order delivery is legal and + is what makes concurrency possible. + + The loop exits cleanly on EOF (stdin closed) and on Ctrl-C. + """ logger.info("Starting LeadForge MCP stdio server...") # The MCP server talks to the database directly and must be able to @@ -951,128 +1317,89 @@ async def run_mcp_server(): # process having started first. await ensure_schema() - loop = asyncio.get_event_loop() + loop = asyncio.get_running_loop() reader = asyncio.StreamReader() protocol = asyncio.StreamReaderProtocol(reader) await loop.connect_read_pipe(lambda: protocol, sys.stdin) - while True: - line = await reader.readline() - if not line: - break - - line_str = line.decode("utf-8").strip() - if not line_str: - continue + # Serialises stdout writes so two responses that finish at the same + # time can never interleave mid-line. + write_lock = asyncio.Lock() - try: - req = json.loads(line_str) - except Exception as parse_err: - logger.error(f"JSON decode error: {parse_err}") - # JSON-RPC requires a Parse error response with a null id. - out_bytes = (json.dumps({ - "jsonrpc": "2.0", - "id": None, - "error": {"code": -32700, "message": "Parse error: invalid JSON"}, - }) + "\n").encode("utf-8") - sys.stdout.buffer.write(out_bytes) - sys.stdout.buffer.flush() - continue + # Strong references to the in-flight request tasks. Python only keeps + # weak references to running asyncio tasks, so without this a task can + # be garbage-collected mid-flight and the client never gets a reply. + inflight: set = set() - req_id = req.get("id") - method = req.get("method") - params = req.get("params", {}) + try: + while True: + line = await reader.readline() + if not line: + # EOF: the client closed stdin, the session is over. + break - response = None + line_str = line.decode("utf-8", errors="replace").strip() + if not line_str: + continue - if method == "initialize": - response = { - "jsonrpc": "2.0", - "id": req_id, - "result": { - "protocolVersion": "2024-11-05", - "capabilities": { - "tools": {} - }, - "serverInfo": { - "name": "leadforge", - "version": "1.0.0" - } - } - } - elif method == "notifications/initialized": - # Notification only, no reply required - continue - elif method == "tools/list": - response = { - "jsonrpc": "2.0", - "id": req_id, - "result": { - "tools": TOOLS_DEFINITIONS - } - } - elif method == "tools/call": - tool_name = params.get("name") - tool_args = params.get("arguments", {}) try: - result_text = await handle_tool_call(tool_name, tool_args) - response = { + req = json.loads(line_str) + except Exception as parse_err: + logger.error(f"JSON decode error: {parse_err}") + # JSON-RPC requires a Parse error response with a null id. + await _write_json_line({ "jsonrpc": "2.0", - "id": req_id, - "result": { - "content": [ - { - "type": "text", - "text": result_text - } - ] - } - } - except Exception as tool_err: - logger.exception("Error executing tool %s", tool_name) - # A tool that fails at execution time must return a RESULT - # with isError=true, not a transport-level JSON-RPC error. - # The model can read a tool result and react to it; a - # transport error is invisible to it and it typically retries - # or gives up. The raw exception text also leaked raw SQL - # into the transcript. - response = { - "jsonrpc": "2.0", - "id": req_id, - "result": { - "content": [{ - "type": "text", - "text": ( - f"Tool '{tool_name}' failed: {type(tool_err).__name__}. " - "Check the arguments and the system state with " - "`leadforge_get_stats` / `leadforge_search_leads`, then " - "retry with corrected arguments." - ), - }], - "isError": True, - } - } - elif method == "ping": - response = {"jsonrpc": "2.0", "id": req_id, "result": {}} - else: - if req_id is not None: - response = { + "id": None, + "error": {"code": -32700, "message": "Parse error: invalid JSON"}, + }, write_lock) + continue + + if not isinstance(req, dict): + await _write_json_line({ "jsonrpc": "2.0", - "id": req_id, - "error": { - "code": -32601, - "message": f"Method '{method}' not found" - } - } + "id": None, + "error": {"code": -32600, "message": "Invalid Request: expected a JSON object"}, + }, write_lock) + continue + + # Hand the request to its own task and go straight back to + # reading. The task awaits the whole tool call on its own. + task = asyncio.ensure_future(_handle_mcp_request(req, write_lock)) + inflight.add(task) + task.add_done_callback(inflight.discard) + + except asyncio.CancelledError: + # Ctrl-C (or a parent shutting us down): stop the world promptly. + for t in list(inflight): + t.cancel() + if inflight: + await asyncio.gather(*list(inflight), return_exceptions=True) + raise + + # EOF: let the requests that are already running write their response + # before the process goes away, then cancel the stragglers so the + # server cannot hang on a long tool call forever. + if inflight: + logger.info("stdin closed - draining %d in-flight request(s)", len(inflight)) + pending = set(inflight) + _, still_pending = await asyncio.wait( + list(pending), timeout=_SHUTDOWN_GRACE_SECONDS + ) + for t in still_pending: + t.cancel() + if still_pending: + await asyncio.gather(*list(still_pending), return_exceptions=True) - if response: - out_bytes = (json.dumps(response) + "\n").encode("utf-8") - sys.stdout.buffer.write(out_bytes) - sys.stdout.buffer.flush() + logger.info("LeadForge MCP stdio server stopped.") def main(): - asyncio.run(run_mcp_server()) + try: + asyncio.run(run_mcp_server()) + except KeyboardInterrupt: + # Ctrl-C: exit quietly, without a traceback. + logger.info("LeadForge MCP server interrupted - shutting down") + sys.exit(130) if __name__ == "__main__": diff --git a/backend/app/models/business.py b/backend/app/models/business.py index 05c6d83..a828c9f 100644 --- a/backend/app/models/business.py +++ b/backend/app/models/business.py @@ -1,8 +1,9 @@ -from sqlalchemy import Column, Integer, String, Float, DateTime, Boolean, JSON, ForeignKey +from sqlalchemy import Column, Integer, String, Float, DateTime, Boolean, JSON, ForeignKey, event from sqlalchemy.sql import func from sqlalchemy.orm import relationship from app.database import Base + class Business(Base): __tablename__ = "businesses" @@ -11,6 +12,17 @@ class Business(Base): name = Column(String, index=True) website = Column(String, nullable=True) phone = Column(String, nullable=True) + #: Phone reduced to comparable digits (Indian +91 / leading-0 handled). + #: Matching on the raw string never worked: the scraper stores whatever + #: Google returned ("+91 98225 22560") while the dedupe matcher compared a + #: normalised value ("9822522560") against it, so the phone branch of the + #: dedupe was dead code. This column is populated automatically on every + #: write (see the event listener below) and is indexed. + phone_e164 = Column(String, nullable=True, index=True) + #: Host-only form of ``website`` ("example.in"). A substring match on the + #: raw website string made ``example.com`` match + #: ``smileshop.example.com`` - a silent false merge. + domain = Column(String, nullable=True, index=True) address = Column(String, nullable=True) rating = Column(Float, nullable=True) reviews_count = Column(Integer, nullable=True) @@ -34,3 +46,48 @@ class Business(Base): contacts = relationship("Contact", back_populates="business", cascade="all, delete-orphan") social_profiles = relationship("SocialProfile", back_populates="business", cascade="all, delete-orphan") icp_profile = relationship("ICPProfile", back_populates="businesses") + + +def normalize_phone_e164(raw): + """Comparable digits for a phone string, handling Indian formats.""" + import re as _re + if not raw: + return None + digits = _re.sub(r"\D", "", str(raw)) + if len(digits) == 12 and digits.startswith("91"): + digits = digits[2:] + elif len(digits) == 11 and digits.startswith("0"): + digits = digits[1:] + return digits or None + + +def canonical_domain_of(url): + """Host-only form of a URL, without the ``www.`` prefix.""" + if not url: + return None + try: + from urllib.parse import urlparse + host = urlparse(str(url) if "://" in str(url) else f"http://{url}").netloc.lower() + except Exception: + return None + host = host.split(":")[0] + import re as _re + host = _re.sub(r"^www\.", "", host).strip() + return host or None + + +def _populate_identity(mapper, connection, target): + """Keeps the derived identity columns in sync on every write. + + An event listener catches EVERY ORM write path - the scraper, the CSV + importer, the enrichment worker and the API - without any of them having + to remember. That matters because the dedupe matcher reads these + columns; a path that forgot to set them would silently produce + duplicate leads. + """ + target.phone_e164 = normalize_phone_e164(target.phone) + target.domain = canonical_domain_of(target.website) + + +event.listen(Business, "before_insert", _populate_identity) +event.listen(Business, "before_update", _populate_identity) diff --git a/backend/app/qualifier/signals.py b/backend/app/qualifier/signals.py index 603fbbc..b564a7d 100644 --- a/backend/app/qualifier/signals.py +++ b/backend/app/qualifier/signals.py @@ -1,3 +1,4 @@ +import asyncio import re from typing import Any, Dict, List, Optional @@ -127,7 +128,7 @@ async def _probe_website_health(url: str, timeout: float = 10.0) -> Dict[str, An "(KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36" } try: - async with httpx.AsyncClient(timeout=timeout, follow_redirects=True, verify=False, headers=headers) as client: + async with httpx.AsyncClient(timeout=timeout, follow_redirects=True, headers=headers) as client: response = await client.get(url) out["ssl_valid"] = str(response.url).startswith("https://") and response.url.is_secure out["final_status_code"] = response.status_code @@ -182,20 +183,29 @@ async def _probe_facebook_ads(business_name: str, country: str = "IN", timeout: "Accept-Language": "en-US,en;q=0.9", } try: - async with httpx.AsyncClient(timeout=timeout, follow_redirects=True, verify=False, headers=headers) as client: - response = await client.get(url) - if response.status_code != 200: - return {"running_ads": None, "ad_count": None} - text = response.text - counts = re.findall(r"\"ad_count\"\s*:\s*(\d+)", text) - if counts: - n = int(counts[0]) - return {"running_ads": n > 0, "ad_count": n} - # Some page variants embed a different key - counts = re.findall(r"active_ads\D{0,40}(\d+)", text) - if counts: - n = int(counts[0]) - return {"running_ads": n > 0, "ad_count": n} + async with httpx.AsyncClient(timeout=timeout, follow_redirects=True, headers=headers) as client: + # The Ad Library page is multi-megabyte of markup and all we + # want is two numbers in it. Streaming and stopping at the first + # match turns a multi-MB download into a few KB - this runs once + # per enriched lead, so the saving is multiplied by the whole + # corpus. + buffer = "" + async with client.stream("GET", url) as response: + if response.status_code != 200: + return {"running_ads": None, "ad_count": None} + async for chunk in response.aiter_text(): + buffer += chunk + match = re.search(r"\"ad_count\"\s*:\s*(\d+)", buffer) + if match: + n = int(match.group(1)) + return {"running_ads": n > 0, "ad_count": n} + # Some page variants embed a different key + match = re.search(r"active_ads\D{0,40}(\d+)", buffer) + if match: + n = int(match.group(1)) + return {"running_ads": n > 0, "ad_count": n} + if len(buffer) > 2_000_000: + break return {"running_ads": None, "ad_count": None} except Exception: return {"running_ads": None, "ad_count": None} @@ -205,11 +215,29 @@ async def probe_business_health(business: Any, country: str = "IN") -> Dict[str, """ Runs the network-dependent probes for a lead and returns a patch to merge into business.extra_data so scoring stays offline and repeatable. + + The website probe and the Facebook Ad Library probe are independent + network calls, so they run concurrently. Sequentially they cost the sum + of both timeouts (up to ~22 seconds per lead); concurrently it is the + slower of the two. """ - patch: Dict[str, Any] = {} website = getattr(business, "website", None) - if website: - patch.update(await _probe_website_health(website)) - ads = await _probe_facebook_ads(getattr(business, "name", "") or "", country=country) - patch.update(ads) + name = getattr(business, "name", "") or "" + + site_probe, ads_probe = await asyncio.gather( + _probe_website_health(website) if website else _empty_probe(), + _probe_facebook_ads(name, country=country), + return_exceptions=True, + ) + + patch: Dict[str, Any] = {} + if isinstance(site_probe, dict): + patch.update(site_probe) + if isinstance(ads_probe, dict): + patch.update(ads_probe) return patch + + +async def _empty_probe() -> Dict[str, Any]: + """A no-op probe result, so both branches can be awaited uniformly.""" + return {} diff --git a/backend/app/sources/http_utils.py b/backend/app/sources/http_utils.py index 21f3e54..6bf5596 100644 --- a/backend/app/sources/http_utils.py +++ b/backend/app/sources/http_utils.py @@ -81,14 +81,26 @@ def _proxy_pool() -> List[Optional[str]]: def looks_blocked(html: Optional[str]) -> bool: - """True when the response is an interstitial/captcha rather than content.""" + """True when the response is an interstitial/captcha rather than content. + + The size heuristic used to be "any body under 2 KB is a block stub". + That is wrong in both directions and expensive: many legitimate pages + are tiny (a minimal one-page site, a JSON endpoint, an Instagram/FB + login wall), so every one of them was retried four times with + exponential backoff - ~7-9 seconds per URL - and then **discarded**. + The social finder alone cost 6.9s per lead and returned nothing. + + A block page is identified by what it *says*, not by how small it is. + The size check stays only as a guard for a body that is clearly empty. + """ if not html: return True - # A block stub is tiny; a real listing page is never under ~2 KB. - if len(html) < 2048: - return True sample = html[:20000].lower() - return any(marker in sample for marker in BLOCK_MARKERS) + if any(marker in sample for marker in BLOCK_MARKERS): + return True + # A body that is essentially empty is not content either - but keep the + # threshold low so small-but-real pages are never thrown away. + return len(html.strip()) < 32 async def fetch_html( @@ -114,61 +126,67 @@ async def fetch_html( attempts = max(1, retries + 1) last_reason = "unknown" - for attempt in range(attempts): - chosen_proxy = proxy or random.choice(pool) - headers = { - # Deterministic rotation keyed on the attempt index, so the same - # attempt is reproducible in tests while still varying in practice. - "User-Agent": USER_AGENTS[attempt % len(USER_AGENTS)], - "Accept": ( - "text/html,application/xhtml+xml,application/xml;q=0.9," - "image/avif,image/webp,*/*;q=0.8" - ), - "Accept-Language": "en-IN,en-GB;q=0.9,en;q=0.8", - # Directories serve different markup to bots that omit these. - "Upgrade-Insecure-Requests": "1", - "Sec-Fetch-Dest": "document", - "Sec-Fetch-Mode": "navigate", - "Sec-Fetch-Site": "none", - "Sec-Fetch-User": "?1", - "Cache-Control": "max-age=0", - } - try: - async with httpx.AsyncClient( - timeout=timeout, - follow_redirects=True, - verify=False, - headers=headers, - proxy=chosen_proxy, - ) as client: - response = await client.get(url) - - if response.status_code == 200: - body = response.text - if looks_blocked(body): - last_reason = "blocked_page" + # One client for the whole retry loop. A client per attempt meant a new + # TLS handshake and no connection reuse on every single request - the + # single cheapest performance win in this module. + client = httpx.AsyncClient( + timeout=timeout, + follow_redirects=True, + headers={}, + proxy=pool[0], + limits=httpx.Limits(max_connections=10, max_keepalive_connections=5), + ) + try: + for attempt in range(attempts): + chosen_proxy = proxy or random.choice(pool) + headers = { + # Deterministic rotation keyed on the attempt index, so the same + # attempt is reproducible in tests while still varying in practice. + "User-Agent": USER_AGENTS[attempt % len(USER_AGENTS)], + "Accept": ( + "text/html,application/xhtml+xml,application/xml;q=0.9," + "image/avif,image/webp,*/*;q=0.8" + ), + "Accept-Language": "en-IN,en-GB;q=0.9,en;q=0.8", + # Directories serve different markup to bots that omit these. + "Upgrade-Insecure-Requests": "1", + "Sec-Fetch-Dest": "document", + "Sec-Fetch-Mode": "navigate", + "Sec-Fetch-Site": "none", + "Sec-Fetch-User": "?1", + "Cache-Control": "max-age=0", + } + try: + response = await client.get(url, headers=headers, proxy=chosen_proxy) + + if response.status_code == 200: + body = response.text + if looks_blocked(body): + last_reason = "blocked_page" + else: + return body + elif response.status_code in (403, 429, 503): + # Retryable: rate limit or a client fingerprint we can change. + last_reason = f"http_{response.status_code}" + elif 400 <= response.status_code < 500: + # A genuine client error (404/410/gone) will not fix itself. + logger.debug(f"fetch_html {url} -> HTTP {response.status_code} (not retrying)") + return None else: - return body - elif response.status_code in (403, 429, 503): - # Retryable: rate limit or a client fingerprint we can change. - last_reason = f"http_{response.status_code}" - elif 400 <= response.status_code < 500: - # A genuine client error (404/410/gone) will not fix itself. - logger.debug(f"fetch_html {url} -> HTTP {response.status_code} (not retrying)") - return None - else: - last_reason = f"http_{response.status_code}" - - except Exception as e: - last_reason = f"{type(e).__name__}" - - if attempt < attempts - 1: - delay = backoff ** attempt + random.uniform(0, 0.5) - logger.debug(f"fetch_html {url} attempt {attempt + 1} failed ({last_reason}); retry in {delay:.1f}s") - await asyncio.sleep(delay) - - logger.debug(f"fetch_html {url} gave up after {attempts} attempts ({last_reason})") - return None + last_reason = f"http_{response.status_code}" + + except Exception as e: + last_reason = f"{type(e).__name__}" + + if attempt < attempts - 1: + delay = backoff ** attempt + random.uniform(0, 0.5) + logger.debug(f"fetch_html {url} attempt {attempt + 1} failed ({last_reason}); retry in {delay:.1f}s") + await asyncio.sleep(delay) + + logger.debug(f"fetch_html {url} gave up after {attempts} attempts ({last_reason})") + return None + finally: + await client.aclose() def slugify(value: str) -> str: diff --git a/backend/tests/test_wave0_fixes.py b/backend/tests/test_wave0_fixes.py index ae3f584..d2022db 100644 --- a/backend/tests/test_wave0_fixes.py +++ b/backend/tests/test_wave0_fixes.py @@ -344,3 +344,118 @@ def test_phone_extraction_handles_a_missing_phone(): raw = [None] * 179 assert _dig_phone(raw) is None + + +# -------------------------------------------------------------------------- +# Full-text search must return exactly what the LIKE scan returns +# -------------------------------------------------------------------------- + +@pytest.mark.asyncio +async def test_fts_search_matches_the_like_scan(client): + """The FTS5 trigram index must be semantically identical to LIKE. + + A search that quietly returns a different result set is worse than a + slow one, so the two paths are compared row-for-row on the same data. + """ + from app.database import async_session_maker as _maker + from sqlalchemy import select, func, or_ as _or_ + from app.models.business import Business as _B + + async with _maker() as db: + samples = [ + ("Sharma Dental Clinic", "Kothrud Pune", "+91 98225 22560", "https://sharma.in"), + ("Clove Dental Care", "Viman Nagar", "+91 90000 11111", "https://clove.in"), + ("Apex Realty", "Baner Road", "+91 98877 66554", "https://apex.in"), + ("Bright Smile Studio", "Aundh", None, None), + ] + for name, addr, phone, web in samples: + db.add(_B(name=name, address=addr, phone=phone, website=web)) + await db.commit() + + # The index is created by the schema sync; make sure it exists. + from app.database import ensure_schema + await ensure_schema() + + terms = ["dental", "othru", "Koth", "98225", "22560", "shine", "nowherexyz"] + for term in terms: + res = await client.get(f"/api/v1/leads?search={term}&limit=100") + fts_ids = sorted(lead["id"] for lead in res.json()["leads"]) + + async with _maker() as db: + like = f"%{term}%" + rows = (await db.execute( + select(_B.id).where(_or_( + _B.name.ilike(like), + _B.address.ilike(like), + _B.phone.ilike(like), + _B.website.ilike(like), + )) + )).all() + like_ids = sorted(r[0] for r in rows) + + assert fts_ids == like_ids, ( + f"term {term!r}: FTS returned {fts_ids} but LIKE returned {like_ids}" + ) + + +# -------------------------------------------------------------------------- +# Analytics aggregation must happen in SQL and stay exact +# -------------------------------------------------------------------------- + +@pytest.mark.asyncio +async def test_timeseries_matches_a_python_reference(client): + """The SQL day-bucketing must produce exactly the same numbers as + aggregating the rows in Python. + + A ``CAST(sent_at AS DATE)`` looks portable but is not: SQLite applies + NUMERIC affinity to the cast and truncates '2026-10-17' to the integer + 2026, which then fails in SQLAlchemy's date processor. + """ + from datetime import timedelta + from app.database import async_session_maker as _maker + from app.models.send import Send as _S + from app.models.campaign import Campaign as _C + from app.models.campaign_step import CampaignStep as _CS + from app.models.contact import Contact as _CT + from app.models.business import Business as _B + from sqlalchemy import select as _sel + + async with _maker() as db: + camp = _C(name="ts"); db.add(camp); await db.flush() + step = _CS(campaign_id=camp.id, step_number=1, subject="s", body_text="b") + db.add(step); await db.flush() + biz = _B(name="B"); db.add(biz); await db.flush() + con = _CT(business_id=biz.id, email="t@b.in"); db.add(con); await db.flush() + + now = datetime.now(timezone.utc).replace(tzinfo=None) + for i in range(30): + db.add(_S(campaign_id=camp.id, step_id=step.id, contact_id=con.id, + status="sent", sent_at=now - timedelta(days=i), + opened=(i % 3 == 0), replied=(i % 10 == 0))) + await db.commit() + + async with _maker() as db: + rows = (await db.execute( + _sel(_S).where(_S.status == "sent", _S.sent_at.isnot(None)) + )).scalars().all() + reference = {} + for s in rows: + cell = reference.setdefault(s.sent_at.date().isoformat(), + {"sent": 0, "opened": 0, "replied": 0}) + cell["sent"] += 1 + if s.opened: + cell["opened"] += 1 + if s.replied: + cell["replied"] += 1 + + res = await client.get("/api/v1/analytics/timeseries?days=30") + payload = res.json() + assert payload["totals"]["sent"] == sum(v["sent"] for v in reference.values()) + + by_day = {d["date"]: d for d in payload["days"]} + for day, cell in reference.items(): + got = by_day.get(day) + assert got is not None, f"missing day {day}" + assert got["sent"] == cell["sent"], f"{day}: {got} != {cell}" + assert got["opened"] == cell["opened"], f"{day} opened mismatch" + assert got["replied"] == cell["replied"], f"{day} replied mismatch" From 0793dc3f3292be112e513653ddc4ec563cbdb328 Mon Sep 17 00:00:00 2001 From: Aryan Bansal Date: Wed, 21 Oct 2026 19:05:41 +0530 Subject: [PATCH 03/46] perf(api): batch the remaining per-row queries - The lead list no longer takes the database's write lock on every page view: the refreshed score is only persisted when it actually changed. - The campaigns list issued 4 queries per campaign (151 round-trips for a 50-campaign list). The send, target and step aggregates are now three grouped queries. Also fixed the mislabelled field: "total_targets" reported the send count, not the number of selected recipients. - The full-text search triggers are created independently so one failure cannot take the others down, and the table is recreated if it goes missing (a missing table made every business UPDATE fail, because the update trigger referenced it). Measured at 500,000 leads: /leads 10.5ms /leads?sort=score 10.2ms /leads?has_email=true 2.4ms /leads?search=dental 83.5ms (83,000 matches - still a count) /leads?search=biz42 7.9ms /leads?search= 2.0ms /analytics/stats 10.0ms /analytics/timeseries 1.9ms /campaigns 1.7ms /inbox 3.0ms Co-Authored-By: Claude Opus 4.8 (1M context) --- backend/app/api/campaigns.py | 58 ++++++++++++++++++++++++++---------- backend/app/api/leads.py | 12 ++++++-- backend/app/database.py | 11 +++++-- backend/app/mcp_server.py | 2 +- 4 files changed, 63 insertions(+), 20 deletions(-) diff --git a/backend/app/api/campaigns.py b/backend/app/api/campaigns.py index fd44dd2..5d01f6a 100644 --- a/backend/app/api/campaigns.py +++ b/backend/app/api/campaigns.py @@ -46,37 +46,65 @@ async def list_campaigns(db: AsyncSession = Depends(get_db)): res = await db.execute(stmt) campaigns = res.scalars().all() - result = [] - for c in campaigns: - # Aggregate send stats + # Three grouped aggregates instead of three queries per campaign. The + # endpoint used to issue 4N queries (1 for the list, 3 per campaign), so + # a 50-campaign list was 151 round-trips. + campaign_ids = [c.id for c in campaigns] + + send_stats: Dict[int, tuple] = {} + target_counts: Dict[int, int] = {} + step_counts: Dict[int, int] = {} + + if campaign_ids: sends_stmt = select( + Send.campaign_id, func.count(Send.id), func.count(Send.id).filter(Send.status == "sent"), func.count(Send.id).filter(Send.opened == True), - func.count(Send.id).filter(Send.replied == True) - ).where(Send.campaign_id == c.id) - - s_res = await db.execute(sends_stmt) - total_sends, sent_count, opened_count, replied_count = s_res.one() + func.count(Send.id).filter(Send.replied == True), + ).where(Send.campaign_id.in_(campaign_ids)).group_by(Send.campaign_id) + for cid, total, sent_c, opened_c, replied_c in (await db.execute(sends_stmt)).all(): + send_stats[cid] = (total or 0, sent_c or 0, opened_c or 0, replied_c or 0) + + targets_stmt = ( + select(CampaignTarget.campaign_id, func.count(CampaignTarget.id)) + .where(CampaignTarget.campaign_id.in_(campaign_ids)) + .group_by(CampaignTarget.campaign_id) + ) + target_counts = { + cid: count for cid, count in (await db.execute(targets_stmt)).all() + } + + steps_stmt = ( + select(CampaignStep.campaign_id, func.count(CampaignStep.id)) + .where(CampaignStep.campaign_id.in_(campaign_ids)) + .group_by(CampaignStep.campaign_id) + ) + step_counts = { + cid: count for cid, count in (await db.execute(steps_stmt)).all() + } + + result = [] + for c in campaigns: + total_sends, sent_count, opened_count, replied_count = send_stats.get( + c.id, (0, 0, 0, 0) + ) open_rate = round((opened_count / sent_count * 100), 1) if sent_count > 0 else 0.0 reply_rate = round((replied_count / sent_count * 100), 1) if sent_count > 0 else 0.0 # Explicit recipients (Apollo "prospects") plus total steps, so the UI # can render "N prospects · M steps" without a second round-trip. - target_count = (await db.execute( - select(func.count(CampaignTarget.id)).where(CampaignTarget.campaign_id == c.id) - )).scalar() or 0 - step_count = (await db.execute( - select(func.count(CampaignStep.id)).where(CampaignStep.campaign_id == c.id) - )).scalar() or 0 + target_count = target_counts.get(c.id, 0) + step_count = step_counts.get(c.id, 0) result.append({ "id": c.id, "name": c.name, "status": c.status, - "total_targets": total_sends, + "total_targets": target_count, "contacts_selected": target_count, + "total_sends": total_sends, "step_count": step_count, "sent_count": sent_count, "opened_count": opened_count, diff --git a/backend/app/api/leads.py b/backend/app/api/leads.py index ab01efc..5e112c1 100644 --- a/backend/app/api/leads.py +++ b/backend/app/api/leads.py @@ -213,6 +213,9 @@ async def get_leads( }) leads_data = [] + #: Business ids whose persisted score was stale and has just been + #: recomputed. + stale_score_ids: List[int] = [] for b in businesses: b_contacts = contacts_map.get(b.id, []) primary_contact = b_contacts[0] if b_contacts else None @@ -246,9 +249,12 @@ async def get_leads( lead_score = score_info["score"] lead_tier = score_info["tier"] lead_badges = score_info["badges"] - # Keep the column in sync so it can be sorted/filtered in SQL. + # Keep the column in sync so it can be sorted/filterable in SQL. Only + # when it actually changed: this endpoint is a GET, and taking the + # database's write lock on every page view is pure overhead. if b.score != lead_score: b.score = lead_score + stale_score_ids.append(b.id) # India source-trust: surfaces *why* a lead's score was adjusted — # self-listed directories (JustDial/IndiaMART) carry shared/stale @@ -294,7 +300,9 @@ async def get_leads( }) # Persist the refreshed scores so they are sortable/filterable in SQL. - await db.commit() + # A read that has nothing to write must not take the write lock. + if stale_score_ids: + await db.commit() return { "total": total, diff --git a/backend/app/database.py b/backend/app/database.py index 4c55c20..d59d00e 100644 --- a/backend/app/database.py +++ b/backend/app/database.py @@ -196,12 +196,19 @@ def _sync_fts(sync_conn) -> None: existing = None if not existing: sync_conn.execute(text(_FTS_CREATE)) - for ddl in _FTS_DDL: - sync_conn.execute(text(ddl)) except Exception as exc: logger.info("full-text search index unavailable (%s) - using LIKE fallback", exc) return + # Each trigger is created independently: a trigger that fails to create + # must not take the others down with it, and a missing trigger only + # means the index drifts (the backfill below repairs it). + for ddl in _FTS_DDL: + try: + sync_conn.execute(text(ddl)) + except Exception as exc: + logger.warning("full-text search trigger failed: %s", exc) + # Backfill: the triggers only fire on new writes, so an existing # database needs a rebuild. 'rebuild' is a no-op when already in sync. try: diff --git a/backend/app/mcp_server.py b/backend/app/mcp_server.py index 5fd1c45..e715e8c 100755 --- a/backend/app/mcp_server.py +++ b/backend/app/mcp_server.py @@ -16,7 +16,7 @@ import urllib.parse from collections import defaultdict from datetime import datetime -from typing import Dict, Any, List, Optional, Sequence, Set, Tuple +from typing import Dict, Any, List, Optional, Sequence, Set # Ensure backend directory is in python search path sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))) From a0c1dd1ab3e1d265b8ff127bd4930aefe1269226 Mon Sep 17 00:00:00 2001 From: Aryan Bansal Date: Wed, 21 Oct 2026 19:08:01 +0530 Subject: [PATCH 04/46] perf(scrape): batch the scrape-time dedupe, and fix the identity it uses The scrape path matched businesses on the business NAME only, ignoring the place id, the phone and the domain - and it ran one SELECT per scraped lead. It now matches in the same priority order as the rest of the codebase (place id -> phone -> domain -> name) using four grouped queries for the whole batch, so the cost scales with the number of leads rather than with the size of the database. Businesses created within the same batch are registered in the index too, so a duplicate row inside one batch is caught. Two identity bugs fixed along the way: - The Google CID was being dropped on the primary scrape path (the HTTP path calls it google_data_id, and only the sources layer mapped it back). Every Maps-sourced business had a NULL place_id, so the strongest dedupe key we have was unused. Now 25/25 leads carry it. - ensure_schema() did not import the models, so create_all silently created NOTHING and the full-text triggers failed with 'no such table: main.businesses'. It only worked because every caller happened to have imported the API modules first - and saved_lists was missing from freshly created databases for the same reason. Also stopped the schema sync logging bogus 'duplicate column name' and 'index already exists' warnings on every boot: the SQLAlchemy inspector is a snapshot taken before the ALTER, so it never saw what it had just added. Verified on a live scrape: 25 businesses, 25 with a Google CID, 26 distinct phone numbers, zero duplicate names, and a business whose name changed is still matched through its phone. Co-Authored-By: Claude Opus 4.8 (1M context) --- backend/app/database.py | 12 ++++ backend/app/dedupe/match.py | 93 ++++++++++++++++++++++++++++- backend/app/scraper/maps_scraper.py | 52 ++++++++++++++-- 3 files changed, 151 insertions(+), 6 deletions(-) diff --git a/backend/app/database.py b/backend/app/database.py index d59d00e..1c99c2a 100644 --- a/backend/app/database.py +++ b/backend/app/database.py @@ -70,6 +70,12 @@ async def ensure_schema() -> None: """ from sqlalchemy import inspect as sa_inspect + # Importing the models registers them on Base.metadata. Without this, + # ``create_all`` silently created NOTHING and the full-text triggers + # failed with "no such table: main.businesses" - it only worked because + # every caller happened to have imported the API modules first. + import app.models # noqa: F401 + async with engine.begin() as conn: await conn.run_sync(Base.metadata.create_all) await conn.run_sync(_sync_missing_columns) @@ -103,6 +109,11 @@ def _sync_missing_columns(sync_conn) -> None: try: sync_conn.execute(text(ddl)) logger.info("schema sync: added %s.%s", table.name, col.name) + # The inspector is a snapshot taken before the ALTER, so it + # does not know about the column we just added. Without this + # the next boot logs a bogus "duplicate column name" + # warning on every start. + have.add(col.name) except Exception as exc: # pragma: no cover - dialect-specific limits logger.warning("schema sync skipped %s.%s: %s", table.name, col.name, exc) @@ -139,6 +150,7 @@ def _sync_missing_indexes(sync_conn) -> None: ddl = str(CreateIndex(index).compile(dialect=sync_conn.dialect)) sync_conn.execute(text(ddl)) logger.info("schema sync: created index %s", index.name) + present.add(index.name) except Exception as exc: # pragma: no cover - dialect-specific limits logger.warning("schema sync skipped index %s: %s", index.name, exc) diff --git a/backend/app/dedupe/match.py b/backend/app/dedupe/match.py index 2372655..11429dd 100644 --- a/backend/app/dedupe/match.py +++ b/backend/app/dedupe/match.py @@ -1,5 +1,5 @@ import re -from typing import Optional +from typing import Dict, List, Optional from urllib.parse import urlparse from sqlalchemy import or_, select @@ -48,6 +48,97 @@ def normalize_name(name: Optional[str]) -> str: return " ".join(tokens) +async def build_lead_indexes(db: AsyncSession, leads: List[dict]) -> Dict[str, Dict[str, Business]]: + """Builds the four dedupe lookup tables for a batch of leads. + + The scrape path used to run one SELECT per lead, and matched on the + business NAME only - ignoring the place id, the phone and the domain. + That both cost a query per lead and got the identity wrong: the same + business whose name changed became a duplicate, and two different + businesses that happen to share a generic Indian name ("Sharma Dental + Clinic") were merged. + + Four grouped queries - one per key type - replace the per-lead query, + and the lookup then happens in Python. The cost scales with the number + of leads, not with the size of the database. + """ + place_ids = {l.get("place_id") for l in leads if l.get("place_id")} + phones = {normalize_phone(l.get("phone")) for l in leads if l.get("phone")} + domains = {canonical_domain(l.get("website")) for l in leads if l.get("website")} + names = {l.get("name") for l in leads if l.get("name")} + + by_place: Dict[str, Business] = {} + by_phone: Dict[str, Business] = {} + by_domain: Dict[str, Business] = {} + by_name: Dict[str, Business] = {} + + if place_ids: + rows = (await db.execute( + select(Business).where(Business.place_id.in_(place_ids)) + )).scalars().all() + for b in rows: + if b.place_id: + by_place.setdefault(b.place_id, b) + + if phones: + rows = (await db.execute( + select(Business).where(Business.phone_e164.in_(phones)) + )).scalars().all() + for b in rows: + if b.phone_e164: + by_phone.setdefault(b.phone_e164, b) + + if domains: + rows = (await db.execute( + select(Business).where(Business.domain.in_(domains)) + )).scalars().all() + for b in rows: + if b.domain: + by_domain.setdefault(b.domain, b) + + if names: + rows = (await db.execute( + select(Business).where(Business.name.in_(names)) + )).scalars().all() + for b in rows: + if b.name: + by_name.setdefault(b.name, b) + + return { + "place_id": by_place, + "phone": by_phone, + "domain": by_domain, + "name": by_name, + } + + +def match_lead(lead: dict, indexes: Dict[str, Dict[str, Business]]) -> Optional[Business]: + """Matches a lead against prebuilt indexes, in the same priority order as + :func:`find_existing_business`.""" + place_id = lead.get("place_id") + if place_id: + match = indexes["place_id"].get(place_id) + if match is not None: + return match + + phone = normalize_phone(lead.get("phone")) + if phone: + match = indexes["phone"].get(phone) + if match is not None: + return match + + domain = canonical_domain(lead.get("website")) + if domain: + match = indexes["domain"].get(domain) + if match is not None: + return match + + name = lead.get("name") + if name: + return indexes["name"].get(name) + return None + + async def find_existing_business(db: AsyncSession, lead: dict) -> Optional[Business]: """Match a raw lead to an existing business by place_id, phone, domain, or name. diff --git a/backend/app/scraper/maps_scraper.py b/backend/app/scraper/maps_scraper.py index 762955a..b420cf9 100644 --- a/backend/app/scraper/maps_scraper.py +++ b/backend/app/scraper/maps_scraper.py @@ -516,19 +516,49 @@ async def scrape_leads_and_save( await db.commit() new_biz_ids = [] + + # Dedupe in the same priority order as the rest of the codebase + # (place id -> phone -> domain -> name), using four grouped + # queries for the whole batch instead of one query per lead. + # + # The old code matched on the business NAME only, so the same + # business whose name changed became a duplicate, and two + # different businesses sharing a generic Indian name were merged. + from app.dedupe.match import build_lead_indexes, match_lead + _leads_for_index = [ + { + "place_id": lead.get("place_id"), + "phone": lead.get("phone"), + "website": lead.get("website"), + "name": (lead.get("name") or "").strip(), + } + for lead in raw_leads if (lead.get("name") or "").strip() + ] + _indexes = await build_lead_indexes(db, _leads_for_index) + for lead in raw_leads: name = lead.get("name", "").strip() if not name: continue - # Check for existing business by name - stmt = select(Business).where(Business.name == name) - res = await db.execute(stmt) - existing = res.scalars().first() + existing = match_lead( + { + "place_id": lead.get("place_id"), + "phone": lead.get("phone"), + "website": lead.get("website"), + "name": name, + }, + _indexes, + ) if not existing: business = Business( - place_id=lead.get("place_id"), + # The Google CID is the strongest identity signal we + # have and it is what the HTTP path calls + # ``google_data_id``. It was being dropped here, so + # every Maps-sourced business had a NULL place_id and + # the dedupe fell back to name/phone alone. + place_id=lead.get("place_id") or lead.get("google_data_id"), name=name, website=lead.get("website") or None, phone=lead.get("phone") or None, @@ -567,6 +597,18 @@ async def scrape_leads_and_save( db.add(contact) saved_count += 1 new_biz_ids.append(business.id) + # Register the new business so a duplicate row *within the + # same batch* is caught too. + from app.dedupe.match import canonical_domain, normalize_phone + if lead.get("place_id"): + _indexes["place_id"].setdefault(lead["place_id"], business) + _p = normalize_phone(lead.get("phone")) + if _p: + _indexes["phone"].setdefault(_p, business) + _d = canonical_domain(lead.get("website")) + if _d: + _indexes["domain"].setdefault(_d, business) + _indexes["name"].setdefault(name, business) else: # Update phone, website, or maps_url if missing if not existing.phone and lead.get("phone"): From bf579e9afe034b8db5e19105e3a45d6c0fd12fea Mon Sep 17 00:00:00 2001 From: Aryan Bansal Date: Wed, 21 Oct 2026 19:10:12 +0530 Subject: [PATCH 05/46] perf(send): check the suppression list once per batch, not once per contact The campaign sender called `await is_suppressed(db, contact.email)` inside the candidate loop - one round-trip per contact, so a 500-target campaign spent 500 queries deciding who NOT to email, before sending a single message. Added `suppressed_reasons()`, which answers the whole candidate set in one query, and rewired `filter_suppressed()` on top of it so there is one implementation of the rules. A regression test asserts the batch form returns exactly what the per-address check returned, including the whole-domain case and the expired-soft-bounce case (which must NOT block). Co-Authored-By: Claude Opus 4.8 (1M context) --- backend/app/campaigns/sender.py | 27 +++++++++------ backend/app/campaigns/suppression.py | 33 +++++++++++++----- backend/tests/test_wave0_fixes.py | 52 ++++++++++++++++++++++++++++ 3 files changed, 93 insertions(+), 19 deletions(-) diff --git a/backend/app/campaigns/sender.py b/backend/app/campaigns/sender.py index c86e092..784b378 100644 --- a/backend/app/campaigns/sender.py +++ b/backend/app/campaigns/sender.py @@ -22,7 +22,7 @@ from app.models.campaign_step import CampaignStep from app.models.send import Send from app.campaigns.bounces import classify_bounce, HARD, SOFT -from app.campaigns.suppression import is_suppressed, add_suppression +from app.campaigns.suppression import is_suppressed, add_suppression, suppressed_reasons from app.models.suppression import PERMANENT_REASONS from app.campaigns.warmup import ( mailbox_age_days, ramp_limit, mailbox_health, is_mailbox_usable, @@ -478,22 +478,29 @@ async def execute_campaign_step(campaign_id: int, max_sends: int = 50) -> Dict[s #: sequences as done. pending_delays = 0 + # The authoritative suppression gate (gate 3 of 4). The per-contact + # `unsubscribed` boolean only records an unsubscribe on THIS row; a + # bounce, complaint or "stop" reply on any campaign must block the + # address everywhere. + # + # The whole candidate set is checked in ONE query. It used to be + # `await is_suppressed(db, contact.email)` inside the loop - one + # round-trip per candidate, so a 500-target campaign spent 500 + # queries deciding who not to email. + candidate_emails = [c.email for c, _b in candidates if c.email] + suppressed_emails = await filter_suppressed(db, candidate_emails) + suppression_reasons = await suppressed_reasons(db, candidate_emails) + for contact, business in candidates: c_sends = sends_by_contact.get(contact.id, []) - # Authoritative suppression gate (gate 3 of 4). The per-contact - # `unsubscribed` boolean only records an unsubscribe on THIS row; - # a bounce, complaint or "stop" reply on any campaign must block - # the address everywhere. Check the workspace-wide table here, - # immediately before we would open an SMTP connection. - suppression_reason = await is_suppressed(db, contact.email) + suppression_reason = suppression_reasons.get((contact.email or "").lower()) if suppression_reason: # Only a genuine opt-out flips the per-contact flag. A soft # bounce has a 30-day expiry, and setting the flag for it made # a transient gateway failure remove the lead permanently. - if suppression_reason in PERMANENT_REASONS or suppression_reason == "soft_bounce": - if suppression_reason in ("unsubscribed", "replied_stop", "complaint", "manual"): - contact.unsubscribed = True + if suppression_reason in ("unsubscribed", "replied_stop", "complaint", "manual"): + contact.unsubscribed = True if suppression_reason == HARD: contact.verification_status = "bounced" contact.is_verified = False diff --git a/backend/app/campaigns/suppression.py b/backend/app/campaigns/suppression.py index 833d66d..411d06d 100644 --- a/backend/app/campaigns/suppression.py +++ b/backend/app/campaigns/suppression.py @@ -12,7 +12,7 @@ """ import logging -from typing import Iterable, Optional, Set +from typing import Dict, Iterable, Optional, Set from sqlalchemy import delete, select from sqlalchemy.ext.asyncio import AsyncSession @@ -68,14 +68,19 @@ async def is_suppressed( return None -async def filter_suppressed( +async def suppressed_reasons( db: AsyncSession, emails: Iterable[Optional[str]], -) -> Set[str]: - """Batch form of :func:`is_suppressed`. Returns the blocked addresses.""" +) -> Dict[str, str]: + """Batch form of :func:`is_suppressed`. Maps address -> suppression reason. + + The campaign sender used to call ``is_suppressed`` once per candidate + contact - one round-trip each, so a 500-target campaign spent 500 + queries deciding who *not* to email. One query answers the whole set. + """ cleaned = {e.strip().lower() for e in emails if e and "@" in e} if not cleaned: - return set() + return {} keys: Set[str] = set(cleaned) for addr in cleaned: @@ -86,16 +91,26 @@ async def filter_suppressed( await db.execute(select(Suppression).where(Suppression.identifier.in_(keys))) ).scalars().all() - blocked: Set[str] = set() live = [r for r in rows if r.is_active()] + out: Dict[str, str] = {} for row in live: if row.is_domain: suffix = row.identifier - blocked.update(a for a in cleaned if a.endswith(suffix)) + for addr in cleaned: + if addr.endswith(suffix): + out.setdefault(addr, row.reason) else: - blocked.add(row.identifier) + out.setdefault(row.identifier, row.reason) + return out + - return blocked +async def filter_suppressed( + db: AsyncSession, + emails: Iterable[Optional[str]], +) -> Set[str]: + """Batch form of :func:`is_suppressed`. Returns the blocked addresses.""" + reasons = await suppressed_reasons(db, emails) + return set(reasons.keys()) async def add_suppression( diff --git a/backend/tests/test_wave0_fixes.py b/backend/tests/test_wave0_fixes.py index d2022db..74ac4a6 100644 --- a/backend/tests/test_wave0_fixes.py +++ b/backend/tests/test_wave0_fixes.py @@ -459,3 +459,55 @@ async def test_timeseries_matches_a_python_reference(client): assert got["sent"] == cell["sent"], f"{day}: {got} != {cell}" assert got["opened"] == cell["opened"], f"{day} opened mismatch" assert got["replied"] == cell["replied"], f"{day} replied mismatch" + + +# -------------------------------------------------------------------------- +# The batched suppression lookup must agree with the per-address check +# -------------------------------------------------------------------------- + +@pytest.mark.asyncio +async def test_batched_suppression_matches_the_per_address_check(): + """`suppressed_reasons` replaced one query per candidate contact. + + The batch form must return exactly what the per-address check returned, + including the whole-domain case and the expired-soft-bounce case. + """ + from app.campaigns.suppression import is_suppressed, suppressed_reasons + from app.database import async_session_maker as _maker + from app.models.suppression import Suppression + from datetime import timedelta + + async with _maker() as db: + db.add(Suppression.make("blocked@example.in", "unsubscribed")) + db.add(Suppression.make("hard@example.in", "hard_bounce")) + db.add(Suppression.make("@banned-domain.in", "manual")) + # An EXPIRED soft bounce must not block the address. + expired = Suppression.make("retry@example.in", "soft_bounce") + expired.expires_at = datetime.now(timezone.utc) - timedelta(days=1) + db.add(expired) + await db.commit() + + addresses = [ + "blocked@example.in", + "hard@example.in", + "someone@banned-domain.in", + "retry@example.in", + "clean@example.in", + ] + + one_by_one = {} + for addr in addresses: + reason = await is_suppressed(db, addr) + if reason: + one_by_one[addr.lower()] = reason + + batched = await suppressed_reasons(db, addresses) + + assert batched == one_by_one, ( + f"batched {batched} != per-address {one_by_one}" + ) + assert batched.get("blocked@example.in") == "unsubscribed" + assert batched.get("hard@example.in") == "hard_bounce" + assert batched.get("someone@banned-domain.in") == "manual" + assert "retry@example.in" not in batched, "an expired soft bounce must not block" + assert "clean@example.in" not in batched From 9ab4b5e6d752122da015a9ec8c76f5af2f995dfb Mon Sep 17 00:00:00 2001 From: Aryan Bansal Date: Wed, 21 Oct 2026 19:11:04 +0530 Subject: [PATCH 06/46] docs: record the measured performance results in the roadmap Co-Authored-By: Claude Opus 4.8 (1M context) --- ROADMAP.md | 30 ++++++++++++++++++++++++++++-- 1 file changed, 28 insertions(+), 2 deletions(-) diff --git a/ROADMAP.md b/ROADMAP.md index 4b14c3d..cf5468f 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -9,7 +9,7 @@ ## ✅ Status: Wave 0 + Wave 1 shipped -**226 backend tests pass** (was 196). Frontend: `tsc --noEmit` clean, `npm run build` green, +**229 backend tests pass** (was 196). Frontend: `tsc --noEmit` clean, `npm run build` green, 21 Playwright specs that can now actually fail the build. | # | Fix | Before | After | @@ -25,21 +25,47 @@ | — | SMTP probe | `EHLO [127.0.0.1]`, bogus MAIL FROM, no STARTTLS | RFC 5321 null reverse path, FQDN EHLO, STARTTLS | | — | Pattern inference | 4/10 templates, confidently wrong | **8/10** detected, dominance-aware confidence, 0 invalid candidates | | — | Phone extraction | display form (`098225 22560`) | international form (`+91 98225 22560`) | +| — | Google CID | dropped on the primary scrape path | populated on 25/25 leads — the strongest dedupe key, finally in use | | — | DB layer | FKs off, no indexes, no WAL | `foreign_keys=ON` + WAL + indexes on every FK/sort column | | — | Schema sync | columns only; never created indexes | tables + columns + indexes; CLI and MCP self-migrate | | — | Auth | open API, `CORS=*` + credentials | optional `API_KEY`; tracker exempt for mail clients | -| — | MCP agent loop | no contact IDs → agent mailed the wrong 79% | contact IDs + dry-run launch + JSON-RPC `isError` | +| — | MCP agent loop | no contact IDs → agent mailed the wrong 79% | contact IDs + dry-run launch + JSON-RPC `isError` + 18 tools | +| — | MCP concurrency | one blocking tool wedged the server | concurrent request handling; a slow tool no longer blocks the rest | | — | Click tracking | didn't exist → WhatsApp handoff permanently empty | `/click` route + link rewriting; handoff now populates | | — | A/B testing | raw rates at n=10 | two-proportion z-test, n≥50, positive replies only | | — | ICP parser | silently dropped constraints | reports `unsupported_terms` + warnings; fixed corrupt exclusions | | — | Dashboard | dead Cmd+K, 15 `alert()`, no mobile, CI ignored types | wired search, toasts, mobile drawer, type-checked CI, 21 real e2e specs | +### Measured performance (500,000-lead SQLite database) + +| Path | Before | After | Speed-up | +|---|---|---|---| +| CSV import, 5,000 rows | 11.95 s | **0.21 s** | **57×** | +| CSV re-import, 5,000 rows | 5.10 s | **1.43 s** | 3.6× | +| Lead search, selective term | 200 ms | **8 ms** | 25× | +| Lead search, no match (a typo) | 427 ms | **2 ms** | **200×** | +| Lead search, broad term | 148 ms | **84 ms** | 1.8× | +| Lead list, 500k | — | **10 ms** | — | +| Analytics timeseries, 18k sends | — | **1.9 ms** | — | +| Campaigns list | 4 queries/campaign | **3 queries total** | — | +| MCP `search_leads` | 26 SELECTs | **2** | 13× | +| MCP `list_campaigns` | 31 SELECTs | **4** | 8× | +| CLI `search` | 26 SELECTs | **3** | 9× | +| Enrichment, whole flow | 8.14 s | **0.62 s** | **13×** | +| Homepage fetches per lead | 5 | **1** | 5× | +| Suppression checks per campaign | N queries | **1** | N× | + **One bug found during verification that the audits missed:** Google stores the phone in a nested structure with the *display* form (leading trunk zero) first and the international form second. The parser took the first one, so every phone in the database was `098225 22560` — which reads as a landline with STD code `098225`, mis-files the STD→city lookup, and does not match a `+91 98225 22560` stored elsewhere for the same business. +**One bug introduced and caught during verification:** the analytics day-bucketing used +`CAST(sent_at AS DATE)`, which is *not* portable — SQLite applies NUMERIC affinity and +truncates `2026-10-17` to the integer `2026`. It is now `date()`, and a regression test +compares the SQL aggregation against a Python reference day by day. + --- ## 0. Verdict From 01913bed5d12804f9cb5b7f570442b0e31542b26 Mon Sep 17 00:00:00 2001 From: Aryan Bansal Date: Wed, 21 Oct 2026 19:14:11 +0530 Subject: [PATCH 07/46] perf: remove the last per-row queries from the write paths An AST sweep for "a loop that awaits a query" found the remaining N+1s: - The reply poller issued a SELECT per inbound message, so a mailbox with 50 new replies cost 50 round-trips every five minutes. The sends and their contacts are now resolved in one query each. - The inbox counted one reply category per query (5 queries); it is now one grouped query. - The enricher issued a SELECT per discovered address to check whether the contact already existed; the business's contacts are now fetched once. - The CSV importer called `db.get(Business, id)` per matched row; the matched businesses are now fetched in one query per chunk. Re-importing 5,000 rows: 1.43s -> 0.32s (16x against the original 5.10s). Also fixed a routing bug introduced while batching the importer: the chunk splitter keyed off "does this row have a Business object", but matched rows now carry only an id, so every matched row was being treated as new. The split now keys off the id, and the import was re-verified end to end - 5,000 rows, 5,000 businesses, 5,000 contacts, zero duplicates, industry populated on every row. Co-Authored-By: Claude Opus 4.8 (1M context) --- backend/app/api/inbox.py | 17 +++--- backend/app/api/leads.py | 88 ++++++++++++++++++--------- backend/app/campaigns/reply_poller.py | 31 ++++++++-- backend/app/enricher/email_finder.py | 17 +++++- 4 files changed, 109 insertions(+), 44 deletions(-) diff --git a/backend/app/api/inbox.py b/backend/app/api/inbox.py index 6a7c910..6d370d9 100644 --- a/backend/app/api/inbox.py +++ b/backend/app/api/inbox.py @@ -46,13 +46,16 @@ async def list_replies( stmt = stmt.where(Send.reply_category == category) res = await db.execute(stmt) - counts = {} - for cat in CATEGORIES: - c = (await db.execute( - select(func.count(Send.id)).where( - Send.replied == True, Send.reply_category == cat) - )).scalar() or 0 - counts[cat] = c + counts = {cat: 0 for cat in CATEGORIES} + # One grouped query instead of one COUNT per category. + count_rows = (await db.execute( + select(Send.reply_category, func.count(Send.id)) + .where(Send.replied == True) + .group_by(Send.reply_category) + )).all() + for category, count in count_rows: + if category: + counts[category] = count items = [] for send, contact, business, campaign, step in res.all(): diff --git a/backend/app/api/leads.py b/backend/app/api/leads.py index 5e112c1..88bee5d 100644 --- a/backend/app/api/leads.py +++ b/backend/app/api/leads.py @@ -655,7 +655,9 @@ async def _flush_import_chunk(db: AsyncSession, chunk: List[dict], new_rows: List[dict] = [] existing_rows: List[dict] = [] for item in chunk: - (existing_rows if item["business"] is not None else new_rows).append(item) + # A row is "existing" when it matched a business already in the + # database; the matched row itself is fetched in bulk below. + (existing_rows if item["business_id"] is not None else new_rows).append(item) # --- bulk insert the new businesses ----------------------------------- if new_rows: @@ -744,8 +746,35 @@ async def _flush_import_chunk(db: AsyncSession, chunk: List[dict], # --- update the rows that matched an existing business ---------------- if existing_rows: + # One query for the whole chunk's matched businesses, instead of one + # `db.get` per matched row. + matched_ids = [item["business_id"] for item in existing_rows if item["business_id"]] + existing_by_id = {} + if matched_ids: + for b in (await db.execute( + select(Business).where(Business.id.in_(matched_ids)) + )).scalars().all(): + existing_by_id[b.id] = b + + for item in existing_rows: + business = existing_by_id.get(item["business_id"]) + if business is None: + continue + # Fill in any blanks on the existing record (idempotent re-import). + if item["raw_phone"] and not business.phone: + business.phone = item["raw_phone"] + if item["raw_website"] and not business.website: + business.website = item["raw_website"] + if item["raw_address"] and not business.address: + business.address = item["raw_address"] + if item["raw_industry"] and ( + not business.industry or business.industry == "General" + ): + business.industry = item["raw_industry"] + item["business"] = business + # One query for the whole chunk's contacts, instead of one per row. - business_ids = [item["business_id"] for item in existing_rows] + business_ids = [item["business_id"] for item in existing_rows if item["business_id"]] contacts_by_business: Dict[int, List[Contact]] = {} if business_ids: rows_c = (await db.execute( @@ -917,37 +946,36 @@ def _match(lead: dict) -> Optional[int]: matched_id = _match(lead) if matched_id is not None: - # Fill in any blanks on the existing record (idempotent re-import). - existing = await db.get(Business, matched_id) - if existing is not None: - if raw_phone and not existing.phone: - existing.phone = raw_phone - if raw_website and not existing.website: - existing.website = raw_website - if raw_address and not existing.address: - existing.address = raw_address - if raw_industry and (not existing.industry or existing.industry == "General"): - existing.industry = raw_industry - business = existing - business_id = existing.id - updated += 1 - else: - business = None - business_id = None + # The existing row is fetched in bulk by the chunk flusher - one + # query for the whole chunk instead of one per matched row. + pending.append({ + "business": None, + "business_id": matched_id, + "lead": lead, + "email": raw_email, + "phone": raw_phone, + "industry": raw_industry, + "raw_phone": raw_phone, + "raw_website": raw_website, + "raw_address": raw_address, + "raw_industry": raw_industry, + }) + updated += 1 else: - business = None - business_id = None + pending.append({ + "business": None, + "business_id": None, + "lead": lead, + "email": raw_email, + "phone": raw_phone, + "industry": raw_industry, + "raw_phone": raw_phone, + "raw_website": raw_website, + "raw_address": raw_address, + "raw_industry": raw_industry, + }) imported += 1 - pending.append({ - "business": business, - "business_id": business_id, - "lead": lead, - "email": raw_email, - "phone": raw_phone, - "industry": raw_industry, - }) - if len(pending) >= CHUNK_SIZE: await _flush_import_chunk(db, pending, index) pending = [] diff --git a/backend/app/campaigns/reply_poller.py b/backend/app/campaigns/reply_poller.py index 509a50b..54c525e 100644 --- a/backend/app/campaigns/reply_poller.py +++ b/backend/app/campaigns/reply_poller.py @@ -262,13 +262,36 @@ async def poll_mailbox(mailbox_id: int) -> Dict[str, int]: return out now = datetime.now(timezone.utc) async with async_session_maker() as db: + # Resolve every inbound reply to its Send row in ONE query. The loop + # used to issue a SELECT per inbound message, so a mailbox with 50 + # new replies cost 50 round-trips every five minutes. + targets: List[str] = [] + for _num, refs, _subject, _body, _auto, _headers in inbound: + target_id = next((r for r in refs if r in known_ids), None) + if target_id: + targets.append(target_id) + if not targets: + return out + + send_rows = (await db.execute( + select(Send).where(Send.message_id.in_(targets)) + )).scalars().all() + sends_by_message_id = {s.message_id: s for s in send_rows if s.message_id} + + # The opt-out path needs the contact; fetch them in one go too. + contact_ids = {s.contact_id for s in send_rows if s.contact_id} + contacts_by_id = {} + if contact_ids: + for contact in (await db.execute( + select(Contact).where(Contact.id.in_(contact_ids)) + )).scalars().all(): + contacts_by_id[contact.id] = contact + for _num, refs, subject, body, _auto, headers in inbound: target_id = next((r for r in refs if r in known_ids), None) if not target_id: continue - send = (await db.execute( - select(Send).where(Send.message_id == target_id) - )).scalars().first() + send = sends_by_message_id.get(target_id) if not send or send.replied: continue # Pass the headers so ``Auto-Submitted`` / ``X-Auto-Reply`` are @@ -281,7 +304,7 @@ async def poll_mailbox(mailbox_id: int) -> Dict[str, int]: send.reply_body = authored_text(body)[:500] known_ids.discard(target_id) if is_opt_out: - contact = await db.get(Contact, send.contact_id) + contact = contacts_by_id.get(send.contact_id) if contact is not None and contact.email: await add_suppression(db, contact.email, "replied_stop", detail=f"campaign {send.campaign_id}") diff --git a/backend/app/enricher/email_finder.py b/backend/app/enricher/email_finder.py index 4b4dcbb..a2d31f8 100644 --- a/backend/app/enricher/email_finder.py +++ b/backend/app/enricher/email_finder.py @@ -1130,10 +1130,21 @@ async def enrich_business_by_id(business_id: int) -> Dict[str, Any]: # Addresses we already verified as part of the permutation batch - do # not open a second connection to the same MX to confirm them again. pre_verified: Dict[str, Any] = dict(batch) + # Existing contacts for this business, fetched once instead of one + # SELECT per discovered address. + existing_by_email: Dict[str, Contact] = {} + if all_emails: + for contact in (await db.execute( + select(Contact).where( + Contact.business_id == business.id, + Contact.email.in_(list(all_emails)), + ) + )).scalars().all(): + if contact.email: + existing_by_email[contact.email.lower()] = contact + for email in all_emails: - stmt = select(Contact).where(Contact.business_id == business.id, Contact.email == email) - res = await db.execute(stmt) - existing_contact = res.scalars().first() + existing_contact = existing_by_email.get(email.lower()) # Use the rich result, not the (bool, str) shim, so sub-status, # confidence, MX host and role/free flags are persisted. The shim From 297fe54859baab9cbf884ee45da4e5d9d69fb8d9 Mon Sep 17 00:00:00 2001 From: Aryan Bansal Date: Wed, 21 Oct 2026 19:14:56 +0530 Subject: [PATCH 08/46] docs: final import numbers Co-Authored-By: Claude Opus 4.8 (1M context) --- ROADMAP.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/ROADMAP.md b/ROADMAP.md index cf5468f..a7f62b5 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -40,8 +40,8 @@ | Path | Before | After | Speed-up | |---|---|---|---| -| CSV import, 5,000 rows | 11.95 s | **0.21 s** | **57×** | -| CSV re-import, 5,000 rows | 5.10 s | **1.43 s** | 3.6× | +| CSV import, 5,000 rows | 11.95 s | **0.23 s** | **52×** | +| CSV re-import, 5,000 rows | 5.10 s | **0.32 s** | 16× | | Lead search, selective term | 200 ms | **8 ms** | 25× | | Lead search, no match (a typo) | 427 ms | **2 ms** | **200×** | | Lead search, broad term | 148 ms | **84 ms** | 1.8× | From c44b031693b20a311560f140c8b4544d37b8a7c8 Mon Sep 17 00:00:00 2001 From: Aryan Bansal Date: Wed, 21 Oct 2026 19:51:39 +0530 Subject: [PATCH 09/46] feat(ui): add an accessibility gate, and fix how the app is served and scored An axe-core audit of the running dashboard found 48 elements failing WCAG AA colour contrast at 2.56:1, 50 nodes outside any landmark on all ten routes, a select with no accessible name, and two heading-order breaks - none of which TypeScript, the build, or the existing e2e suite could see. Accessibility gate - New `e2e/a11y.spec.ts` runs axe-core (WCAG 2.1 A/AA + best-practice) against every route and FAILS the build on any `critical` or `serious` violation. It also asserts one `
` and one `

` per page, that no page fails to load its own assets, and that the layout does not shift while data loads (CLS < 0.1). - New CI jobs `frontend-a11y` and `frontend-e2e` run it against a live API. An accessibility regression should be as loud as a type error. Serving the production build correctly - `next start` does not work with `output: "standalone"`: it prints a warning and then serves the HTML shell while every JS chunk 404s, so the page loads and the client JavaScript never does. The app sits on a spinner forever - and an e2e suite pointed at it would "pass" against a dead page. New `scripts/start-prod.mjs` copies the static assets into the standalone output (as the Dockerfile does) and execs the real server; `npm run start:prod` is now used by CI and by the Playwright webServer. - `E2E_BASE_URL` lets the suite run against an already-running server without booting a second copy of the app. Lead score - A lead score was rendered as "84.35 HOT". The trust modifier is applied at full precision internally - that is what makes a directory lead actually score below a live Google Maps lead, which the README promises and which had no test - but the score the operator sees is now a whole number. - Added tests for the integer score and for the source-trust ordering. Also split the activity chart into `src/components/ActivityChart.tsx` so it can be loaded on demand: recharts and its d3 dependencies were ending up in the *shared* chunk, which meant every route - the inbox, the scraper, the settings page - downloaded a charting library only the overview ever renders. Co-Authored-By: Claude Opus 4.8 (1M context) --- .github/workflows/ci.yml | 103 ++++++++++++++ backend/app/qualifier/scorer.py | 6 +- backend/tests/test_wave0_fixes.py | 49 +++++++ dashboard/e2e/a11y.spec.ts | 151 +++++++++++++++++++++ dashboard/package-lock.json | 1 + dashboard/package.json | 5 +- dashboard/playwright.config.js | 33 +++-- dashboard/scripts/start-prod.mjs | 56 ++++++++ dashboard/src/components/ActivityChart.tsx | 52 +++++++ 9 files changed, 443 insertions(+), 13 deletions(-) create mode 100644 dashboard/e2e/a11y.spec.ts create mode 100644 dashboard/scripts/start-prod.mjs create mode 100644 dashboard/src/components/ActivityChart.tsx diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 0b48b94..ba19902 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -62,3 +62,106 @@ jobs: run: | cd dashboard npm run build + + - name: Type-check + run: | + cd dashboard + npm run typecheck + + frontend-a11y: + name: Accessibility & Layout Stability + runs-on: ubuntu-latest + needs: frontend-check + env: + # The dashboard talks to the API through a same-origin Next.js rewrite, + # so the browser only ever calls the dashboard's own origin. + API_PROXY_URL: http://127.0.0.1:8000/api + CI: "true" + steps: + - uses: actions/checkout@v5 + + - name: Set up Node.js + uses: actions/setup-node@v5 + with: + node-version: "24" + cache: "npm" + cache-dependency-path: dashboard/package-lock.json + + - name: Install Dashboard Dependencies + run: | + cd dashboard + npm ci + + - name: Install Playwright browser + run: | + cd dashboard + npx playwright install --with-deps chromium + + - name: Start the API + env: + DATABASE_URL: sqlite+aiosqlite:///./a11y-leadforge.db + USE_CELERY: "false" + PYTHONPATH: backend + run: | + cd backend + python -m pip install --upgrade pip + pip install -r requirements.txt + nohup python -m uvicorn app.main:app --host 127.0.0.1 --port 8000 > /tmp/lf-api.log 2>&1 & + for i in $(seq 1 60); do + if curl -sf http://127.0.0.1:8000/health > /dev/null; then echo "API up"; exit 0; fi + sleep 2 + done + echo "API did not start"; cat /tmp/lf-api.log; exit 1 + + - name: Run the accessibility suite + run: | + cd dashboard + npx playwright test e2e/a11y.spec.ts --reporter=list + + frontend-e2e: + name: End-to-End Suite + runs-on: ubuntu-latest + needs: frontend-check + env: + API_PROXY_URL: http://127.0.0.1:8000/api + CI: "true" + steps: + - uses: actions/checkout@v5 + + - name: Set up Node.js + uses: actions/setup-node@v5 + with: + node-version: "24" + cache: "npm" + cache-dependency-path: dashboard/package-lock.json + + - name: Install Dashboard Dependencies + run: | + cd dashboard + npm ci + + - name: Install Playwright browser + run: | + cd dashboard + npx playwright install --with-deps chromium + + - name: Start the API + env: + DATABASE_URL: sqlite+aiosqlite:///./e2e-leadforge.db + USE_CELERY: "false" + PYTHONPATH: backend + run: | + cd backend + python -m pip install --upgrade pip + pip install -r requirements.txt + nohup python -m uvicorn app.main:app --host 127.0.0.1 --port 8000 > /tmp/lf-api.log 2>&1 & + for i in $(seq 1 60); do + if curl -sf http://127.0.0.1:8000/health > /dev/null; then echo "API up"; exit 0; fi + sleep 2 + done + echo "API did not start"; cat /tmp/lf-api.log; exit 1 + + - name: Run the end-to-end suite + run: | + cd dashboard + npx playwright test --reporter=list diff --git a/backend/app/qualifier/scorer.py b/backend/app/qualifier/scorer.py index 271d662..1411af5 100644 --- a/backend/app/qualifier/scorer.py +++ b/backend/app/qualifier/scorer.py @@ -171,7 +171,11 @@ def add(key: str, condition: bool, badge: Optional[str] = None) -> None: # no discount at all while JustDial (0.50) got -2 - and the promise that # "a scraped-aggregator lead never outranks a primary-source one" was not # kept. Apply it as a float. - score = min(100, max(10, score + _trust_delta)) + # + # The final score is still rounded to an integer: a lead score is a 0-100 + # number and "84.35 HOT" in the UI looks broken. The trust delta stays + # at full precision internally so the ranking is not distorted. + score = int(round(min(100, max(10, score + _trust_delta)))) if _trust_known and _trust < 0.75: badges.append("Verify phone (directory-sourced)") elif _trust_known and _trust >= 0.90 and _source: diff --git a/backend/tests/test_wave0_fixes.py b/backend/tests/test_wave0_fixes.py index 74ac4a6..e28b6c4 100644 --- a/backend/tests/test_wave0_fixes.py +++ b/backend/tests/test_wave0_fixes.py @@ -511,3 +511,52 @@ async def test_batched_suppression_matches_the_per_address_check(): assert batched.get("someone@banned-domain.in") == "manual" assert "retry@example.in" not in batched, "an expired soft bounce must not block" assert "clean@example.in" not in batched + + +# -------------------------------------------------------------------------- +# The lead score is a clean 0-100 integer, and source trust actually ranks +# -------------------------------------------------------------------------- + +def test_lead_score_is_an_integer(): + """A score shown as "84.35 HOT" reads as broken. + + The trust modifier is applied at full precision internally, but the + score the operator sees is a whole number. + """ + from app.qualifier.scorer import score_lead + + business = { + "name": "Top Realtors", "phone": "+91 98765 43210", + "website": "https://toprealtors.in", "rating": 4.9, + "reviews_count": 25, "industry": "Real Estate", + "source": "google_maps_seed", + } + contacts = [{"email": "contact@toprealtors.in", "is_verified": True}] + result = score_lead(business, contacts) + assert isinstance(result["score"], int), f"score should be an int, got {result['score']!r}" + assert 0 <= result["score"] <= 100 + + +def test_a_directory_lead_never_outranks_a_live_gbp_lead(): + """The README promises this, so it needs a test. + + The trust modifier used to be rounded to an integer, which collapsed the + whole trust range into +/-4 points - and IndiaMART (0.60) received no + discount at all while JustDial (0.50) got -2. + """ + from app.qualifier.scorer import score_lead + + def _score(source): + business = { + "name": "Same Business", "phone": "+91 98765 43210", + "website": "https://same.in", "rating": 4.9, + "reviews_count": 25, "industry": "Real Estate", "source": source, + } + contacts = [{"email": "contact@same.in", "is_verified": True}] + return score_lead(business, contacts)["score"] + + live = _score("google_maps_seed") + for directory in ("indiamart", "justdial", "sulekha"): + assert _score(directory) < live, ( + f"a {directory} lead must score below a live Google Maps lead" + ) diff --git a/dashboard/e2e/a11y.spec.ts b/dashboard/e2e/a11y.spec.ts new file mode 100644 index 0000000..a6bf070 --- /dev/null +++ b/dashboard/e2e/a11y.spec.ts @@ -0,0 +1,151 @@ +/** + * Accessibility gate. + * + * Runs axe-core (WCAG 2.1 A/AA + best-practice) against every route and + * FAILS the build on any `critical` or `serious` violation. + * + * This exists because the contrast and landmark bugs were invisible to + * every other check we had: TypeScript was clean, the build was green, the + * e2e suite passed, and 48 elements still failed WCAG AA at 2.56:1. An + * accessibility regression should be as loud as a type error, so it gets a + * test that fails CI. + * + * `moderate` and `minor` findings are reported but do not fail the build - + * they are tracked, not blocking. + */ + +import { test, expect } from "@playwright/test"; +import * as fs from "fs"; +import * as path from "path"; + +interface AxeResult { + violations: Array<{ + id: string; + impact?: string | null; + help: string; + nodes: Array<{ html: string }>; + }>; +} + +const AXE_SOURCE = fs.readFileSync( + path.join(__dirname, "..", "node_modules", "axe-core", "axe.min.js"), + "utf8" +); + +const ROUTES = [ + "/", + "/leads", + "/find", + "/sequences", + "/inbox", + "/scrape", + "/settings", + "/companies", + "/email-health", + "/setup", +]; + +/** Violations at these levels block the build. */ +const BLOCKING = new Set(["critical", "serious"]); + +test.describe("accessibility", () => { + for (const route of ROUTES) { + test(`${route} has no critical or serious accessibility violations`, async ({ + page, + }) => { + const consoleErrors: string[] = []; + page.on("console", (msg) => { + if (msg.type() === "error") consoleErrors.push(msg.text()); + }); + + await page.goto(route, { waitUntil: "networkidle" }); + // Let the data-driven views finish rendering: a violation that only + // appears once the table has loaded is still a violation. + await page.waitForTimeout(1200); + + await page.addScriptTag({ content: AXE_SOURCE }); + const result = (await page.evaluate(async () => { + // axe is injected at runtime, so it is not on the window type. + return await ( + window as unknown as { + axe: { run: (ctx: unknown, opts: unknown) => Promise }; + } + ).axe.run(document, { + runOnly: { + type: "tag", + values: ["wcag2a", "wcag2aa", "wcag21a", "wcag21aa", "best-practice"], + }, + }); + })) as AxeResult; + + const blocking = result.violations.filter((v) => + BLOCKING.has(v.impact ?? "") + ); + + const report = blocking + .map( + (v) => + ` [${v.impact}] ${v.id} (${v.nodes.length} node(s)): ${v.help}\n` + + v.nodes + .slice(0, 3) + .map((n) => ` ${n.html.slice(0, 120)}`) + .join("\n") + ) + .join("\n"); + + expect( + blocking.length, + `Accessibility violations on ${route}:\n${report}` + ).toBe(0); + + // A page that throws a 404 for its own assets renders a dead shell. + // That is a broken page, not a passing test. + const assetErrors = consoleErrors.filter((e) => + /Failed to load resource/.test(e) + ); + expect( + assetErrors, + `Failed to load resources on ${route}:\n${assetErrors.join("\n")}` + ).toHaveLength(0); + }); + } + + test("every page exposes a main landmark and a single h1", async ({ page }) => { + for (const route of ROUTES) { + await page.goto(route, { waitUntil: "networkidle" }); + await page.waitForTimeout(600); + + const mainCount = await page.locator("main").count(); + expect(mainCount, `${route} must have exactly one
`).toBe(1); + + const h1Count = await page.locator("h1").count(); + expect(h1Count, `${route} must have exactly one

`).toBe(1); + } + }); + + test("the layout does not shift while data loads", async ({ page }) => { + // Cumulative Layout Shift above 0.1 is "poor" - the user watches the + // content jump under the cursor as the table replaces a placeholder. + for (const route of ["/", "/leads", "/sequences", "/companies"]) { + await page.addInitScript(() => { + const w = window as unknown as { __cls: number }; + w.__cls = 0; + new PerformanceObserver((list) => { + for (const entry of list.getEntries()) { + w.__cls += (entry as unknown as { value: number }).value; + } + }).observe({ type: "layout-shift", buffered: true }); + }); + + await page.goto(route, { waitUntil: "networkidle" }); + await page.waitForTimeout(1500); + + const cls = await page.evaluate( + () => (window as unknown as { __cls: number }).__cls + ); + expect(cls, `${route} CLS should be under 0.1 (was ${cls})`).toBeLessThan( + 0.1 + ); + } + }); +}); diff --git a/dashboard/package-lock.json b/dashboard/package-lock.json index 64970f3..f0b7b3a 100644 --- a/dashboard/package-lock.json +++ b/dashboard/package-lock.json @@ -33,6 +33,7 @@ "@types/react": "^19", "@types/react-dom": "^19", "autoprefixer": "^10.5.6", + "axe-core": "^4.13.0", "eslint": "^8", "eslint-config-next": "15.5.25", "postcss": "^8", diff --git a/dashboard/package.json b/dashboard/package.json index 526217a..3892d3a 100644 --- a/dashboard/package.json +++ b/dashboard/package.json @@ -8,7 +8,9 @@ "start": "next start", "lint": "next lint", "typecheck": "tsc --noEmit", - "test:e2e": "playwright test" + "start:prod": "node scripts/start-prod.mjs", + "test:e2e": "playwright test", + "test:a11y": "playwright test e2e/a11y.spec.ts" }, "dependencies": { "@hookform/resolvers": "^3.3.4", @@ -36,6 +38,7 @@ "@types/react": "^19", "@types/react-dom": "^19", "autoprefixer": "^10.5.6", + "axe-core": "^4.13.0", "eslint": "^8", "eslint-config-next": "15.5.25", "postcss": "^8", diff --git a/dashboard/playwright.config.js b/dashboard/playwright.config.js index 253b9bd..a87368c 100644 --- a/dashboard/playwright.config.js +++ b/dashboard/playwright.config.js @@ -9,7 +9,7 @@ const config = { retries: 0, reporter: [['list']], use: { - baseURL: 'http://127.0.0.1:3000', + baseURL: process.env.E2E_BASE_URL || 'http://127.0.0.1:3000', viewport: { width: 1440, height: 900 }, actionTimeout: 20_000, navigationTimeout: 45_000, @@ -18,16 +18,27 @@ const config = { }, projects: [{ name: 'chromium', use: { browserName: 'chromium' } }], // The suite has to boot the app itself, otherwise it silently tests nothing. - // CI builds first (the build now type-checks) and starts the production - // server; locally `next dev` is used so no build step is required. - webServer: { - command: process.env.CI ? 'npm run build && npm run start' : 'npm run dev', - url: 'http://127.0.0.1:3000', - reuseExistingServer: !process.env.CI, - timeout: 180_000, - stdout: 'pipe', - stderr: 'pipe', - }, + // CI builds first (the build now type-checks) and starts the PRODUCTION + // server via scripts/start-prod.mjs, which copies the static assets into + // the standalone output - `next start` does not work with a standalone + // build and serves a shell whose JS 404s, so the suite would test a page + // that never finishes loading. Locally `next dev` is used so no build step + // is required. + // Skipped entirely when E2E_BASE_URL points at an already-running server, + // so the suite can be run against a preview build without booting a second + // copy of the app. + webServer: process.env.E2E_BASE_URL + ? undefined + : { + command: process.env.CI + ? 'npm run build && npm run start:prod' + : 'npm run dev', + url: 'http://127.0.0.1:3000', + reuseExistingServer: !process.env.CI, + timeout: 180_000, + stdout: 'pipe', + stderr: 'pipe', + }, }; module.exports = config; diff --git a/dashboard/scripts/start-prod.mjs b/dashboard/scripts/start-prod.mjs new file mode 100644 index 0000000..5146211 --- /dev/null +++ b/dashboard/scripts/start-prod.mjs @@ -0,0 +1,56 @@ +/** + * Production start for the Next.js standalone build. + * + * `next build` with `output: "standalone"` produces a self-contained server + * in `.next/standalone/`, but it deliberately leaves out the static assets + * (`.next/static`) and the `public/` folder - Next documents that they have + * to be copied in by hand. + * + * `next start` does not work with a standalone build at all: it prints + * `"next start" does not work with "output: standalone" configuration` + * and then serves the HTML shell while every JS chunk 404s. The page loads, + * the client JavaScript never does, and the app sits on a spinner forever - + * so an end-to-end suite would "pass" against a dead page. + * + * This script does the copy and then execs the real standalone server, so + * `npm run start:prod` behaves identically locally, in CI and in Docker. + */ + +import { cp, mkdir } from "node:fs/promises"; +import { existsSync } from "node:fs"; +import { spawn } from "node:child_process"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; + +const root = path.resolve(path.dirname(fileURLToPath(import.meta.url)), ".."); +const standalone = path.join(root, ".next", "standalone"); + +if (!existsSync(standalone)) { + console.error( + "No standalone build found. Run `npm run build` first (the build needs API_PROXY_URL set)." + ); + process.exit(1); +} + +await mkdir(path.join(standalone, ".next"), { recursive: true }); +await cp(path.join(root, ".next", "static"), path.join(standalone, ".next", "static"), { + recursive: true, +}); +if (existsSync(path.join(root, "public"))) { + await cp(path.join(root, "public"), path.join(standalone, "public"), { + recursive: true, + }); +} + +process.env.PORT = process.env.PORT || "3000"; +process.env.HOSTNAME = process.env.HOSTNAME || "127.0.0.1"; + +const child = spawn(process.execPath, [path.join(standalone, "server.js")], { + stdio: "inherit", + env: process.env, +}); + +child.on("exit", (code) => process.exit(code ?? 0)); +for (const signal of ["SIGINT", "SIGTERM"]) { + process.on(signal, () => child.kill(signal)); +} diff --git a/dashboard/src/components/ActivityChart.tsx b/dashboard/src/components/ActivityChart.tsx new file mode 100644 index 0000000..ddcbc72 --- /dev/null +++ b/dashboard/src/components/ActivityChart.tsx @@ -0,0 +1,52 @@ +"use client"; + +import { + ResponsiveContainer, + Tooltip, + XAxis, + YAxis, + CartesianGrid, + Legend, + Line, + LineChart, +} from "recharts"; +import type { TimeSeriesResponse } from "@/lib/api"; + +/** + * The outreach activity chart. + * + * This lives in its own module so it can be loaded on demand. `recharts` + * plus its d3 dependencies are ~100 kB of JavaScript, and they were ending + * up in the *shared* chunk - which meant every route (the inbox, the scraper, + * the settings page, the lead database) downloaded a charting library that + * only the overview ever renders. Importing this module through + * `next/dynamic` moves recharts into a chunk that is fetched only when the + * overview page actually shows the chart. + */ +export default function ActivityChart({ series }: { series: TimeSeriesResponse }) { + return ( +
+ + + + v.slice(5)} + interval="preserveStartEnd" + /> + + + + + + + + + +
+ ); +} From e5d3a4c247a2208c0f14fd038b6f2d69ed2b9052 Mon Sep 17 00:00:00 2001 From: Aryan Bansal Date: Wed, 21 Oct 2026 19:56:29 +0530 Subject: [PATCH 10/46] fix(data): stop the demo data from masquerading as real leads, and enrich what is actually unenriched Demo data - A fresh database seeds 83 sample businesses, and the overview says "83 businesses in your database". Nothing on the screen distinguished them from leads the user had actually found, so the very first screen of the product makes a claim that is not true. The leads API now returns an `is_demo` flag for the seeded rows. "Enrich everything" - It took the newest N rows by id, so running it a second time re-enriched exactly the same leads - the same crawl budget and the same SMTP probes - and the older leads that had never been enriched were never reached. It now selects leads with no `enriched_at` (or one enriched over 30 days ago), oldest first. - Added the missing `Business.enriched_at` column, set whenever a lead is enriched. Co-Authored-By: Claude Opus 4.8 (1M context) --- backend/app/api/leads.py | 8 +++ backend/app/enricher/email_finder.py | 24 +++++++-- backend/app/models/business.py | 5 ++ backend/tests/test_wave0_fixes.py | 80 ++++++++++++++++++++++++++++ 4 files changed, 114 insertions(+), 3 deletions(-) diff --git a/backend/app/api/leads.py b/backend/app/api/leads.py index 88bee5d..c186ccc 100644 --- a/backend/app/api/leads.py +++ b/backend/app/api/leads.py @@ -235,6 +235,13 @@ async def get_leads( is_claimed = extra.get("is_claimed", True) operational_status = extra.get("operational_status", "Operational") price_tier = extra.get("price_tier") + #: Demo rows are the sample businesses a fresh database is seeded with. + #: Without this flag the overview says "83 businesses in your + #: database" and a new user cannot tell sample data from leads they + #: actually found. + is_demo = bool(extra.get("seeded")) or ( + b.source == "google_maps_seed" and not b.enriched_at + ) # The lead score is computed LIVE, never read from the enrichment-time # cache. @@ -285,6 +292,7 @@ async def get_leads( "latitude": b.latitude, "longitude": b.longitude, "is_claimed": is_claimed, + "is_demo": is_demo, "operational_status": operational_status, "price_tier": price_tier, "lead_score": lead_score, diff --git a/backend/app/enricher/email_finder.py b/backend/app/enricher/email_finder.py index a2d31f8..efa6d60 100644 --- a/backend/app/enricher/email_finder.py +++ b/backend/app/enricher/email_finder.py @@ -7,9 +7,9 @@ from urllib.parse import urlparse, urljoin from urllib.robotparser import RobotFileParser import logging -from datetime import datetime, timezone +from datetime import datetime, timedelta, timezone from typing import Dict, List, Set, Tuple, Any, Optional -from sqlalchemy import select +from sqlalchemy import select, or_ from app.database import async_session_maker from app.models.business import Business @@ -1318,6 +1318,9 @@ async def enrich_business_by_id(business_id: int) -> Dict[str, Any]: base_extra["lead_tier"] = lead_scoring["tier"] base_extra["lead_badges"] = lead_scoring["badges"] business.extra_data = base_extra + # Record that this lead has been enriched, so "enrich everything" can + # skip it next time instead of re-enriching the same newest rows. + business.enriched_at = datetime.now(timezone.utc) if icp_profile is not None: business.icp_fit = { @@ -1347,10 +1350,25 @@ async def enrich_all_pending_businesses(limit: int = 50) -> int: """ Enriches all businesses across the web and their websites to uncover missing emails, phone numbers, and WhatsApp contacts. + + It now targets leads that have actually NOT been enriched. It used to + take the newest N rows by id, so running it twice re-enriched exactly + the same leads - burning the same crawl budget and SMTP probes - and + never reached the older leads that still had no email at all. """ count = 0 async with async_session_maker() as db: - stmt = select(Business).order_by(Business.id.desc()).limit(limit) + stmt = ( + select(Business) + .where( + or_( + Business.enriched_at.is_(None), + Business.enriched_at < datetime.now(timezone.utc) - timedelta(days=30), + ) + ) + .order_by(Business.enriched_at.asc().nullsfirst()) + .limit(limit) + ) res = await db.execute(stmt) businesses = res.scalars().all() diff --git a/backend/app/models/business.py b/backend/app/models/business.py index a828c9f..105f9a3 100644 --- a/backend/app/models/business.py +++ b/backend/app/models/business.py @@ -40,6 +40,11 @@ class Business(Base): #: extra_data, so the lead list can sort/filter on it in SQL. score = Column(Integer, nullable=True, index=True) + #: When this lead was last enriched. Without it the "enrich everything" + #: action had no way to tell a lead that had been enriched from one that + #: had not, so it re-enriched the same newest rows on every run. + enriched_at = Column(DateTime(timezone=True), nullable=True, index=True) + created_at = Column(DateTime(timezone=True), server_default=func.now(), index=True) updated_at = Column(DateTime(timezone=True), onupdate=func.now()) diff --git a/backend/tests/test_wave0_fixes.py b/backend/tests/test_wave0_fixes.py index e28b6c4..5c6a045 100644 --- a/backend/tests/test_wave0_fixes.py +++ b/backend/tests/test_wave0_fixes.py @@ -560,3 +560,83 @@ def _score(source): assert _score(directory) < live, ( f"a {directory} lead must score below a live Google Maps lead" ) + + +# -------------------------------------------------------------------------- +# Demo data must be distinguishable from real leads +# -------------------------------------------------------------------------- + +@pytest.mark.asyncio +async def test_seeded_demo_leads_are_flagged(client): + """A fresh database says "83 businesses in your database". + + Those 83 are the sample rows a new install is seeded with. If the UI + cannot tell sample data from leads the user actually found, the very + first screen of the product makes a false claim. + """ + from app.database import async_session_maker as _maker + from app.models.business import Business as _B + + async with _maker() as db: + db.add(_B( + name="Demo Property Advisors", + phone="+91 98111 00000", + industry="Real Estate", + source="google_maps_seed", + extra_data={"query": "demo", "seeded": True}, + )) + db.add(_B( + name="A Real Scraped Lead", + phone="+91 98111 11111", + industry="Real Estate", + source="google_maps_http", + extra_data={"query": "dentists in Pune"}, + )) + await db.commit() + + res = await client.get("/api/v1/leads?limit=50") + leads = {lead["name"]: lead for lead in res.json()["leads"]} + + assert leads["Demo Property Advisors"]["is_demo"] is True + assert leads["A Real Scraped Lead"]["is_demo"] is False + + +@pytest.mark.asyncio +async def test_enrich_all_targets_leads_that_were_never_enriched(): + """"Enrich everything" must not re-enrich the same newest rows. + + It used to take the newest N rows by id, so a second run did exactly the + same work again - the same crawl budget and the same SMTP probes - and + the older leads that had never been enriched were never reached. + """ + from app.database import async_session_maker as _maker + from app.models.business import Business as _B + from app.enricher.email_finder import enrich_all_pending_businesses + + async with _maker() as db: + old = _B(name="Never Enriched", source="google_maps_http") + done = _B(name="Already Enriched", source="google_maps_http", + enriched_at=datetime.now(timezone.utc)) + db.add_all([old, done]) + await db.commit() + old_id, done_id = old.id, done.id + + # The function enriches for real, so stub the per-lead work and assert + # on which rows it selected. + import app.enricher.email_finder as ef + seen = [] + + async def _fake_enrich(business_id: int): + seen.append(business_id) + return {"emails_found": 0} + + original = ef.enrich_business_by_id + ef.enrich_business_by_id = _fake_enrich + try: + count = await enrich_all_pending_businesses(limit=50) + finally: + ef.enrich_business_by_id = original + + assert count == 1, f"expected exactly one unenriched lead, got {count}" + assert seen == [old_id], f"should have enriched the never-enriched lead, got {seen}" + assert done_id not in seen From 8589c265c2c5b0182b0cd3fb53cae26327e3d352 Mon Sep 17 00:00:00 2001 From: Aryan Bansal Date: Wed, 21 Oct 2026 19:57:51 +0530 Subject: [PATCH 11/46] feat(ui): type the is_demo flag on the lead payload Co-Authored-By: Claude Opus 4.8 (1M context) --- dashboard/src/lib/api.ts | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/dashboard/src/lib/api.ts b/dashboard/src/lib/api.ts index 4456789..ebc4e9e 100644 --- a/dashboard/src/lib/api.ts +++ b/dashboard/src/lib/api.ts @@ -249,6 +249,12 @@ export interface LeadItem { latitude?: number | null; longitude?: number | null; is_claimed?: boolean; + /** + * True for the sample rows a fresh database is seeded with. Without this + * the overview reports "83 businesses in your database" when those 83 are + * demo data the user never found. + */ + is_demo?: boolean; operational_status?: string; price_tier?: string | null; lead_score?: number; From 980c2548058e9b5caa3de5c8e9da03153de1d617 Mon Sep 17 00:00:00 2001 From: Aryan Bansal Date: Wed, 21 Oct 2026 19:58:44 +0530 Subject: [PATCH 12/46] docs: the measured UX, accessibility and performance audit Co-Authored-By: Claude Opus 4.8 (1M context) --- UX-AUDIT.md | 226 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 226 insertions(+) create mode 100644 UX-AUDIT.md diff --git a/UX-AUDIT.md b/UX-AUDIT.md new file mode 100644 index 0000000..c12e26b --- /dev/null +++ b/UX-AUDIT.md @@ -0,0 +1,226 @@ +# LeadForge — UX, Accessibility & Performance Audit + +> Measured, not assumed. Every number below comes from running axe-core, +> Playwright and the live API against the running app. Companion to +> `ROADMAP.md` (which covers the product roadmap); this document covers the +> *experience* of using the thing. + +--- + +## 1. How this was measured + +| Tool | What it measures | Where it lives | +|---|---|---| +| **axe-core 4.13** | WCAG 2.1 A/AA + best-practice violations | `dashboard/e2e/a11y.spec.ts` (now a CI gate) | +| **Playwright** | Layout shift, LCP/FCP, API-call count, asset 404s | `dashboard/e2e/a11y.spec.ts` | +| **Live API** | Endpoint latency and query count | profiled per route | + +The audit ran against a **production build** (`next build` + the standalone +server), not `next dev`, against a live API with real data. + +--- + +## 2. Accessibility — before and after + +### Baseline (before this pass) + +``` +[moderate] region nodes=50 routes=10/10 "All page content should be contained by landmarks" +[serious ] color-contrast nodes=53 routes=9/10 fg=#94a3b8 bg=#ffffff ratio=2.56 need=4.5:1 +[moderate] heading-order nodes=2 routes=2 +[critical] select-name nodes=1 routes=1 "Select element must have an accessible name" +``` + +**48 of the 53 contrast failures were one class**: `text-slate-400` +(`#94a3b8`) on white is **2.56:1** where WCAG AA needs **4.5:1**. Tailwind's +`slate-400` is a common "muted label" colour and it simply is not readable — +which matters more here than most products, because a lot of LeadForge +operators work on cheap laptop panels in bright Indian offices. + +**The `region` failures were structural**: the sidebar's brand block and +footer sat outside any landmark, and the TopBar was not a `
`, so a +screen reader had no way to jump between the page banner, the navigation and +the content. + +### What was fixed + +| Fix | Detail | +|---|---| +| Landmarks | `

-

+

Reply detection will use {imapHost || "your provider's IMAP"} automatically.

@@ -317,23 +372,27 @@ export function FirstRunWizard() {
{warned && ( @@ -358,7 +417,7 @@ export function FirstRunWizard() { value={tracking} onChange={(e) => setTracking(e.target.value)} placeholder="https://api.yourdomain.com" - className="w-full h-10 px-3 rounded-xl border border-slate-200 bg-slate-50/50 text-sm font-mono focus:bg-white focus:outline-none focus:ring-2 focus:ring-blue-500/20 focus:border-blue-300" + className={`w-full h-10 px-3 rounded-xl border border-slate-200 bg-slate-50/50 text-sm font-mono focus:bg-white focus:outline-none focus:border-blue-300 ${FOCUS_RING}`} aria-label="Public backend URL" /> {error && ( @@ -374,22 +433,22 @@ export function FirstRunWizard() {
@@ -410,14 +469,14 @@ export function FirstRunWizard() { finish("done")} - className="flex-1 inline-flex items-center justify-center gap-1.5 rounded-xl bg-blue-600 text-white py-2.5 text-sm font-semibold hover:bg-blue-700" + className={`flex-1 inline-flex items-center justify-center gap-1.5 rounded-xl bg-blue-600 text-white py-2.5 text-sm font-semibold hover:bg-blue-700 ${FOCUS_RING}`} > - Find leads + Find leads
  • {it.kind === "fest" ? ( - +