diff --git a/.env.example b/.env.example index ea50c42..9e7151d 100644 --- a/.env.example +++ b/.env.example @@ -37,6 +37,21 @@ CORS_ORIGINS=* # Public base URL used in open/click tracking and unsubscribe links. TRACKING_DOMAIN=http://localhost:8000 +# --- API authentication (OPTIONAL but strongly recommended) ----------------- +# Unset => the REST API is OPEN. That is fine for a local clone-and-run, and +# unsafe the moment the port is reachable from a LAN: with CORS_ORIGINS="*" +# any website could delete your lead database or send cold email from your +# mailbox. +# +# When set, every endpoint requires it: +# curl -H "X-API-Key: " http://localhost:8000/api/v1/leads +# The campaign tracker (/api/v1/campaigns/tracker/*) is exempt - it serves +# the one-click unsubscribe and the click tracker, which are fetched by the +# recipient's mail client. +# +# Generate with: openssl rand -hex 32 +API_KEY= + # --- Scraper ---------------------------------------------------------------- # Optional rotating egress proxies for directory scraping, comma-separated. # Recommended for sustained Google Maps volume (residential/Datacenter IPs diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 0b48b94..ba19902 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -62,3 +62,106 @@ jobs: run: | cd dashboard npm run build + + - name: Type-check + run: | + cd dashboard + npm run typecheck + + frontend-a11y: + name: Accessibility & Layout Stability + runs-on: ubuntu-latest + needs: frontend-check + env: + # The dashboard talks to the API through a same-origin Next.js rewrite, + # so the browser only ever calls the dashboard's own origin. + API_PROXY_URL: http://127.0.0.1:8000/api + CI: "true" + steps: + - uses: actions/checkout@v5 + + - name: Set up Node.js + uses: actions/setup-node@v5 + with: + node-version: "24" + cache: "npm" + cache-dependency-path: dashboard/package-lock.json + + - name: Install Dashboard Dependencies + run: | + cd dashboard + npm ci + + - name: Install Playwright browser + run: | + cd dashboard + npx playwright install --with-deps chromium + + - name: Start the API + env: + DATABASE_URL: sqlite+aiosqlite:///./a11y-leadforge.db + USE_CELERY: "false" + PYTHONPATH: backend + run: | + cd backend + python -m pip install --upgrade pip + pip install -r requirements.txt + nohup python -m uvicorn app.main:app --host 127.0.0.1 --port 8000 > /tmp/lf-api.log 2>&1 & + for i in $(seq 1 60); do + if curl -sf http://127.0.0.1:8000/health > /dev/null; then echo "API up"; exit 0; fi + sleep 2 + done + echo "API did not start"; cat /tmp/lf-api.log; exit 1 + + - name: Run the accessibility suite + run: | + cd dashboard + npx playwright test e2e/a11y.spec.ts --reporter=list + + frontend-e2e: + name: End-to-End Suite + runs-on: ubuntu-latest + needs: frontend-check + env: + API_PROXY_URL: http://127.0.0.1:8000/api + CI: "true" + steps: + - uses: actions/checkout@v5 + + - name: Set up Node.js + uses: actions/setup-node@v5 + with: + node-version: "24" + cache: "npm" + cache-dependency-path: dashboard/package-lock.json + + - name: Install Dashboard Dependencies + run: | + cd dashboard + npm ci + + - name: Install Playwright browser + run: | + cd dashboard + npx playwright install --with-deps chromium + + - name: Start the API + env: + DATABASE_URL: sqlite+aiosqlite:///./e2e-leadforge.db + USE_CELERY: "false" + PYTHONPATH: backend + run: | + cd backend + python -m pip install --upgrade pip + pip install -r requirements.txt + nohup python -m uvicorn app.main:app --host 127.0.0.1 --port 8000 > /tmp/lf-api.log 2>&1 & + for i in $(seq 1 60); do + if curl -sf http://127.0.0.1:8000/health > /dev/null; then echo "API up"; exit 0; fi + sleep 2 + done + echo "API did not start"; cat /tmp/lf-api.log; exit 1 + + - name: Run the end-to-end suite + run: | + cd dashboard + npx playwright test --reporter=list diff --git a/.gitignore b/.gitignore index de019d9..432eb25 100644 --- a/.gitignore +++ b/.gitignore @@ -7,6 +7,9 @@ __pycache__/ *.sqlite *.db leadforge.db +# SQLite WAL sidecar files (created when journal_mode=WAL is enabled) +*.db-wal +*.db-shm *.log # Node & Next.js diff --git a/README.md b/README.md index d29867c..bf63476 100644 --- a/README.md +++ b/README.md @@ -10,7 +10,7 @@ [![Next.js 15](https://img.shields.io/badge/Frontend-Next.js%2015-black.svg?style=for-the-badge&logo=next.js&logoColor=white)](https://nextjs.org/) [![Model Context Protocol](https://img.shields.io/badge/MCP-Native%20Server-8A2BE2.svg?style=for-the-badge)](https://modelcontextprotocol.io/) [![Twenty CRM](https://img.shields.io/badge/CRM-Twenty%20Compatible-indigo.svg?style=for-the-badge)](https://twenty.com/) -[![Tests Passing](https://img.shields.io/badge/Tests-196%2F196%20Passing_(100%25)-10b981.svg?style=for-the-badge&logo=pytest&logoColor=white)](https://github.com/Aryanban/leadforge) +[![Backend Tests](https://img.shields.io/badge/Backend%20Tests-404%20Passing-10b981.svg?style=for-the-badge&logo=pytest&logoColor=white)](https://github.com/Aryanban/leadforge) [![Docker](https://img.shields.io/badge/Deployment-Docker%20Compose-2496ed.svg?style=for-the-badge&logo=docker&logoColor=white)](https://www.docker.com/)
@@ -32,10 +32,10 @@ Modern B2B revenue and agency teams spend thousands of dollars annually on SaaS 4. **Generic AI Hallucinations**: Standard LLM outreach generators produce bland, templated pitches that trigger commercial spam filters. **LeadForge** resolves these challenges through a unified, self-hosted autonomous architecture: -* **Concurrent HTTP Maps Scraper**: Speaks Google's internal protobuf `pb=` transport directly, so listings are parsed straight out of `APP_INITIALIZATION_STATE` with no browser in the loop — and a viewport-grid sweep fans queries out concurrently to blow past the ~20-results-per-query ceiling. -* **Seed-First Email Discovery & Non-Blocking Verification**: Reverses Cloudflare `data-cfemail` tokens, harvests real addresses as *seed evidence*, then **learns each domain's actual format** instead of guessing. Verification runs in a thread pool, groups candidates per domain to reuse one MX connection, and classifies into a 20-status enum via a multi-probe catch-all sentinel — never defaulting to valid. +* **Concurrent HTTP Maps Scraper**: Speaks Google's internal `search?tbm=map&pb=...` transport directly, so listings are parsed straight out of the positional-JSON response envelope with no browser in the loop — and a viewport-grid sweep fans queries out concurrently to blow past the ~20-results-per-query ceiling. The text query is kept **category-only** ("dentists", not "dentists in Delhi") so the viewport does the geographic scoping; that single change is worth **+76% unique leads per query** at identical request count. +* **Seed-First Email Discovery & Non-Blocking Verification**: Reverses Cloudflare `data-cfemail` tokens, harvests real addresses as *seed evidence*, then **learns each domain's actual format** instead of guessing. All 10 mailbox templates are detected by reverse-engineering the local part (not shape-guessing), name-based and shape-based evidence are unioned rather than exclusive, and confidence is dominance-aware so one ambiguous seed never overrides what most domains do. Verification runs off the event loop, groups candidates per domain to reuse one MX connection, and classifies into a 20-status enum via a multi-probe catch-all sentinel — never defaulting to valid. * **Catch-All Sentinel & 80+ Spam Trigger Linter**: Identifies permissive mail servers and lints subject/body copy against 80+ commercial spam triggers before delivery. -* **Algorithmic 1-on-1 AI Icebreakers**: Analyzes business reviews, sentiment ratings, and municipal pain points to craft tailored hooks. +* **Algorithmic 1-on-1 Icebreakers**: Category-, rating- and city-aware opening hooks generated on-device from industry templates — no LLM API key required. (A local-LLM path can be plugged in; the shipped implementation is deliberately template-based so the tool runs with zero external dependencies.) * **Native Model Context Protocol (MCP)**: Empowers autonomous AI agents (Google Antigravity, Claude, Cursor) to manage prospecting, enrichment, and cold campaigns end-to-end. --- @@ -47,14 +47,17 @@ Modern B2B revenue and agency teams spend thousands of dollars annually on SaaS | **Monthly Cost** | **$0 / Free Forever** | $99 / mo | $49 / mo | ~$50 / 10k leads | $149 / mo | | **Lead Limits** | **Unlimited** | 1,000 credits/mo | 500 searches/mo | Pay-per-record | 2,500 leads | | **Self-Hosted / Local Private Data** | **Yes (100% Local)** | No (SaaS Cloud) | No (SaaS Cloud) | No (SaaS Cloud) | No (SaaS Cloud) | -| **Google Maps Scraping** | **Built-in (HTTP `pb=` + Stealth fallback)** | Add-on | Not included | API only | Not included | +| **Google Maps Scraping** | **Built-in (HTTP `pb=` primary + browser fallback)** | Add-on | Not included | API only | Not included | | **Cloudflare `data-cfemail` Decoding** | **Yes (Native XOR cipher)** | No | No | No | No | -| **Email Pattern Learning** | **Yes (per-domain inference from harvested seeds)** | Proprietary | Paid Add-on | No | No | +| **Email Pattern Learning** | **Yes (per-domain inference from harvested seeds, 10 templates)** | Proprietary | Paid Add-on | No | No | | **Catch-All & Safe SMTP Verification** | **Yes (multi-probe catch-all, 20-status enum, never defaults to valid)** | Paid | Yes | No | Add-on | | **Cold Email Spam & Deliverability Linter**| **Yes (80+ words, 0-100 score)** | No | No | No | Included | | **Algorithmic 1-to-1 AI Icebreakers** | **Yes (Sentiment & pain points)** | Paid Add-on | No | No | Included | -| **Multi-Step Sequences & Bounce Guard** | **Yes (Delay days, hard/soft bounce auto-suppress)**| Included | Add-on | No | Included | -| **AI Agent Native (MCP Server)** | **Yes (Claude, Cursor, AGY)** | No | No | No | No | +| **People Table (one row per human)** | **Yes (role, city, size, turnover, stack — all filterable)** | Yes | No | No | No | +| **India-Aware Role Taxonomy** | **Yes (Proprietor, Promoter, MD, Principal)** | No | No | No | No | +| **Activity Log (what it did, and why)** | **Yes (every send, bounce, deferral)** | No | No | No | No | +| **Multi-Step Sequences & Bounce Guard** | **Yes (autonomous step delays, hard/soft bounce auto-suppress)**| Included | Add-on | No | Included | +| **AI Agent Native (MCP Server)** | **Yes (19 tools, Claude, Cursor, AGY)** | No | No | No | No | | **Open-Source CRM Integration** | **Twenty CRM & Webhooks** | HubSpot / SF only | Zapier only | Webhooks only | Zapier only | --- @@ -111,9 +114,10 @@ flowchart TD ## ⚡ Core Engineering Innovations ### 1. Concurrent HTTP Maps Scraping & Anti-Detection Fallback -* **Concurrent HTTP Maps Scraper (Primary Path)**: Reverse-engineers Google's internal `search?tbm=map&pb=...` protobuf endpoint — the same transport the Maps frontend itself uses. Parses the `APP_INITIALIZATION_STATE` payload directly, so it extracts **name, phone, address, rating, review count and coordinates with no browser at all**. A viewport-grid sweep fans queries out concurrently across a locality, which is how production tools scale past the ~20-results-per-query ceiling. -* **Playwright Stealth Fallback**: If the HTTP path is blocked or returns an empty payload, a headless-Chromium scraper takes over. Route interception aborts stylesheets, web fonts, tracking beacons and media (`**/*.{png,jpg,jpeg,svg,woff,woff2,ttf}`) for a **3x–5x reduction in bandwidth**, plus humanized deceleration and viewport jitter to avoid bot fingerprinting. -* **Humanized Deceleration & Viewport Jitter**: Simulates randomized mouse curves and natural scrolling velocities, avoiding bot fingerprinting. +* **Concurrent HTTP Maps Scraper (Primary Path)**: Reverse-engineers Google's internal `search?tbm=map&pb=...` endpoint — the same transport the Maps frontend itself uses. Parses the positional-JSON response envelope directly, so it extracts **name, phone, address, rating, review count and coordinates with no browser at all**. A viewport-grid sweep fans queries out concurrently across a locality, which is how production tools scale past the ~20-results-per-query ceiling. + *The text query must stay **category-only***: sending "dentists in Delhi" makes Google return one city-wide relevance set, so every viewport returns a near-identical top-20 and the sweep is wasted. Measured on the same 8-cell grid: `q="dentists in Delhi"` → 34 unique leads, `q="dentists"` → 60. +* **Playwright Fallback**: The HTTP path is the default; if it is blocked or returns an empty payload, a headless-Chromium scraper takes over. Route interception aborts stylesheets, images, web fonts and media for a large bandwidth reduction, and the browser context uses a randomised user agent, a randomised viewport and humanised page-load deceleration. (The browser path is genuinely a *fallback*: it is ~3.7x slower and cannot extract phone numbers, so the HTTP path is tried first.) +* **Humanized Deceleration & Viewport Jitter**: Randomised user agent, randomised viewport and randomised page-load delay, so the browser fingerprint does not repeat identically on every run. * **Unclaimed GBP Detection**: Evaluates DOM nodes for the canonical *"Claim this business"* anchor, immediately classifying listings with highest agency acquisition value. ### 2. Seed-Driven Pattern Inference & Cloudflare XOR Deobfuscation @@ -129,20 +133,46 @@ flowchart TD * **Multi-Probe Catch-All Detection**: a single random probe cannot separate a catch-all from a block or a greylist. LeadForge sends a reachability control, multiple random probes, and a known-invalid control, then classifies `CATCH_ALL / NOT_CATCH_ALL / GREYLISTED_RETRY / BLOCKED / ANTISPAM_GATEWAY`. * **No-Probe MX Awareness**: Gmail, Zoho, Outlook and Yahoo accept-all on `RCPT TO` then reject after `DATA`. Probing them yields false “valid” results *and* gets your egress IP penalised, so they are never probed. This matters disproportionately in India, where a Gmail address is a legitimate and often the *only* business address. * **Role, Free, Disposable & Typo Detection**: `info@`/`sales@`/`support@` are flagged as group mailboxes and never counted as a “person found” (the usual cause of inflated lead counts and complaint spikes); webmail is treated as a quality signal rather than a rejection reason; typo'd domains receive a `did_you_mean` suggestion instead of being silently dropped. -* **80+ Spam Keyword Linter**: Scores copy 0–100 against commercial spam triggers, subject-line extremes, all-caps density, exclamation ratios and hyperlink density. +* **80+ Spam Keyword Linter**: Scores copy 0–100 against commercial spam triggers and subject-line extremes, and flags the Indian cold-email clichés that US-trained lists miss ("Dear Sir/Madam", "hope this email finds you well", "kindly revert back"). ### 6. India-Specific Outreach Readiness * **TRAI-Accurate Phone Normalisation**: Trunk-zero STD codes resolve against a verified SDCA table (so `0481` → Kottayam and `0487` → Thrissur, not duplicates), landline vs. mobile is distinguished by subscriber-length rules, and everything is normalised to **E.164** `+91`. Foreign numbers are no longer mangled into `+91...`. * **Channel Routing**: Landlines route to voice/email; mobiles route to WhatsApp first — which is how Indian SMBs actually answer. * **Regulatory Advisor**: The app surfaces consent, DLT and TRAI guidance as **advisory information** and leaves every decision to the operator — LeadForge is self-hosted and does not gate or veto your sending. -* **Hinglish & Festival-Aware Copy**: Generates regionally natural phrasing and suggests culturally appropriate timing windows. -* **Source-Trust Scoring**: Weights leads by the observed reliability of each source (Google Maps vs. aggregator directories), so a scraped-aggregator lead never outranks a primary-source one. +* **Hinglish & Festival-Aware Copy**: Generates regionally natural phrasing and surfaces culturally appropriate timing windows, so the operator can adapt a step to a festival week before sending. +* **Source-Trust Scoring**: Weights leads by the documented reliability of each source (live Google Maps vs. self-listed aggregator directories), so a scraped-aggregator lead never outranks a primary-source one. The trust table covers every source the scrapers actually write — including the primary scrape path — and the modifier reported by the API is the one actually applied to the score. * **Warm WhatsApp Handoff**: Emits a pre-filled `wa.me` deep link so the operator can continue in the channel the lead will actually reply on. +### 7. A People Table That Doesn't Lie +* **One row per human, with the company joined in** — the surface Apollo is actually built around, and the one LeadForge was missing entirely. Every filter is a server-side predicate applied *before* the page limit, so a filtered page reports an honest total and every matching row is reachable. +* **An India-aware role taxonomy**: `Proprietor`, `Promoter`, `Managing Director`, `Principal` and `Partner` are how Indian SMBs name the person who can sign off. A US/EU title list files all five as "unknown" — and an unknown title is a founder the table cannot show you. Matching is on word boundaries, not substrings, because `"director"` contains `"cto"`. +* **People are read off the company's own website** — from JSON-LD (`Person` / `Organization.employee` with a `jobTitle`), from "Name — Title", from "Title: Name", and from "Name is a Role". Testimonials and reviews are stripped first, so a customer writing *"I bought my office from Mr. Omkar"* does not become a lead. +* **A person with no email is still a person.** A site that names its proprietor has told you who the decision maker is, even when the mailbox is role-only or the domain is catch-all. Dropping them for want of a verified address is what left the table with 13 rows out of 149 businesses. +* **Business targets are labelled as such.** Discovery finds companies; the people behind them may not be known yet. A row without a person is shown as a business target, never as a founder. +* **An activity log.** Every send, hard bounce, suppression, deferred send and A/B verdict is recorded with a *sentence* explaining it, so "why did this lead get no email?" is answerable without reading the source. The recorder never raises: an audit failure must not break the action being audited. + ### 4. Algorithmic 1-on-1 AI Outreach Icebreakers * **Review Sentiment Analysis**: Evaluates customer reviews, Google star ratings, municipal locations, and business categories. * **Personalized Hooks**: Automatically synthesizes authentic opening hooks, hyper-specific compliments, and category-relevant pain points tailored to each individual business prospect. +### Source coverage + +Not every directory can be read without a browser, and this table says so +rather than implying a coverage that is not there. The live status of each +source is available at runtime: `GET /api/v1/icp/sources/health` probes every +adapter concurrently and returns the reason for any failure. + +| Source | Status | What it actually returns | +| :--- | :--- | :--- | +| **Google Maps** | ✅ Working (primary) | Name, phone, website, address, rating, reviews, coordinates. Fastest path — no browser in the loop. | +| **Sulekha** | ✅ Working | Name, address, locality, the Sulekha quality score (out of 10) and the "Verified by Sulekha" badge. **No phone** — the number sits behind a click-to-call wall. | +| **IndiaMART** | ⚠️ Limited | A *product* supplier directory. Only the static promoted slots are server-rendered, so this returns real suppliers for a manufacturing/wholesale niche and nothing for a service niche. | +| **JustDial** | ❌ Not readable server-side | The listing grid is loaded by a client-side XHR from a private endpoint, and JustDial resolves the city from the request IP (a `/Pune/` URL still returns Mumbai). Needs a browser engine. | + +The Sulekha score is kept in its own field rather than folded into the Google +rating, because it is out of 10 and mixing the two would silently corrupt the +0–5 scale the scorer uses. + ### 5. Deterministic Lead Scoring & Bounce Suppression * **0–100 Weighted Quality Score**: Evaluates each lead across email deliverability ($w_e$), phone presence ($w_p$), website activity ($w_w$), Google rating ($w_r$), and unclaimed status ($w_u$): * **🔥 HOT (75–100)**: Verified email, active website, top ratings ($\ge 4.5$), high reviews. @@ -150,6 +180,53 @@ flowchart TD * **❄️ COLD (<45)**: Limited contact footprint. * **Classified Bounce Suppression**: SMTP responses are classified into **hard** (dead mailbox, domain gone, no-such-user → permanent suppression), **soft** (mailbox full, over-quota, greylisted → retry), and **policy** (blocked, content-filtered → *not* treated as a dead address). Suppression is enforced again at queue time, so a bounce that lands mid-batch stops the remaining sends. + +--- + +## 🔐 Security & Operations (read this before exposing the API) + +LeadForge is a single-user, self-hosted tool, so the local clone-and-run +experience stays zero-config — but two defaults are unsafe the moment the +port is reachable from a LAN (which this README recommends). + +**Set an API key.** With `API_KEY` unset the REST API is open, and +`CORS_ORIGINS` defaults to `*`. With `allow_credentials` enabled that +combination lets *any website on the internet* call your API with the +operator's session — delete your lead database, or send cold email from +your mailbox. + +```bash +# .env +API_KEY=$(openssl rand -hex 32) +CORS_ORIGINS=http://localhost:3000 +ENV=production +SECRET_KEY=$(openssl rand -hex 32) +``` + +With `API_KEY` set, every router requires `X-API-Key: ` (or an +`Authorization: Bearer ` header) **except** the campaign tracker — +the RFC 8058 one-click unsubscribe and the click tracker are fetched by the +recipient's mail client, not by the dashboard, so they must stay reachable. + +**Background loops.** The API runs two in-process loops: a 5-minute IMAP +reply sweep and a 5-minute campaign scheduler that advances multi-step +sequences on their own. Under `USE_CELERY=true` they are handed to Celery +beat instead (`app/workers/beat_schedule.py`) so they are never +double-counted. If you deploy with Compose, **do not remove the +`celery-beat` service** — without it, reply detection and sequence +scheduling never run. + +**Schema management.** The schema is reconciled on every boot: missing +tables are created, missing columns are added, and missing indexes are +created. The migration is additive only — it never drops or retypes +anything — so it is safe to run against a database you care about. The CLI +and the MCP server perform the same reconciliation, so they never depend +on the API process having started first. + +**Agent safety.** `leadforge_launch_campaign` performs a **dry run by +default**: it returns the exact recipient list and subject lines without +sending anything. An agent must pass `dry_run=false` to actually send. + --- ## 🛠️ Technical Specifications & Engineering Decisions @@ -157,11 +234,11 @@ flowchart TD | Dimension | Engineering Decision | Rationale | | :--- | :--- | :--- | | **Backend Framework** | FastAPI (Python 3.10+) | High-performance asynchronous execution, auto-generated OpenAPI documentation, and low-latency websocket streams. | -| **Maps Extraction** | HTTP protobuf `pb=` endpoint, Playwright Stealth fallback | Primary path parses `APP_INITIALIZATION_STATE` with no browser; fallback uses headless Chromium with asset-blocking. | -| **Frontend Dashboard** | Next.js 15 (App Router) + Tailwind CSS | Modern, responsive interface with server-side rendering, real-time job status streaming, and instant search filtering. | +| **Maps Extraction** | HTTP `pb=` endpoint (primary), headless-Chromium fallback | Primary path parses Maps' positional-JSON envelope with no browser; fallback uses headless Chromium with asset-blocking. | +| **Frontend Dashboard** | Next.js 15 (App Router) + Tailwind CSS | Modern, responsive interface with client-side live job-status polling, instant search filtering, and a working mobile layout. | | **Persistence Layer** | SQLAlchemy 2.0 Async + SQLite/PostgreSQL | Zero-configuration local development via SQLite with seamless production migration to PostgreSQL. | -| **Agent Interface** | Model Context Protocol (MCP) SDK | First-class integration with modern AI coding assistants and autonomous agents. | -| **CRM Integration** | Twenty CRM & Standard REST Webhooks | Direct synchronization with modern open-source CRMs without vendor lock-in. | +| **Agent Interface** | Model Context Protocol (MCP), stdio JSON-RPC 2.0 | First-class integration with modern AI coding assistants and autonomous agents. Hand-rolled transport (no SDK dependency), 19 tools, JSON-RPC error handling that surfaces tool failures to the model, and a `dry_run` default on campaign launch. | +| **CRM Integration** | Twenty CRM export & a standard REST webhook POST | Pushes a lead export to modern open-source CRMs without vendor lock-in. (Push-only today — no inbound sync and no signed retrying event stream.) | --- @@ -232,13 +309,14 @@ how to fix it: | :--- | :--- | :--- | | 0 | **`/setup`** | **Connections hub + guided wizard.** First-time visitors get a 3-step wizard in plain English: pick your email provider (Gmail/Outlook/Zoho/Yahoo presets fill every server setting and explain exactly where to find an app password), connect + live-test the mailbox, optionally set your public URL. Afterwards the hub shows a readiness checklist, mailbox overview, every environment variable with its purpose, **editable quick settings** (public URL + proxies — saved for you, no file editing), a copy-paste MCP config, and the confirmation that **no AI API key is needed**. | | 1 | **`/scrape`** | Enter a query like *"Dentists in Austin TX"* → concurrent Google Maps scrape saves listings, then auto-discovers public emails from the businesses' websites and directories. | -| 2 | **`/find`** | Alternative entry point: save an **Ideal Customer Profile** and run multi-source discovery against it (JustDial, directories, web). | +| 2 | **`/find`** | Alternative entry point: save an **Ideal Customer Profile** and run multi-source discovery against it (Google Maps, Sulekha, and the directory adapters — see [Source coverage](#source-coverage)). | | 3 | **`/leads`** | Browse & search your database. Click any row to open the **detail drawer**: contacts, verification badges, full send history, one-click company suppression. Save current filters as **saved views** (Apollo dynamic lists) and bulk-select → Add to Sequence. | | 4 | **`/companies`** | The Accounts view: one row per company with contact coverage (total vs verified), rating, and location. | -| 5 | **`/settings`** | **Add an SMTP mailbox** (e.g. Gmail + App Password) — the only hard requirement for sending. Daily limit is capped at 50/day for warmup safety. | -| 6 | **`/sequences`** | Apollo-style execution: stat-tabbed list (Active/Paused/Draft/Completed) → build sequences with `{{first_name}}`-style variables, **A/B subject variants** (winner auto-declared on reply rate once both variants clear 10 sends), select leads → **Add to Sequence** → **Launch** → monitor per-step performance → **Pause** anytime. Suppression, warmup ramps, daily limits and randomized send jitter are enforced automatically. | -| 7 | **`/inbox`** | Inbound replies detected by the IMAP reply poller (every 5 min, or "Poll now" on Email Health). Thread-attributed via stored Message-IDs; categorized Interested / Not interested / Opt-out / Away / Needs triage. **A "stop" reply auto-suppresses the address workspace-wide** — compliance gate 4 of 4. Humans can re-categorize. | -| 8 | **`/email-health`** | Mailbox reputation, warmup progress (ramp stage), daily budget bars, health-paused banners, per-mailbox **IMAP settings editor** (reply-detection host/port, empty = auto-derive from SMTP), and one-click reply polling. | +| 5 | **`/people`** | The People view: **one row per human**, with the company joined in. Filter by role, city, market tier, company size, turnover band and technology — the attributes an ICP is expressed in. A row says whether it is a **person** or a **business target** (a company with a phone and a WhatsApp number, but no human behind it yet), because a table that calls a company's phone-only row a "founder" is worse than no table at all. | +| 6 | **`/settings`** | **Add an SMTP mailbox** (e.g. Gmail + App Password) — the only hard requirement for sending. Daily limit is capped at 50/day for warmup safety. | +| 7 | **`/sequences`** | Apollo-style execution: stat-tabbed list (Active/Paused/Draft/Completed) → build sequences with `{{first_name}}`-style variables, **A/B subject variants** (winner auto-declared on reply rate once both variants clear 10 sends), select leads → **Add to Sequence** → **Launch** → monitor per-step performance → **Pause** anytime. Suppression, warmup ramps, daily limits and randomized send jitter are enforced automatically. | +| 8 | **`/inbox`** | Inbound replies detected by the IMAP reply poller (every 5 min, or "Poll now" on Email Health). Thread-attributed via stored Message-IDs; categorized Interested / Not interested / Opt-out / Away / Needs triage. **A "stop" reply auto-suppresses the address workspace-wide** — compliance gate 4 of 4. Humans can re-categorize. | +| 9 | **`/email-health`** | Mailbox reputation, warmup progress (ramp stage), daily budget bars, health-paused banners, per-mailbox **IMAP settings editor** (reply-detection host/port, empty = auto-derive from SMTP), and one-click reply polling. | ### What you need to provide (and what you don't) @@ -275,7 +353,7 @@ Prefer the terminal? The **CLI** and **MCP agent** usage are documented below. --- -## 🧪 Automated Test Suite (196/196 Passing) +## 🧪 Automated Test Suite LeadForge includes a comprehensive Pytest suite covering email status/role/free/typo/gibberish classification, multi-probe catch-all sentinel, SMTP verification status mapping, per-domain pattern learning, seed harvesting, lead scoring and source-trust ranking, spam checking, AI icebreakers, Indian phone/STD normalisation and DND scrubbing, deliverability (bounce classification, suppression, warm-up ramps, India send-timing advice), deduplication/entity resolution, discovery pipeline, and the sources + leads/campaign/ICP APIs: @@ -287,7 +365,32 @@ LeadForge includes a comprehensive Pytest suite covering email status/role/free/ (cd backend && ../.venv/bin/python -m pytest -q tests/) ``` -All **196/196 tests pass synchronously** in under 1s — no network, no browser, no external services required. +The current backend suite has **404 passing tests** and runs without external network services. The dashboard has **51 Playwright tests**, including route accessibility checks; those require the dashboard test environment. + +### Accessibility & layout stability are enforced in CI + +The dashboard has an **axe-core** gate (`dashboard/e2e/a11y.spec.ts`) that runs +WCAG 2.1 A/AA + best-practice checks against every route and **fails the build** +on any `critical` or `serious` violation. It also asserts exactly one `
` +and one `

` per page, that no page fails to load its own assets, and that +the layout does not shift while data loads (CLS < 0.1). + +```bash +# Accessibility gate on its own (needs the API running): +(cd dashboard && npm run test:a11y) + +# Or the whole end-to-end suite: +(cd dashboard && npm run test:e2e) +``` + +This exists because the accessibility defects were invisible to every other +check we had: TypeScript was clean, the build was green, the end-to-end suite +passed, and 48 elements still failed colour contrast at 2.56:1 where WCAG AA +requires 4.5:1. An accessibility regression should be as loud as a type error. + +The measured before/after numbers, the design findings and the remaining +product gaps are in [`UX-AUDIT.md`](./UX-AUDIT.md); the product roadmap is +in [`ROADMAP.md`](./ROADMAP.md). --- @@ -349,12 +452,12 @@ export PYTHONPATH=backend | Competency Area | Architecture & Implementation | Engineering Impact | | :--- | :--- | :--- | -| **Asynchronous Web Automation** | Playwright Stealth with network asset route interception | 3x–5x faster crawling; resilient extraction across dense Google Maps nodes without IP bans. | +| **Asynchronous Web Automation** | HTTP-first Maps transport with a Playwright fallback that intercepts asset routes | ~3.7x faster than the browser path, with full phone/website extraction the browser path cannot do. | | **Reverse Cipher Deobfuscation** | Cloudflare `data-cfemail` single-byte XOR reverse cipher | Unmasks protected executive emails directly from obfuscated DOM trees. | | **Network & Socket Engineering** | Direct DNS MX query resolution and low-level SMTP socket handshakes | Eliminates third-party API verification costs; prevents catch-all bounce penalties. | | **Deliverability Linter** | 80+ spam trigger word scanner + deliverability health scoring | Protects outbound sender reputation by catching spam flags before transmission. | | **Agentic AI Protocol (MCP)** | JSON-RPC 2.0 Model Context Protocol server exposing tool contracts | Enables autonomous agents to execute complex prospecting sequences end-to-end. | -| **Full-Stack SaaS Architecture** | FastAPI + Next.js 15 App Router + Twenty CRM webhook engine | Sub-second lead querying, batch CSV/JSON export, and one-click CRM ingestion. | +| **Full-Stack SaaS Architecture** | FastAPI + Next.js 15 App Router + Twenty CRM webhook export | Sub-second lead querying on indexed columns, batch CSV/JSON export, and one-click CRM ingestion. | --- diff --git a/ROADMAP.md b/ROADMAP.md new file mode 100644 index 0000000..18dcaf3 --- /dev/null +++ b/ROADMAP.md @@ -0,0 +1,876 @@ +# LeadForge — Apollo-Parity Roadmap (India) + +> Generated from 8 parallel codebase audits (scraping, enrichment, verification, targeting, +> outreach engine, backend/API, frontend, MCP/agent layer) + competitive research. +> Every finding below was **verified by reading code and executing it**, not inferred. +> Target: be faster, more accurate, and more complete than Apollo.io for the Indian SMB market. + +--- + +## ✅ Status: core reliability, People, and India workflow waves shipped + +**404 backend tests pass**. The dashboard currently lists 51 Playwright tests, including +route accessibility checks. Type checking and production builds are enforced in CI. + +| # | Fix | Before | After | +|---|-----|--------|-------| +| B1 | Mailbox daily cap | lifetime counter → bricked after 50 sends | `sent_today` is a calendar-day counter; rolling-24h is the real gate | +| B2 | Send gate | mailed `invalid`/`gibberish`/`no_mx` addresses | refuses 10 undeliverable states; un-enriched leads stay eligible | +| B3 | Sequence scheduler | manual re-click required forever | in-process 5-min loop + Celery beat entry; "waiting on a delay" ≠ "completed" | +| B4 | Maps `q=` split | 34 unique leads / 8 requests | **60** unique leads / 8 requests (**+76%**, measured live) | +| B5 | `has_email` pagination | 4 of 20 leads visible, `total` wrong, Next disabled | SQL predicate before LIMIT; correct totals; page 2 reachable | +| B6 | Sender-vs-recipient bounce | one bad sender config nuked the whole lead DB | `sender_rejected:` / `data_rejected:` never suppress a lead | +| B7 | Soft-bounce retry | 1 blip → 30-day suppression | 3-attempt budget before suppression | +| — | Enrichment concurrency | 9s/lead, serial, inside the event loop | bounded fan-out (8×) + async verifier → event loop never stalls | +| — | SMTP probe | `EHLO [127.0.0.1]`, bogus MAIL FROM, no STARTTLS | RFC 5321 null reverse path, FQDN EHLO, STARTTLS | +| — | Pattern inference | 4/10 templates, confidently wrong | **8/10** detected, dominance-aware confidence, 0 invalid candidates | +| — | Phone extraction | display form (`098225 22560`) | international form (`+91 98225 22560`) | +| — | Google CID | dropped on the primary scrape path | populated on 25/25 leads — the strongest dedupe key, finally in use | +| — | DB layer | FKs off, no indexes, no WAL | `foreign_keys=ON` + WAL + indexes on every FK/sort column | +| — | Schema sync | columns only; never created indexes | tables + columns + indexes; CLI and MCP self-migrate | +| — | Auth | open API, `CORS=*` + credentials | optional `API_KEY`; tracker exempt for mail clients | +| — | MCP agent loop | no contact IDs → agent mailed the wrong 79% | contact IDs + dry-run launch + JSON-RPC `isError` + 18 tools | +| — | MCP concurrency | one blocking tool wedged the server | concurrent request handling; a slow tool no longer blocks the rest | +| — | Click tracking | didn't exist → WhatsApp handoff permanently empty | `/click` route + link rewriting; handoff now populates | +| — | A/B testing | raw rates at n=10 | two-proportion z-test, n≥50, positive replies only | +| — | ICP parser | silently dropped constraints | reports `unsupported_terms` + warnings; fixed corrupt exclusions | +| — | Dashboard | dead Cmd+K, 15 `alert()`, no mobile, CI ignored types | wired search, toasts, mobile drawer, type-checked CI, 46 real e2e specs | + +### Wave 2 — The People table, and the honesty work + +Apollo's core surface is **one row per human**, with attributes you can filter on. +LeadForge had a business table with a nested contact, so the question the product +exists to answer — *"show me every owner in Pune"* — could not even be expressed. + +| # | Fix | Before | After | +|---|-----|--------|-------| +| P1 | People endpoint | no person list at all | `GET /api/v1/contacts` — one row per human, 14 server-side filters, honest totals | +| P2 | Fabricated founders | `first_name="Gurucity"`, `title="Owner / Decision Maker"` on every business | 126 of 149 rows counted as founders; now **no** invented names, and `person_identified` says which rows are real people | +| P3 | Location | a free-text address blob | `city` / `state` / `market_tier` derived on write; "Jain Mandir Rd" is not a city | +| P4 | Role taxonomy | US/EU titles only | `Proprietor`, `Promoter`, `Managing Director`, `Principal` — and word-boundary matching, because `"director"` contains `"cto"` | +| P5 | Decision-maker extraction | a 4-title regex | JSON-LD, `Name — Title`, `Title: Name`, `Name is a Role`; testimonials stripped first | +| P6 | People without an email | dropped on the floor | kept — a site that names its proprietor has told you who the decision maker is | +| P7 | Company attributes | columns that nothing wrote | headcount, turnover band and technology read off the company's own site | +| P8 | ICP-relative score | one global "how complete is this record" number | scored against the profile; the same lead moves when the ICP does | +| P9 | Structured ICP | a word list — the one thing a salesperson cannot work with | headcount, turnover, size, tier, state, role, decision-makers-only, require-email | +| P10 | Funnel | four flat counters | nine stages from "businesses found" to "interested", with the drop-off per stage | +| P11 | Directory sources | 1 of 4 working — silently | Sulekha **working**; JustDial and IndiaMART **say why** they cannot be read | +| P12 | HTTP layer | every directory request raised `TypeError` and was swallowed | httpx 0.26 proxy handling fixed; the layer works at all | +| P13 | Boot backfills | reported success, changed nothing | raw SQL; two silent ORM traps that undid every correction | +| P14 | Interrupted jobs | stuck at "running" forever | failed with the reason, so the operator can start them again | +| P15 | India features | built, called from nowhere | festival line localised, WhatsApp routing live, A/B winner declared by the sender | +| P16 | Copy kit | served by the API, reachable only by hand | in the step editor, where the email is written | +| P17 | Activity log | decisions were invisible | every send, bounce, suppression and deferral recorded, with the reason | +| P18 | Regional copy | Hinglish only | the same closers in Marathi, Telugu, Bengali, Kannada, Tamil and Gujarati | +| P19 | Job queue | `BackgroundTasks` — died with the process | Celery when enabled, in-process fallback, orphaned jobs failed with their reason | +| P20 | Send window | sent at 03:00 IST | deferred to 09:00–20:00 IST, per lead, never dropped | + +### Measured performance (500,000-lead SQLite database) + +| Path | Before | After | Speed-up | +|---|---|---|---| +| CSV import, 5,000 rows | 11.95 s | **0.23 s** | **52×** | +| CSV re-import, 5,000 rows | 5.10 s | **0.32 s** | 16× | +| Lead search, selective term | 200 ms | **8 ms** | 25× | +| Lead search, no match (a typo) | 427 ms | **2 ms** | **200×** | +| Lead search, broad term | 148 ms | **84 ms** | 1.8× | +| Lead list, 500k | — | **10 ms** | — | +| Analytics timeseries, 18k sends | — | **1.9 ms** | — | +| Campaigns list | 4 queries/campaign | **3 queries total** | — | +| MCP `search_leads` | 26 SELECTs | **2** | 13× | +| MCP `list_campaigns` | 31 SELECTs | **4** | 8× | +| CLI `search` | 26 SELECTs | **3** | 9× | +| Enrichment, whole flow | 8.14 s | **0.62 s** | **13×** | +| Homepage fetches per lead | 5 | **1** | 5× | +| Suppression checks per campaign | N queries | **1** | N× | + +**One bug found during verification that the audits missed:** Google stores the phone in a +nested structure with the *display* form (leading trunk zero) first and the international +form second. The parser took the first one, so every phone in the database was +`098225 22560` — which reads as a landline with STD code `098225`, mis-files the STD→city +lookup, and does not match a `+91 98225 22560` stored elsewhere for the same business. + +**One bug introduced and caught during verification:** the analytics day-bucketing used +`CAST(sent_at AS DATE)`, which is *not* portable — SQLite applies NUMERIC affinity and +truncates `2026-10-17` to the integer `2026`. It is now `date()`, and a regression test +compares the SQL aggregation against a Python reference day by day. + +--- + +## 0. Verdict + +> **This section is the original audit, kept as the record of what was wrong.** +> The scorecard below is the *audit's* view, before Wave 0, Wave 1 and the +> People table landed. The current state is in **Wave 2** above. + +LeadForge is a **competent single-shot mailer with a reply inbox attached**. The README +describes Apollo. The code describes something meaningfully different. + +The engineering instincts are good in places — RFC 8058 one-click unsubscribe, workspace-wide +suppression re-checked immediately before every SMTP call, a 20-state verification enum that +never coerces `unknown` → `valid`, a per-domain pattern learner, a real Cloudflare +`data-cfemail` decoder, per-test DB isolation in the test suite. Those are the pieces worth +keeping and defending. + +But five product-killing bugs mean **the core loop does not work today**, and roughly a third +of the README's headline claims are false. Fixing the five blockers plus Wave 1 is roughly +**3–5 days of work** and converts this from a demo into a product. + +**Scorecard vs Apollo (at the time of the audit):** + +| Capability | Apollo | LeadForge | Gap severity | +|---|---|---|---| +| Leads/day per IP | 10k+ | ~2,000 (and 5h of that is a serial enrichment loop) | 🔴 | +| Unique leads per scrape query | 100+ | 30 (viewport grid collapses — `q=` bug) | 🔴 | +| Named decision-maker found | ~40% of contacts | 8–15% | 🔴 | +| Multi-source ingestion | 275M contacts + 50+ providers | 1 working source (3 of 4 return 0) | 🔴 | +| Autonomous multi-step sequences | ✅ | ❌ no scheduler — requires manual re-click | 🔴 | +| Per-day send capacity | 50/mailbox/day, forever | 50/mailbox/**lifetime**, then bricked | 🔴 | +| Hard-bounce rate | <5% | ~25–30% (sends to known-invalid addresses) | 🔴 | +| ICP filter expressiveness | 20+ attributes | 3 loose filters | 🟠 | +| ICP fit scoring | per-saved-search, live | cached global contact-completeness score | 🟠 | +| Opportunities / pipeline board | ✅ | ❌ no model exists | 🟠 | +| Analytics | funnel + cohorts + rep leaderboard | 4 flat counters | 🟠 | +| People table (row = person) | ✅ core surface | ❌ only a company table | 🟠 | +| Click tracking | ✅ | ❌ no route exists | 🟠 | +| India: GSTIN / Udyam / DLT / pincode / city tier | ❌ | ❌ | 🟢 **our moat** | +| India: WhatsApp-first routing | ❌ | computed, then ignored | 🟢 **our moat** | +| India: STD→city resolution | ❌ | ✅ built (`in_phone.py`) | 🟢 | +| India: mobile vs landline | ❌ | ✅ built, never scored | 🟢 | + +--- + +## 1. Wave 0 — The five blockers (product does not work without these) + +### B1. Every mailbox bricks after 50 lifetime sends 🔴 P0 + +`backend/app/campaigns/sender.py:251` +```python +m_stmt = select(Mailbox).where(Mailbox.is_active == True, Mailbox.sent_today < Mailbox.daily_limit) +``` +`sent_today` is incremented at `sender.py:477` and **reset nowhere** — verified: the only +other writes are `api/mailboxes.py:69` and `seeds.py:198`, both at creation time. The +`warmup.py` docstring at `:18-21` calls it "a fast pre-filter" and names +`rolling_24h_count` as "the authoritative gate" — but the stale pre-filter fires first and +the mailbox never reaches the real gate. **After 50 emails the mailbox is dead forever.** + +```python +# sender.py:251 — drop the broken counter from the filter +m_stmt = select(Mailbox).where(Mailbox.is_active == True) +# sender.py:477 — stop persisting a drifting counter +# (expose rolling_24h_count in GET /mailboxes for the UI instead) +``` +**Gain:** unbounded daily throughput; `warmup.py`'s entire ramp/rolling-24h machinery +becomes reachable for the first time. + +--- + +### B2. The sender emails known-invalid addresses 🔴 P0 + +`backend/app/campaigns/sender.py:294-302` filters on exactly one exclusion: +```python +Contact.verification_status != "bounced" +``` +`should_send()` exists at `backend/app/verifier/email_status.py:379-389` and is **never +called from the sender** — only from the MCP display path. So `invalid`, `syntax_error`, +`gibberish`, `disposable`, `no_mx`, `unknown`, `catch_all` are all mailed. The verifier's +"never defaults to valid" guarantee is enforced in one file and discarded two files away. + +For an Indian SMB list where ~30% of scraped addresses are junk, that is a ~30% hard-bounce +rate on day one. Gmail/Yahoo's bulk-sender threshold is **5%**. + +```python +# sender.py:294 +Contact.verification_status.notin_( + ("bounced", "invalid", "syntax_error", "gibberish", "disposable", + "no_mx", "domain_error", "spamtrap", "abuse", "do_not_mail")) +``` +**Gain:** hard bounce rate ~25–30% → <5%. Sender reputation preserved. + +--- + +### B3. No scheduler — multi-step sequences never advance on their own 🔴 P0 + +`execute_campaign_step` has exactly two call sites: `api/campaigns.py:273` (manual Launch) +and `mcp_server.py:686`. `main.py:84-103` has a `_reply_poll_loop` but no campaign loop; +`workers/beat_schedule.py:8-13` contains only the IMAP sweep. The delay arithmetic at +`sender.py:374-376` is correct — **nothing ever calls it again at T+3 days.** An operator +must manually re-click Launch for every step of every sequence, forever. + +Compounding: `sender.py:381-385` marks the campaign `COMPLETED` whenever the queue is empty +— which is true both when everyone finished **and** when everyone is waiting on a delay. +The two are indistinguishable, so campaigns falsely report "Completed". + +```python +# app/main.py — mirror the existing reply loop +CAMPAIGN_TICK_SECONDS = 300 + +async def _campaign_run_loop() -> None: + while True: + try: + async with async_session_maker() as db: + ids = (await db.execute( + select(Campaign.id).where(Campaign.status == CampaignStatus.ACTIVE.value) + )).scalars().all() + for cid in ids: + await execute_campaign_step(cid) + except Exception as exc: + logger.warning("campaign tick error: %s", exc) + await asyncio.sleep(CAMPAIGN_TICK_SECONDS) +``` +Plus a Celery beat entry, and a per-campaign lock so the tick and a manual Launch can't +double-send. And distinguish "all steps done" from "waiting on a delay" at `sender.py:381`. + +**Gain:** the README's headline feature ("multi-step sequences with delay days") becomes +true. This is the difference between a mailer and an outreach engine. + +--- + +### B4. The `q=` parameter collapses the viewport grid by 3.4× 🔴 P0 + +`backend/app/scraper/maps_http.py:250` sends `"q": query` where `query` is +`build_query(niche, area)` → `"dentists in Delhi"`. Google then returns one city-wide +relevance set, so **every viewport in the grid returns a near-identical top-20**. + +Measured live (8-cell grid, Delhi, identical requests, only `q` varies): + +| `q` | raw rows | unique leads | +|---|---|---| +| `"dentists in Delhi"` (current) | 160 | **30** | +| `"dentists"` (category only) | 160 | **103** | + +Distance distribution confirms it: with `q="dentists"` the median result sits **1.3 km** from +its own viewport centre (p90 = 5.0 km); with `q="dentists in Delhi"` it sits 5.4 km away and +the set is identical across cells. The viewport grid is only geographic scoping when the +text query is category-only. + +```python +# sources/google_maps.py — split "what we're looking for" from "where" +async def search(self, niche, area, limit=20): + raw = await scrape_google_maps_http( + query=self.build_query(niche, area), # label only + q=industry_of_query(niche), # the actual request + max_results=limit, max_cells=8) +``` +**Gain: +243% unique leads per query at identical request count (30 → 103).** Highest +gain-per-diff in the entire codebase — roughly a 3-line change. + +--- + +### B5. `has_email` is applied *after* pagination and excluded from `total` 🔴 P0 + +`backend/app/api/leads.py:113-115`: +```python +if has_email is True and not any(c.get("email") for c in b_contacts): + continue +``` +This runs **after** `.offset(offset).limit(limit)` at `:82`, and `count_stmt` (`:64-77`) +has no `has_email` predicate at all. Verified against the repo's own DB (75 businesses, +20 with email): + +| Request | `total` reported | rows returned | +|---|---|---| +| `?page=1&limit=20` | 75 | 20 | +| `?page=1&limit=20&has_email=true` | **75** (should be 20) | **4** | +| `?page=2&limit=20&has_email=true` | 75 | 5 | + +The frontend compounds it — `dashboard/src/app/leads/page.tsx:874` disables Next when +`leads.length < 20`, so with a filtered subset the user is **permanently pinned to page 1** +and can never reach the other 16 leads. This is the Lead Database, the product's front door. + +```python +# api/leads.py — one predicate, used by BOTH the count and the page +email_ids = select(Contact.business_id).where(Contact.email.isnot(None)) +if has_email is True: + stmt = stmt.where(Business.id.in_(email_ids)) + count_stmt = count_stmt.where(Business.id.in_(email_ids)) +elif has_email is False: + stmt = stmt.where(Business.id.notin_(email_ids)) + count_stmt = count_stmt.where(Business.id.notin_(email_ids)) +``` +Delete the Python-side `continue` block, and change the frontend to paginate on `total`. + +**Gain:** the filter becomes truthful; pagination stops silently truncating. **16 of 20 +matching leads become reachable.** The same bug exists in `cli.py:127` and +`mcp_server.py:363` — fix all three, or extract one shared `build_lead_query()`. + +--- + +## 2. Wave 1 — Speed & accuracy (the "faster, better" mandate) + +### S1. Enrichment is 100% serialized: ~9s per lead, blocking the event loop + +`backend/app/discovery/pipeline.py:20-31` awaits `enrich_business_by_id` one lead at a time. +Measured **8.7s and 9.5s** per lead (Yahoo + Bing + 4-page crawl + social probe + SMTP). +A 100-lead job = **~15 minutes** inside the FastAPI event loop. `maps_scraper.py:528-534` +does the opposite — unbounded `asyncio.create_task` with no reference kept (CPython may GC +the task mid-flight) and no semaphore. + +```python +# discovery/pipeline.py +async def _enrich_new_leads(business_ids: List[int], concurrency: int = 8) -> int: + sem = asyncio.Semaphore(concurrency) + async def one(bid): + async with sem: + try: + await enrich_business_by_id(bid); return 1 + except Exception as e: + logger.warning("Enrichment failed for %s: %s", bid, e); return 0 + return sum(await asyncio.gather(*(one(b) for b in business_ids))) +``` +**Gain: ~8× throughput (9s → ~1.1s amortised). 100-lead job 15 min → ~2 min.** + +### S2. `email_finder.py` calls the *sync* SMTP verifier inside `async def` + +`backend/app/enricher/email_finder.py:453` → `check_catch_all_domain(w_domain)` (sync) +and `:501` → `verify_email(email)` (sync, up to 3 RCPT probes, 8s timeout each). + +`smtp_verifier.py:336-345` says verbatim: *"calling them directly stalls the whole event +loop for the 3-8s of a DNS+SMTP round trip"*. The module ships +`verify_email_async` / `check_catch_all_domain_async` — **the enricher doesn't use them.** + +**Gain:** API p99 during bulk enrichment goes from seconds to milliseconds; the whole API +stops freezing. + +### S3. Three SMTP sessions per domain per lead, and every email verified twice + +Per lead per domain the code opens **2–5 distinct MX connections** and performs **3–8 DNS +lookups**: `check_catch_all_domain` (conn #1) → `verify_batch` (conn #2) → a per-address +`verify_email` loop (conn #3, #4, #5…). `verify_batch` already computes the per-domain +catch-all verdict internally (`smtp_verifier.py:464`) but doesn't surface it, which is why +a *second* redundant probe was added. + +**Gain:** MX connections per lead **3–5 → 1–2**; wall-clock **−20–40%**; port-25 blocklist +exposure cut by an order of magnitude. + +### S4. The SMTP probe announces `EHLO [127.0.0.1]` and uses a bogus MAIL FROM + +`backend/app/verifier/smtp_verifier.py:138-148`: +```python +self.server = smtplib.SMTP(self.mx_host, port=25, timeout=SMTP_TIMEOUT) +self.server.ehlo_or_helo_if_needed() +self.server.mail(self.sender_email) # return code DISCARDED +``` +Four defects: (a) `smtplib.mail()` returns `(code, msg)` and does **not** raise — a +`550 5.7.1 Sender rejected` is silently dropped and the next RCPT gets `503`, which is in +no bucket → `UNKNOWN`; (b) the default envelope sender is `verify@leadforge.dev` — no MX, +no SPF, no DKIM, rejected by Gmail since Jan 2024 and by Postfix with +`reject_unknown_sender_hostname` (very common on Indian cPanel/Hostinger); (c) **no +STARTTLS** is attempted, so any MTA with `smtpd_tls_security_level=encrypt` answers +`530` → `UNKNOWN`; (d) `smtplib` defaults `local_hostname` to `socket.getfqdn()`, which in +Docker is the container ID with no dot → falls back to the domain literal +**`[127.0.0.1]`** — the canonical signature of an open-relay harvester. + +```python +self.server = smtplib.SMTP(self.mx_host, port=25, timeout=SMTP_TIMEOUT, + local_hostname=settings.SMTP_HELO_NAME) +self.server.ehlo_or_helo_if_needed() +if self.server.has_extn("starttls"): + self.server.starttls(context=ssl.create_default_context()) + self.server.ehlo_or_helo_if_needed() +code, msg = self.server.mail("<>") # RFC 5321 null reverse path +if code != 250: + self.error = f"mail_from_rejected:{code}"; return False +``` +**Gain:** verification stops returning `UNKNOWN` on hardened MTAs; the single most +blocklist-worthy signal in the codebase disappears. + +### S5. Catch-all detection: one random probe, and `unknown` silently means "not catch-all" + +`backend/app/verifier/smtp_verifier.py:262-315` sends **one** random probe and one +pseudo-control, then classifies `CATCH_ALL / NOT_CATCH_ALL / GREYLISTED_RETRY / BLOCKED` — +the README's "reachability control + multiple random probes + known-invalid control" and +the `ANTISPAM_GATEWAY` verdict **do not exist**. Worse, `"unknown"` is treated as *strict*: +an anti-probing MTA that 250s the first RCPT and 554s the second yields `"unknown"` → the +real address is reported **VALID at confidence 0.95**. + +**Gain:** eliminates the false-`valid` class — the #1 cause of sender-reputation burn on +catch-all-heavy Indian domains. + +### S6. `/api/v1/scraper/start` is Playwright-first and produces garbage leads + +`backend/app/api/scraper.py:35` → `maps_scraper.py:430` runs Playwright **first**, never the +HTTP path. Measured live (Delhi, 20 results): + +| field | HTTP path | Playwright path | +|---|---|---| +| wall clock | **2.4s** | **8.9s** | +| phone present | 20/20 | **0/20** | +| website present | 17/20 | **0/20** | +| valid `place_id` (`ChIJ…`) | 20/20 | **0/20** | +| address contains PUA glyph | 0/20 | **17/20** | +| `operational_status` | correct | `"Closed"` (misread) | + +The Playwright extractor keeps the category **icon glyph** as the first address token +(`maps_scraper.py:252-259`), and sets `place_id` from `href.split("?")[0][-60:]` — a +truncated URL fragment stored into a `unique=True` column that is the **primary dedupe key**. +Playwright-sourced and HTTP-sourced rows for the same business have incompatible keys. + +**Gain: ~4× faster and ~85% more usable fields per lead**, on the endpoint the dashboard's +scrape button actually calls. One-line reorder: try HTTP first. + +### S7. Three of four registered sources return zero leads + +Measured live: `justdial → 0`, `indiamart → 0` (the URL is a **404**; the endpoint moved +to `dir.indiamart.com/search.mp`), `sulekha → 0` (selectors are legacy Tailwind classes; +business names live in `

`), `google_maps → 20`. The "multi-source router" is +Google-Maps-only in practice. `tests/test_sources.py:61` asserts only that the names are +*registered*, never that they return anything — which is why all 196 tests pass while 3 +sources are dead. + +Also: **JustDial and Sulekha carry websites** (→ email enrichment) and verified/trust badges +that Maps doesn't have. Fixing them adds both volume *and* enrichability. + +### S8. Dedupe silently discards real leads and false-merges across cities + +`backend/app/dedupe/match.py:99` matches `Business.name == lead["name"]` with **no city +guard** — and Indian SMBs are dominated by generic names ("Sharma Dental Clinic", "Krishna +Medicals"). `dedupe/persist.py:39-45` treats a match as "already known" and `continue`s, so +the **Jaipur row is never inserted** and its phone/website/address are **thrown away**. + +Also verified: phone dedupe is **dead** (`match.py:60-67` compares normalised phone against +the raw stored value); domain matching is a **substring** so `example.com` matches +`smileshop.example.com`; and `normalize_name` is **computed then discarded** at +`match.py:97-102` (the query uses the raw name). + +**Gain:** at 50k rows the name-collision rate in Indian SMB data is 5–15%. Fixing this is +worth more than adding any new source. + +### S9. `rank_existing` loads the entire database to return 50 rows + +`backend/app/qualifier/ranker.py:108-130` — `select(Business)` with **no limit**, then +pure-Python scoring with **no `await` inside** (so it blocks the event loop for the full +duration). Benchmarked: **13.6s and 837 MB** at 100k rows, returned 50. Extrapolated to +500k: **~70s and ~4 GB per request.** This is the "rank my saved search" button. + +Same anti-pattern in the CSV export and the Twenty-CRM export (both unbounded full-table +joins, no LIMIT, no streaming). + +### S10. No indexes on any foreign key, filter, or sort column + +Live schema dump: +``` +contacts: ['ix_contacts_email', 'ix_contacts_id'] <-- NO business_id +sends: ['ix_sends_id', 'ix_sends_message_id'] <-- NO campaign_id/contact_id/mailbox_id/step_id +businesses: ix_businesses_{icp_profile_id,id,name,place_id} <-- NO created_at +``` +Measured at 500k businesses / 1M contacts (SQLite): +``` +BEFORE: contacts.business_id IN (25 ids) 42 ms + businesses ORDER BY created_at DESC LIMIT 25 32 ms +AFTER adding 2 indexes: + contacts.business_id IN (25 ids) <1 ms + businesses ORDER BY created_at DESC LIMIT 25 <1 ms +``` +**And the schema-sync mechanism won't fix it** — `main.py:25-58` only adds *columns*; it +never creates indexes or constraints, so existing deployments silently never get them. + +### S11. CSV import is O(rows) round-trips inside one transaction + +`backend/app/api/leads.py:453-546` — ~5 queries per row (4 for dedupe + 1 contact lookup), +one commit at the end. Measured: **5000 rows = 15.24s**, and it holds SQLite's global write +lock the whole time, so the entire app is unresponsive. + +### S12. SQLite foreign keys are OFF — every `ondelete="CASCADE"` is a no-op + +`backend/app/database.py:9-16` never issues `PRAGMA foreign_keys=ON`. SQLAlchemy doesn't do +it for you. `api/leads.py:385` uses a Core `delete(Business)` which bypasses the ORM's +`cascade="all, delete-orphan"` entirely → **every deleted lead orphans its contacts.** +Verified: `PRAGMA foreign_keys: 0`, orphan count 1 after one delete. + +```python +@event.listens_for(engine.sync_engine, "connect") +def _sqlite_pragmas(dbapi_conn, _rec): + cur = dbapi_conn.cursor() + cur.execute("PRAGMA foreign_keys=ON") + cur.execute("PRAGMA journal_mode=WAL") # removes reader/writer blocking + cur.execute("PRAGMA synchronous=NORMAL") + cur.execute("PRAGMA busy_timeout=5000") + cur.close() +``` + +### S13. Pattern inference detects 4 of 10 patterns and mis-learns 4 more at 0.95 confidence + +`backend/app/enricher/pattern.py:131` requires `len(p) >= 2` for every token, making the +`f.last` and `firstl` branches **unreachable**. A single alpha token is unconditionally +classified `"first"`. Verified: + +``` +rsharma@acme.in → got "first" (expected "flast") conf 0.95 ← WRONG, confident +rajeshsharma@acme.in → got "first" (expected "firstlast") conf 0.95 ← WRONG, confident +sharma@acme.in → got "first" (expected "last") conf 0.95 ← WRONG, confident +a.sharma@acme.in → got "first.last" (expected "f.last") conf 0.25 +``` +And `pattern.py:121-129` **discards all seed evidence** whenever the exec names don't match +the seed — because the name-based and shape-based branches are mutually exclusive instead of +unioned. Finally, the computed confidence (`:148-150`) is **not dominance-aware** (it +contradicts the repo's own design doc at `.research/lead-data.md:83`), and `email_finder.py:447` +**throws it away** (`permutations, _pat, _conf = ...`). + +`flast` / `firstlast` / `last` are disproportionately German/Indian-IT. + +### S14. Every contact inherits `executives[0]`'s name and title + +`backend/app/enricher/email_finder.py:510-514` runs **inside** the per-email loop: +```python +exec_match = execs[0] if execs else None +c_first = exec_match["name"].split()[0] if exec_match else (business.name.split()[0] if business.name else "Owner") +``` +So `info@acme.in` is stored with `first_name="Rajesh", title="Founder"` — simultaneously +`is_role_account=True` **and** named. With no exec, the fallback produces *"Hi Apex,"* for +"Apex Realty Solutions". `sender.py:413` then renders **"Hi Rajesh," to a shared mailbox.** + +### S15. Decision-maker extraction misses every real Indian SMB title + +`backend/app/enricher/email_finder.py:237-244` `EXEC_TITLES` misses **MD, Principal +(Principal Dentist — the dental norm), General Manager, Promoter, Chairman**. The regex +also manufactures garbage: `"read more - owner"` → `('read more', 'owner')`, +`"call now - partner"` → `('call now', 'partner')`. Those become `person_names` and +**consume the 4-slot candidate budget** (`read.more@domain`, `call.now@domain`). + +### S16. `is_verified=True` is asserted for addresses that were never SMTP-probed + +`email_finder.py:502` sets `is_verified = result.is_deliverable`, and `is_deliverable` +includes `ROLE_ACCOUNT` and `FREE_EMAIL` — both of which `classify_email` short-circuits +**before any network call**. So `info@acme.in` gets `is_verified=True`, a `verified_at` +timestamp, and the scorer's full **30 points** plus the "Verified Email" badge. The shipped +test at `tests/test_email_finder.py:78` actively asserts this behaviour. + +### S17. The spam linter misses every Indian cold-email cliché + +`backend/app/campaigns/spam_checker.py:11-46` — the 120-word high-risk list is US +consumer-spam vocabulary (`viagra`, `weight loss`, `mortgage rates`). Verified: *"Hope this +email finds you well / I am writing to introduce / Dear Sir/Madam / kindly revert back"* +scores **75 = "Good", `is_safe: True`**. Meanwhile ordinary words false-positive +(`score`, `promise`, `certified`, `win`). The README's "all-caps density, exclamation +ratios and hyperlink density" — **none of the three exist.** + +### S18. The agent cannot obtain a `contact_id`, so it can never target anyone + +`mcp_server.py:198-200` requires `contact_ids`, but **no tool output contains one** — +`search_leads` and `scrape` return business `id` only. In the live DB, `contact.id != +business_id` for **100 of 126 contacts**. An agent doing the only thing it can mails the +**wrong 79% of the database**. The entire "scrape → target → launch" loop is a dead end. + +### S19. `leadforge_scrape` returns the wrong leads + +`mcp_server.py:300-303` returns the **last N businesses in the whole database**, not the +ones this scrape created — while the header says "Scrape Results for: {query}". The +`_new_biz_ids` returned by `scrape_leads_and_save` is discarded at `:297`. + +### S20. No auth + wide-open CORS = any website can delete data and send cold email + +`main.py:113-123` with `config.py:34` `CORS_ORIGINS = "*"` **and** `allow_credentials=True` +— Starlette echoes the attacker's origin on preflight. Verified: `DELETE /api/v1/leads/999` +→ 200, `POST /campaigns/{id}/launch` → cold email sent from the user's Gmail. + +**Gain:** the only finding where an attacker needs zero credentials. Cheap fix that doesn't +break local DX: an optional `API_KEY` (unset = auth off) applied to every router **except** +`tracker` (the unsubscribe link and the open pixel must stay reachable by mail clients). + +### S21. SSRF via the Twenty-CRM export webhook + +`api/leads.py:340-344` POSTs to a user-supplied `webhook_url` with no scheme check, no +host allowlist, no private-IP block — and the payload contains **the entire lead database**. +Against a cloud host this reads instance metadata / IAM credentials. + +### S22. Soft bounce = immediate 30-day suppression, and there is no retry path + +`sender.py:486-491` suppresses on the **first** soft bounce; `bounces.py:130-132` +`should_retry()` is **defined and never called**. `bounces.py:101-102` classifies *any* +unrecognised error (connection refused, `SMTPServerDisconnected`) as soft. One transient +gateway blip removes a valid lead for a month. + +Worse: `bounces.py:105-127` `_HARD_RE` matches bare `\b550\b` / `\b553\b`, and +`smtplib` raises `SMTPSenderRefused` when **our own MAIL FROM** is rejected — a *sender* +problem classified as a *recipient* hard bounce → **one misconfigured sending domain +permanently suppresses the entire lead database.** + +### S23. `purge_expired` never fires + +`campaigns/suppression.py:134-138` compares `Suppression.expires_at < Suppression.created_at` +— always false, since `expires_at = created_at + 30d`. It should be `< func.now()`. It's +also never called from anywhere. + +### S24. `{{tokens}}` ship literally to prospects + +`sender.py:30-38` `interpolate_template` does a plain `str.replace` with **no fallback** — +an unknown token leaves the braces in. A typo'd `{{frist_name}}` reaches the prospect. +No HTML escaping either (`:440` injects `business.name` raw into `body_html`). And +`api/campaigns.py:288-308`'s preview uses a sample dict **missing** `{{ai_icebreaker}}`, +`{{compliment}}`, `{{pain_point}}`, `{{call_to_action}}` — so the preview renders +`{{pain_point}}` literally while the real send renders a sentence. + +### S25. Click tracking doesn't exist → the WhatsApp handoff can never fire + +`models/send.py:19` has a `clicked` column that **nothing ever writes**; `tracker.py` has no +`/click` route. The handoff rule at `engagement.py:23-27` requires ≥3 opens or ≥1 click, +and open tracking defaults **off** (`config.py:62`). `GET /campaigns/india/handoff/{id}` +returns **an empty list, permanently.** The README's flagship India feature is a no-op. + +### S26. The `AI icebreaker` is a template engine with a hardcoded Delhi default + +`enricher/ai_icebreaker.py:11-48` is a 9-key static dict + 3 string-format branches. No +LLM, no review text is ever collected (only `reviews_count: int`), no sentiment analysis, +and the `website` parameter is **never read**. `mcp_server.py:502` hardcodes +`city = arguments.get("city", "Delhi NCR")` — so a Kochi lead generated via MCP gets +*"…other dental leaders in Delhi NCR"*. + +--- + +## 3. Wave 2 — Product completeness (the Apollo surface we're missing) + +### 2a. The People table — a company table where the row is a business, not a person + +Apollo's core surface is *People*: one row per human, filterable by title and seniority. +`/leads` returns businesses with nested contacts, and **there is no contacts-list endpoint** +(verified: zero `@router.get` routes matching "contact"). `Contact.title` is stored and +shown in the drawer but is **not a column and not filterable**. + +> This is the single biggest structural gap. It needs: a `/api/v1/contacts` endpoint, a +> people table in the dashboard, and real decision-maker extraction (S14, S15) to be worth +> anything. + +### 2b. ICP filters: the parser silently drops constraints + +`qualifier/nl_icp.py:63-68` — the char class `[a-z0-9\- ]` includes a space, so it greedily +eats following words. Verified: + +``` +'dentists in Kothrud Pune with no website and verified emails' + → exclude_keywords: ["website and verified emails"] ← corrupt +'shops in Mumbai with 50 employees and revenue over 10cr' + → employee count + revenue: SILENTLY DROPPED +'exporters in Noida with Shopify and looking for a sales director' + → tech + title: SILENTLY DROPPED +``` +A user who types "50 employees, revenue over 10cr, Shopify" gets **every shop in Mumbai** +with no indication their constraints were ignored. This is the "the product lied to me" bug. + +**Fix (minimum):** add a `UNSUPPORTED` regex and return `unsupported_terms` + `warnings`, +surfaced in the wizard. **Fix (full):** a structured `ICPFilters` model — `industries`, +`cities`, `areas`, `pincodes`, `company_size_min/max`, `revenue_band`, `tech`, `titles`, +`seniority`, `has_gstin`, `city_tier`. + +### 2c. Source trust still needs outcome-based calibration + +The code now recognizes the live scrape source names, applies the trust modifier at full +precision, and records each source's own deduplicated ranked leads. Its running average is +weighted by the number of leads ranked, so duplicate results from a second source no longer +inherit the first source's score. + +The score modifier now starts from the documented priors and calibrates against the last +90 days of email-verifier results and completed SMTP outcomes, grouped by the business's +discovery source. A source's prior contributes 20 pseudo-observations; verification samples +count once and completed SMTP acceptances/hard bounces count twice, with each evidence +stream capped at 50 samples per calibration. This keeps sparse evidence from swinging a +source's score while allowing repeated hard bounces or clean SMTP acceptance to move it. +Recent transient SMTP failures are excluded because they measure sender health, not lead quality. The source router uses +learned trust as a bounded tie-breaker alongside observed discovery yield, and lead scores +display the same calibrated modifier they apply. + +Still open: verified phone outcomes and positive-reply/conversion outcomes. Email trust is +not a proxy for either, and this calibration does not claim to measure them. A `score_samples` +counter marks legacy blended averages so their first correctly attributed score replaces +the old value instead of inheriting it. + +### 2d. The score is cached at enrichment time and never recomputed + +`api/leads.py:134-142` reads `extra["lead_score"]` — computed once at enrichment. Verified: +a fresh ICP-scoped score of 25 COLD is displayed as **95 HOT** from the stale cache. Changing +an ICP has **zero** effect on existing leads — which makes the entire "Save Search" story a +query builder on top of a ranking that ignores the query. + +**The structural fix:** `lead_score` should be **ICP-relative and live**, not a global +cached contact-completeness number. Fixing this one thing collapses a large share of the +targeting findings. + +### 2e. Analytics is four flat counters + +No bounce *rate*, no unsubscribe rate, no positive-reply rate, no per-source, no +per-mailbox, no hour-of-day heatmap, no funnel, no cohort, no reply→meeting. Meanwhile +`Send.bounced` and `Send.clicked` are **never written** — so `engagement.py:55` always +reports `bounced: 0` even when the mailbox is bouncing. + +### 2f. A/B testing has no statistics and the wrong metric + +`api/campaigns.py:414-431` compares raw reply rates at n=10 with no significance test. +Cold-email reply rates are 1–3%, so one extra reply = 10pp and a coin flip declares a +winner. The metric counts OOO auto-replies and opt-outs as "replies" — and the reply +poller **drops the auto-reply headers** (`reply_poller.py:257` computes `auto` then +discards it at `:266`), so `Auto-Submitted` is dead code at runtime. An out-of-office +auto-responder can pick your subject line. + +Also: the winner is only finalised as a side effect of a `GET`, and `sender.py` reads +`step.variant_winner` but never writes it. + +### 2g. Missing Apollo surfaces with no model behind them + +Opportunities/deals pipeline, tasks, meetings/calendar, notes on a lead, unified +conversation inbox with threading/ownership, per-rep attribution, inbound webhooks, signed +outbound webhooks, per-record activity timeline, field-level provenance on every attribute, +and a searchable contacts list. + +### 2h. Dashboard gaps + +- **Global Cmd+K search is dead.** `TopBar.tsx:32` pushes `/leads?search=…` but **nothing + ever reads a query param** (`useSearchParams` → zero hits). Type "dentist", press Enter, + get all 75 leads. +- **`/docs` 404s.** `Sidebar.tsx:174` links to a route that doesn't exist. +- **15 native `alert()` dialogs**, no toast system — even though `@radix-ui/react-toast` is + already a dependency and never imported. **10 of 15 runtime deps are dead weight** + (react-query, zod, react-hook-form, radix-dialog/label/slot/icons). +- **Launch has no confirmation** (`SequenceDetail.tsx:109`) — the highest-consequence action + (sends real email) has the least protection, while lead/mailbox delete do get `confirm()`. +- **Mobile is structurally broken.** `Sidebar.tsx:94-97` is `w-60 shrink-0` with **zero + responsive prefixes**; on a 390px viewport the sidebar takes 240px of 390px. The e2e test + that "verifies" this asserts `document.scrollWidth - clientWidth <= 4` — but `body` is + `overflow-hidden`, so it **passes while the layout is unusable.** +- **No design system.** `tailwind.config.ts` defines semantic tokens + dark mode; **zero** + components use them and there are **zero `dark:` classes** in the app. `cn()` is never + imported. 175 occurrences of `text-[8px]`–`text-[11px]` (9px fails legibility floors). + **8 uses of `backdrop-blur-xs`** — an invalid Tailwind class that compiles to nothing. +- **`lib/api.ts` has no timeout, no retry, no `AbortController`** → stale-response races + (two in-flight `getLeads` can resolve out of order and overwrite each other). `enrichLead` + can hang for 30–120s with no abort. +- **CI ignores type errors.** `next.config.mjs` sets `ignoreBuildErrors` + `ignoreDuringBuilds` + → `tsc --noEmit` finds **3 live errors** CI never sees. Playwright is not in CI at all. +- **9 of 12 e2e tests can never fail the build** — they `findings.push(...)` instead of + asserting. The committed `dashboard/shots/findings.json` proves it was already failing. +- **The home page fabricates a KPI delta**: `page.tsx:180` renders "+30 discovered" when the + total is 30 — directly under a comment saying *"never fake demo numbers."* +- **No rupee pricing anywhere** — the entire product thesis ("Apollo but free for Indian + SMBs") is absent from the UI. +- **No GSTIN field, no Hindi/Hinglish UI** (no `t()`, no locale switcher, no RTL readiness). + +--- + +## 4. Wave 3 — The India moat (where we beat Apollo, not just match it) + +Apollo has **no Indian registry data, no WhatsApp routing, no STD-code resolution, no DLT +context**. These are the features that make an Apollo clone irrelevant. + +1. **GSTIN / Udyam (MSME) / CIN registry lookup** — a GST-scoped list is a genuine moat and + one public registry away. Currently no such column exists anywhere. +2. **STD-code → city resolution** — `sources/in_phone.py:37-131` already has a verified SDCA + table. `city_from_phone()` is **never called outside its own module.** This is a + done-and-buried asset: pincode clusters, city tiers, and per-city analytics for free. +3. **Mobile vs landline as a score signal** — `in_phone.is_mobile` is independent + information (currently wasted: the `whatsapp` signal is **100% collinear with `phone`**, + so 15 of 100 score points are free). `Contact.preferred_channel` is computed and then + **never read by the sender**. +4. **WhatsApp-first channel routing** — a lead with a mobile should route to WhatsApp + before email. Apollo has no equivalent. +5. **Real festival + IST timing windows** — currently decorative: `india_copy.py`'s + `localize_template()` is **never called by the sender**, so `{{festival_line}}` ships + **literally** to prospects. There is no send-window control at all — a launch at 3 a.m. + IST sends at 3 a.m. IST. +6. **Real multi-language copy** — Hinglish today is 6 static closer strings to copy-paste. + Marathi/Telugu/Bengali/Kannada/Tamil are absent. +7. **India's directory ecosystem** — JustDial, Sulekha, IndiaMART, InPhone, Practo, and + (for the wedge) IndiaMART's *product* directory for manufacturing/wholesale ICPs, which + is high-value and currently noise for service ICPs. + +--- + +## 5. README truth-check (credibility) + +> **Kept as the record of what was false.** Most of these have since been +> fixed, and the README now states the source coverage honestly — including +> which directories cannot be read and why. + +Roughly a third of the README's headline claims are **false against the code**. A reviewer +who runs the documented commands and finds them broken costs more credibility than any +feature gap. + +| README claim | Reality | +|---|---| +| "parses `APP_INITIALIZATION_STATE`" | **Zero occurrences in the repo.** The code does `json.loads(...)` on positional JSON. | +| "internal protobuf `pb=` transport" | It's not protobuf — it's positional JSON. No protobuf dep. | +| "Humanized deceleration & viewport jitter" | `random_delay` and `get_random_viewport` are **imported and never called**. No mouse curves. | +| "Unclaimed GBP Detection … canonical *Claim this business* anchor" | The HTTP path never sets `is_claimed`; the Playwright path does a **text match**. The scoring weight never fires. | +| "Playwright Stealth Fallback" | It's **primary** on `/scraper/start`, not fallback. Stylesheets are *not* blocked (only image/media/font). | +| "multi-source" | 3 of 4 sources return **0 leads** against live sites. | +| "learns each domain's actual format" | Learns 4 of 10 patterns; mis-learns 4 more at 0.95 confidence; **discards seeds** when exec names don't match. | +| "Confidence-Blended Inference" | Not dominance-aware; and the confidence is **discarded** by the caller. | +| "Algorithmic 1-on-1 AI Icebreakers (sentiment & pain points)" | Static 9-key dict + 3 format branches. No LLM, no review text, no sentiment. | +| "Source-Trust Scoring … *observed* reliability" | Hand-written constants. Never observed. Not applied to the main scrape path. | +| "Multi-Step Sequences & Bounce Guard" | Delay math exists; **no scheduler**. Requires manual re-click. | +| "Hinglish & Festival-Aware Copy" | `localize_template()` is **never called by the sender**. | +| "warm WhatsApp handoff" | Requires ≥1 click; **click tracking doesn't exist**. Permanently empty. | +| "80+ words, 0-100 score … all-caps density, exclamation ratios and hyperlink density" | The 3 named signals **don't exist**. Indian clichés score "Good". | +| "Native Model Context Protocol (MCP) SDK" | Hand-rolled `asyncio.StreamReader` loop. No `mcp` package. | +| "Twenty CRM & Webhook sync engine" | One endpoint, one unsigned POST, no events/retries/signing. | +| "server-side rendering" | Every route builds as `○ (Static)` — zero SSR. | +| "no built-in auth" | ✅ True — but it's the shipped Docker default, and CORS is `*` + credentials. | +| "196/196 passing" | ✅ True. But 9/12 e2e tests can't fail, and CI skips type-checking. | + +--- + +## 6. Suggested execution order + +**Day 1–2 — Unbreak it** (≈8h): B1 mailbox cap · B2 send gate · B5 `has_email` + pagination · +S20 auth+CORS · the one-line `generate_ai_icebreaker` import bug (500 on a shipped feature) · +S23 `purge_expired` predicate. + +**Day 3–4 — Make it fast** (≈8h): B4 `q=` split (3.4× coverage) · S6 HTTP-first scraping · +S1+S2+S3 enrichment concurrency + async verifier · S10 indexes + S12 SQLite pragmas · +S21 SSRF guard. + +**Day 5–6 — Make it true** (≈8h): B3 campaign scheduler + "waiting vs done" · +S22 bounce retry + sender-vs-recipient classification · S13 pattern inference rewrite · +S24 token fallback + preview parity · S18+S19 MCP contact-ids and scrape scoping. + +**Week 2 — Make it Apollo** (2 weeks): 2a People table + `/contacts` API · 2d live +ICP-relative scoring · 2b structured ICP filters + honest warnings · 2c real source-trust +feedback loop · S14/S15 decision-maker extraction · 2e analytics funnel · 2f A/B statistics · +S25 click tracking · S17 India-aware spam linter. +> **Shipped.** The People table (`/contacts` + `/people`), ICP-relative scoring, structured +> ICP criteria, decision-maker extraction, the analytics funnel, click tracking, A/B +> statistics and the India-aware spam linter are in. Source trust now learns from recent +> verification and hard-bounce outcomes. Still open from this week: phone validation and +> positive-reply/conversion feedback for source calibration. + +**Week 3 — Make it ours** (2 weeks): GSTIN/Udyam registry · WhatsApp-first channel routing +· real festival/IST send windows · multilingual copy · directory ecosystem expansion +(JustDial/Sulekha/IndiaMART fixes) · rupee-denominated cost-vs-Apollo UI · a real design +system + dark mode + mobile. +> **Core workflow shipped.** GSTIN and Udyam identifiers can be extracted from company +> sites; mobile leads can route to the WhatsApp handoff; festival copy, regional-language +> closers and 09:00–20:00 IST send windows are wired into the workflow. Sulekha returns live +> results. JustDial and IndiaMART still require a browser engine and report that limitation. +> Remaining: verified government-registry lookup for GSTIN/Udyam, direct WhatsApp messaging, +> broader localization beyond email closers, and additional accessible directory sources. + +**Week 4 — Harden** (1 week): job queue with retries/DLQ · audit log · Prometheus metrics · +Alembic migrations (the dep is already in `requirements.txt` with zero migrations) · +Postgres in CI · e2e tests that actually fail the build. +> **Partly shipped.** Celery workers and beat are available when enabled, with an in-process +> fallback; interrupted jobs are failed with their reason. The audit log is shipped, and +> the current Playwright suite lists 51 tests with route accessibility checks. Remaining: +> make durable queue operation the production default, add Prometheus metrics, add versioned +> Alembic migrations, and run Postgres in CI (schema sync remains additive-only DDL). + +--- + +## 7. Measured outcomes + +| Metric | Today | After Waves 0–1 | After Wave 2 | +|---|---|---|---| +| Unique leads per scrape query | 30 | **103** | 103 + multi-source | +| Enrichment throughput | 9s/lead serial | **~1.1s/lead** | + persisted pattern cache | +| Scrape wall clock | 8.9s (Playwright) | **2.4s** (HTTP) | — | +| Leads/day per IP | ~2,000 | **~10,000** | — | +| Working data sources | 1 of 4 | **4 of 4** | + registry sources | +| Hard-bounce rate | 25–30% | **<5%** | <3% | +| Mailbox capacity | 50 **lifetime** | **50/day, forever** | + warmup ramp effective | +| `rank_existing` @100k | 13.6s | **<100ms** | — | +| Lead-list query @500k | ~345ms | **<100ms** | + FTS/trigram | +| CSV import 5k rows | 15.2s | **1–2s** | + streaming export | +| Event loop stall | 10s | **0s** | — | +| Named decision-maker found | 8–15% | 20–30% | **40%+** (People table) | +| Pattern detection | 4/10 | **8/10** | + per-domain persistence | +| API p99 during enrichment | seconds | **milliseconds** | — | +| Sequences running autonomously | 0% | **100%** | + branching | +| ICP filters expressible | 3 | 3 | **20+ attributes** | +| Analytics dimensions | 4 counters | 4 counters | **funnel + cohorts + heatmap** | diff --git a/UX-AUDIT.md b/UX-AUDIT.md new file mode 100644 index 0000000..5cd2443 --- /dev/null +++ b/UX-AUDIT.md @@ -0,0 +1,263 @@ +# LeadForge — UX, Accessibility & Performance Audit + +> Measured, not assumed. Every number below comes from running axe-core, +> Playwright and the live API against the running app. Companion to +> `ROADMAP.md` (which covers the product roadmap); this document covers the +> *experience* of using the thing. + +--- + +## 1. How this was measured + +| Tool | What it measures | Where it lives | +|---|---|---| +| **axe-core 4.13** | WCAG 2.1 A/AA + best-practice violations | `dashboard/e2e/a11y.spec.ts` (now a CI gate) | +| **Playwright** | Layout shift, LCP/FCP, API-call count, asset 404s | `dashboard/e2e/a11y.spec.ts` | +| **Live API** | Endpoint latency and query count | profiled per route | + +The audit ran against a **production build** (`next build` + the standalone +server), not `next dev`, against a live API with real data. + +--- + +## 2. Accessibility — before and after + +### Baseline (before this pass) + +``` +[moderate] region nodes=50 routes=10/10 "All page content should be contained by landmarks" +[serious ] color-contrast nodes=53 routes=9/10 fg=#94a3b8 bg=#ffffff ratio=2.56 need=4.5:1 +[moderate] heading-order nodes=2 routes=2 +[critical] select-name nodes=1 routes=1 "Select element must have an accessible name" +``` + +**Total: 22 violation types / 106 nodes across ten routes.** + +**48 of the 53 contrast failures were one class**: `text-slate-400` +(`#94a3b8`) on white is **2.56:1** where WCAG AA needs **4.5:1**. Tailwind's +`slate-400` is a common "muted label" colour and it simply is not readable — +which matters more here than most products, because a lot of LeadForge +operators work on cheap laptop panels in bright Indian offices. + +**The `region` failures were structural**: the sidebar's brand block and +footer sat outside any landmark, and the TopBar was not a `
`, so a +screen reader had no way to jump between the page banner, the navigation and +the content. + +### What was fixed + +| Fix | Detail | +|---|---| +| Landmarks | `