██╗   ██╗ ███╗   ██╗ ██████╗  ███████╗ ██████╗  ████████╗  ██████╗  ██╗    ██╗
██║   ██║ ████╗  ██║ ██╔══██╗ ██╔════╝ ██╔══██╗ ╚══██╔══╝ ██╔═══██╗ ██║    ██║
██║   ██║ ██╔██╗ ██║ ██║  ██║ █████╗   ██████╔╝    ██║    ██║   ██║ ██║ █╗ ██║
██║   ██║ ██║╚██╗██║ ██║  ██║ ██╔══╝   ██╔══██╗    ██║    ██║   ██║ ██║███╗██║
╚██████╔╝ ██║ ╚████║ ██████╔╝ ███████╗ ██║  ██║    ██║    ╚██████╔╝ ╚███╔███╔╝
 ╚═════╝  ╚═╝  ╚═══╝ ╚═════╝  ╚══════╝ ╚═╝  ╚═╝    ╚═╝     ╚═════╝   ╚══╝╚══╝ 

Pulls the real signal out of a flood of alerts before it wakes anyone up.

🌐 Live Demo: undertow-dev.vercel.app

Undertow is an AI incident triage system. A GitHub webhook fires, the incident is deduplicated against open incidents with local embeddings, classified by an LLM with a provider fallback chain, and streamed live to a dashboard over SSE — where a human on-call engineer approves, edits, or rejects the machine's call. Every correction is logged as ground truth, and a nightly LLM-judged eval scores the classifier against those corrections and blocks CI on F1 drift.

It is human-in-the-loop by design, not autonomous end to end. The classifier proposes a severity, an owning team, and a draft response; a human decides. The corrections are the product, not a side effect — they are the ground truth the eval measures against.

🔍 How It Works

  GitHub issue event
         │
         ▼
  POST /api/webhook
         │
         ├─ HMAC-SHA256 verify (x-hub-signature-256, raw bytes) ──► 401 on mismatch
         │
         ├─ normalize ─► non-issue / ping events ──► 200 { ignored }
         │
         ├─ externalId unique check ──► redelivery ──► 200 { duplicateDelivery }
         │
         ▼
  embed(title + body)                    all-MiniLM-L6-v2, 384-dim, local, zero API cost
         │
         ▼
  findDuplicate(cosine ≥ 0.87) ──► HIT ──► Incident{status:DUPLICATE, duplicateOfId}
         │                                        │         (no LLM call, no job)
         │ MISS                                   └──► SSE broadcast
         ▼
  ┌─ transaction ─────────────────────────┐
  │  Incident{status:OPEN, severity:LOW*} │      *placeholder until classified
  │  Job{type:CLASSIFY, status:PENDING}   │
  └───────────────────────────────────────┘
         │
         ├──► SSE broadcast (feed shows it pending)
         ▼
  worker  ─ claimNext(): UPDATE ... WHERE id = (
              SELECT id ... ORDER BY createdAt
              LIMIT 1 FOR UPDATE SKIP LOCKED )   ◄── atomic claim, no double-processing
         │
         ▼
  classify()
         │
         ├─ Groq  llama-3.3-70b-versatile ──► ok ──┐
         │        │ any failure (429 / timeout /   │
         │        │ malformed JSON / other)        │
         │        ▼                                │
         └─ Cerebras  gpt-oss-120b ──────► ok ─────┤
                  │ both fail                      │
                  ▼                                │
            AllProvidersFailedError                │
            └─► fail(job): attempts+1, 30s backoff │
                └─► attempts ≥ 3 ──► FAILED (manual review)
                                                   │
         ┌─────────────────────────────────────────┘
         ▼
  zod-validated { severity, suggestedTeam, draftResponse }
         │
         ▼
  Incident updated ──► pg_notify('incident_events', id)
                              │
                              ▼
                       LISTEN (web process) ─► re-fetch row ─► broadcast
                              │
                              ▼
                    GET /api/incidents/stream  (event: incident)
                              │
                              ▼
                        dashboard feed
                              │
                              ▼
              human approve / edit / reject   ◄── ONCALL only
                              │
                              ▼
              POST /api/incidents/[id]/correct
                              │
                    ┌─ transaction ──────────────┐
                    │  Correction{field, from,to}│  ◄── ground truth
                    │  Incident updated          │
                    └────────────────────────────┘
                              │
                              ▼  (nightly, 03:00 UTC — GitHub Actions cron)
              POST /api/cron/eval  (x-cron-secret) ──► 202 { jobId }
                              │
                              ▼
              Job{type:EVAL} ──► worker ──► runEval()
                              │
                              ├─ re-run classify() over incidents WITH corrections
                              ├─ strict compare vs latest correction per field
                              └─ disagreement ──► Gemini 2.0 Flash judge
                                                  { reasonable, rationale }
                              │
                              ▼
              EvalRun{precision, recall, f1, sampleSize, driftFlag}
                              │
                              ▼
              drift = (rollingAvg − f1) / rollingAvg > 0.05
                      rollingAvg = last 5 runs, sampleSize ≥ 10 only
                              │
                              ▼
              Eval Gate (PRs touching backend/lib/llm/**)
              exit 1 on drift ──► merge blocked

What makes this a system rather than a demo is where the guarantees sit. Every LLM response crosses a zod schema before it touches the database, so a malformed generation is a caught error and a provider fallback, never a corrupt row. Job claiming is a single FOR UPDATE SKIP LOCKED statement rather than a read-then-write, so two concurrent workers can never process the same incident — and a job orphaned by a dead worker becomes reclaimable after 5 minutes instead of being lost. Retries are hard-capped at 3 attempts with a 30-second backoff, so a permanently failing job parks in FAILED for manual review instead of burning API budget in a hot loop.

✨ Features

🔀 Provider fallback Groq (llama-3.3-70b-versatile) is the primary classifier. Any failure — 429, timeout, malformed JSON — falls through to Cerebras (gpt-oss-120b) with the identical prompt. Both failing raises AllProvidersFailedError and flags the incident for manual review.	🧬 Zero-cost dedupe all-MiniLM-L6-v2 runs locally via transformers.js — no API call, no per-incident cost. A cosine similarity ≥ 0.87 against open incidents marks the incident DUPLICATE and skips the LLM entirely. Dedupe is the cheapest possible path.	📡 Live SSE feed pg_notify bridges the worker and web processes, so a classification completing in the worker reaches the browser without polling. Heartbeat every 25s; listeners unsubscribe on client abort so dead connections never accumulate.
✍️ Correction logging Every human override writes a Correction row — field, original value, corrected value, author — in the same transaction as the incident update. This is the ground-truth corpus the eval scores against.	⚖️ LLM-judged eval + CI gate A nightly job re-runs the live classifier against corrected incidents. Strict mismatches escalate to a Gemini 2.0 Flash judge. A relative F1 drop > 5% vs the rolling 5-run average sets driftFlag and blocks the merge.	🔐 Role-gated actions NextAuth (GitHub OAuth + credentials, JWT sessions) with ONCALL / VIEWER. The dashboard hides correction controls from viewers as a hint; the server independently 403s them. The UI is never the security boundary.

🛠️ Tech Stack

Layer	Technology	Purpose
Frontend	Next.js 14 (App Router)	Landing, auth, dashboard, eval metrics
Backend	Next.js 14 (App Router), separate app on :3001	Route handlers, SSE, webhook
Auth	NextAuth v4 — GitHub OAuth + Credentials, JWT strategy	Sign-in, session, ONCALL/VIEWER roles
Database	Neon Postgres + Prisma 7 (@prisma/adapter-pg)	Incidents, corrections, users, eval runs
Job queue	The same Postgres — Job table, FOR UPDATE SKIP LOCKED	Async classify + eval. No Redis, no BullMQ
Live updates	Server-Sent Events + Postgres LISTEN/NOTIFY	Cross-process push to the dashboard
Classifier	Groq llama-3.3-70b-versatile (groq-sdk)	Primary severity/team/draft classification
Fallback	Cerebras gpt-oss-120b (raw fetch, OpenAI-compatible)	Retry once on any Groq failure
Judge	Gemini 2.0 Flash (@google/generative-ai)	Eval adjudication + long-form drafts
Embeddings	transformers.js Xenova/all-MiniLM-L6-v2 (384-dim)	Local dedupe, no API cost
Validation	zod 4	LLM output + API request/response boundaries
State	Zustand	Incident store (80ms SSE ingest buffer), session store
UI	Tailwind CSS 3, Framer Motion 12, Recharts 3	Design tokens, motion, drift charts
CI	GitHub Actions	Eval gate on PRs, nightly eval cron
Hosting	Vercel (frontend) + Render (backend, worker)	See Deployment below

Stack notes, verified against package.json:

• Cerebras is not Llama 3.1 70B. A code comment in backend/lib/llm/cerebras.ts records that Llama 70B is no longer served on this account; the model is gpt-oss-120b. There is also no Cerebras SDK dependency — the client is a raw fetch against their OpenAI-compatible endpoint.
• GEMINI_MODEL overrides the judge model when gemini-2.0-flash has no quota on an account.

🚀 Deployment

Frontend on Vercel, backend + worker on Render (free tier). The worker binds process.env.PORT with a minimal health server (backend/scripts/worker.ts) purely so Render's free plan — which only offers Web Services, not Background Workers — will accept it as a healthy deploy; the port is otherwise unused.

• Free-tier services sleep after 15 min idle. Both the backend and worker are kept warm with external HTTP pings (UptimeRobot, 5 min interval) against /api/incidents and the worker's health root respectively. Without this, incidents queue up unclassified until something wakes the worker.
• NEXTAUTH_URL on the backend must be the deployed frontend origin, not localhost — a blank or missing value crashes every NextAuth route (next-auth's internal new URL() call throws on an empty string).
• GitHub OAuth App callback URL points at the deployed frontend (https://<frontend>/api/auth/callback/github); switching it from localhost breaks local OAuth sign-in (credentials login is unaffected).
• EVAL_URL repository variable must point at the deployed backend for nightly-eval.yml to reach it.

📁 Project Structure

Undertow/
├── .github/workflows/
│   ├── eval-gate.yml           # Blocks PRs touching backend/lib/llm/** on F1 drift
│   └── nightly-eval.yml        # 03:00 UTC cron -> POST /api/cron/eval -> poll -> report
│
├── frontend/                   # Next.js app, :3000 — proxies /api/* to the backend
│   ├── app/
│   │   ├── admin/metrics/      # Eval + drift dashboard
│   │   ├── auth/               # Sign-in / register
│   │   ├── dashboard/          # Live incident feed
│   │   ├── error.tsx           # Shared route error boundary
│   │   ├── fonts/
│   │   ├── layout.tsx
│   │   └── page.tsx            # Landing
│   ├── components/
│   │   ├── motion/             # Shared Framer variants (+ reduced-motion twins)
│   │   └── ui/                 # Badge, Button, Card, Input, Modal
│   ├── features/
│   │   ├── auth/               # AuthTabs, LoginForm, RegisterForm, useAuthForm
│   │   ├── dashboard/          # IncidentFeed/Card, CorrectionModal, WaterlineHeader,
│   │   │                       #   SeverityBadge, DuplicateSinkIndicator, ConnectionStatus,
│   │   │                       #   useIncidentStream, useCorrection
│   │   ├── eval-metrics/       # DriftChart, EvalRunTable, ThresholdBreachBanner,
│   │   │                       #   drift.ts (mirrors backend drift rule), useEvalHistory
│   │   └── landing/            # Hero, PulseLine
│   ├── hooks/useReducedMotion.ts
│   ├── lib/
│   │   ├── api/                # Typed fetch client, zod schemas, incidents, evalRuns, poll
│   │   ├── sse/                # createIncidentStream (state machine + polling fallback)
│   │   └── utils/              # cn, callback-url
│   ├── store/                  # incidentStore (SSE buffer), sessionStore, ui
│   ├── styles/tokens.css       # Single source of truth for colour
│   ├── types/                  # incident, evalRun, job, api, next-auth
│   ├── middleware.ts           # Route gating via getToken
│   ├── next.config.mjs         # /api/* rewrite -> BACKEND_URL (keeps cookies same-origin)
│   └── tailwind.config.ts
│
└── backend/                    # Next.js app, :3001 + the worker process
    ├── app/api/
    │   ├── auth/[...nextauth]/ # NextAuth handler
    │   ├── cron/eval/          # POST enqueue (202) + GET [jobId] status
    │   ├── eval-runs/          # GET paginated EvalRun history (session-gated)
    │   ├── incidents/          # GET list, [id]/correct, stream (SSE)
    │   ├── register/           # POST create account
    │   └── webhook/            # POST GitHub ingestion
    ├── lib/
    │   ├── auth/               # rbac (requireOncall), password (bcrypt)
    │   ├── eval/               # runner (drift rule), metrics
    │   ├── jobs/               # queue (SKIP LOCKED claim), worker, process, types
    │   ├── llm/                # router, groq, cerebras, gemini, parse, prompts/
    │   ├── sse/                # bus (LISTEN/NOTIFY), broadcaster
    │   ├── webhook/            # verify (HMAC), normalize
    │   ├── dedupe.ts           # cosine similarity, 0.87 threshold
    │   ├── embeddings.ts       # all-MiniLM-L6-v2 singleton
    │   └── env.ts              # Fail-fast env validation
    ├── prisma/
    │   ├── migrations/         # 4 migrations
    │   └── schema.prisma       # User, Incident, Correction, Job, EvalRun, Account, Session
    └── scripts/
        ├── worker.ts           # Long-running job processor (--drain for one pass)
        ├── eval-runner.ts      # CLI eval (--fail-on-drift --min-sample)
        └── seed-historical.ts  # Seed incidents + corrections

There is no root package.json — frontend/ and backend/ are independent npm projects with their own lockfiles, installed and run separately.

⚙️ Getting Started

Prerequisites

• Node.js — no engines field is set in either package.json. CI runs Node 22 (.github/workflows/eval-gate.yml), which is the only version this project is verified against.
• A Postgres database — the project targets Neon (serverless Postgres).
• API keys — Groq, Cerebras, Google AI Studio (Gemini).
• A GitHub OAuth App — for the OAuth sign-in path.
• A GitHub webhook secret — for incident ingestion.

1. Clone

git clone <repo-url>
cd Undertow

2. Backend

cd backend
npm install

Create backend/.env from backend/.env.example:

DATABASE_URL=            # Neon Postgres connection string
GROQ_API_KEY=
CEREBRAS_API_KEY=
GEMINI_API_KEY=
GEMINI_MODEL=            # optional; overrides gemini-2.0-flash when it has no quota
GITHUB_WEBHOOK_SECRET=
CRON_SHARED_SECRET=
NEXTAUTH_URL=            # the PUBLIC frontend origin — the browser reaches the API
                         # through the frontend's /api/* proxy rewrite
NEXTAUTH_SECRET=         # must match frontend/.env exactly
GITHUB_ID=
GITHUB_SECRET=

Apply migrations and generate the client:

npx prisma migrate deploy
npx prisma generate
npm run seed:historical   # optional — seeds incidents + corrections

3. Frontend

cd ../frontend
npm install

Create frontend/.env from frontend/.env.example:

BACKEND_URL=                          # e.g. http://localhost:3001 — rewrite target
NEXTAUTH_SECRET=                      # byte-identical to backend/.env
NEXT_PUBLIC_APP_URL=
NEXT_PUBLIC_API_BASE_URL=             # same-origin by default; only if API is split
NEXT_PUBLIC_ENABLE_SSE_RECONNECT=true
NEXT_PUBLIC_SSE_RECONNECT_MS=3000
NEXT_PUBLIC_POLL_FALLBACK_MS=         # polling interval when SSE drops
NEXT_PUBLIC_REDUCED_MOTION_DEFAULT=false

NEXTAUTH_SECRET must be byte-identical across both files: the backend issues the session JWT, and the frontend's middleware.ts decrypts it with getToken().

4. Run

No root script runs these together — there is no root package.json. Use three terminal panes:

# Pane 1 — backend API
cd backend && npm run dev        # :3001

# Pane 2 — worker (nothing gets classified without this)
cd backend && npm run worker

# Pane 3 — frontend
cd frontend && npm run dev       # :3000

Open http://localhost:3000. The frontend proxies /api/* to BACKEND_URL, so everything is same-origin from the browser's point of view — which is what lets the NextAuth session cookie work across the split.

The worker is a separate process, not part of either Next app. Without it, incidents are ingested and appear in the feed but stay unclassified (severity: LOW, suggestedTeam: null).

5. Grant yourself ONCALL

New accounts default to VIEWER, which cannot submit corrections. There is no self-serve promotion — update the row directly:

UPDATE "User" SET role = 'ONCALL' WHERE email = 'you@example.com';

🔌 API

All routes live on the backend (:3001) and are reached through the frontend's /api/* rewrite in normal use.

Method	Path	Auth	Purpose
POST	/api/webhook	HMAC-SHA256 (x-hub-signature-256)	GitHub issue ingestion → dedupe → enqueue
GET	/api/incidents	None	List incidents. status, severity, limit (1–100, default 20), offset
GET	/api/incidents/stream	None	SSE live feed
POST	/api/incidents/[id]/correct	Session + ONCALL	Log a correction and update the incident
POST	/api/register	None	Create a credentials account (defaults to VIEWER)
GET/POST	/api/auth/[...nextauth]	—	NextAuth (GitHub OAuth + credentials)
POST	/api/cron/eval	x-cron-secret	Enqueue an EVAL job → 202 { jobId }
GET	/api/cron/eval/[jobId]	x-cron-secret	Poll eval job status + resulting EvalRun
GET	/api/eval-runs	Session (VIEWER+)	Paginated EvalRun history, newest first. limit (1–100, default 50), opaque cursor → { runs, nextCursor }

POST /api/incidents/[id]/correct accepts { field: "severity" | "team" | "duplicate", correctedValue: string } and returns 400 on an invalid severity or a self-referential duplicate, 403 for a VIEWER, 404 for an unknown incident.

SSE event shape

GET /api/incidents/stream opens with retry: 3000 and a : connected comment, then sends a : ping comment every 25s. Incident events carry the full row minus embedding:

event: incident
data: {"id":"clx…","source":"github","externalId":"I_kw…","title":"…","body":"…",
       "severity":"HIGH","status":"OPEN","suggestedTeam":"Platform",
       "draftResponse":"…","duplicateOfId":null,"createdAt":"2026-07-17T09:49:00.000Z"}

severity is non-nullable in the schema, so a not-yet-classified incident arrives as severity: "LOW" with suggestedTeam: null — the null team, not the severity, is the honest signal that classification is still pending.

⚠️ Known Limitations

• /admin/metrics re-derives the drift rule client-side. GET /api/eval-runs now backs the page with real history, so the drift chart draws its rolling average and gate line from actual runs and the ?jobId= path is only a fallback for when the route is unavailable. But EvalRun persists just the boolean driftFlag — not the rollingAvg or threshold it was judged against — so frontend/features/eval-metrics/drift.ts still re-implements DRIFT_THRESHOLD, ROLLING_WINDOW, and MIN_TRUSTWORTHY_SAMPLE to reconstruct the gate line, duplicating the rule across two npm projects with no shared import path.
• The incident read paths are unauthenticated. Any client that can reach the backend can read every incident via GET /api/incidents and GET /api/incidents/stream; only corrections are role-gated. Roles themselves are not self-serve either — every account defaults to VIEWER and ONCALL is granted by a manual UPDATE against the database.

🔮 Future Improvements

• Self-serve ONCALL promotion — an admin-gated role management surface, replacing manual SQL.
• Persist the classifying provider on Incident (e.g. classifiedBy) so ProviderTicker can show real Groq/Cerebras attribution instead of rendering null.
• Persist rollingAvg and threshold on EvalRun so the frontend can draw the gate line from recorded fact instead of re-deriving the rule and risking drift between the two copies.
• pgvector for dedupe — replace the in-memory O(n) cosine scan with an indexed nearest- neighbour query.
• Auth on the incident read paths — close the unauthenticated list/stream endpoints.