README rewrite, eval harness, server-side citation guard

eval/README.md

@@ -0,0 +1,62 @@
+# InfraWatch evaluation harness
+
+A small, reproducible harness that measures the parts of InfraWatch that can be measured **without paid API calls or labeled imagery**, and provides a ready-to-run harness for the part that cannot (vision accuracy).
+
+## Run
+
+```bash
+cd server && npm install && cd ..     # the failover test imports the real server code
+cd eval && npm test                   # or from repo root: node --test "eval/tests/*.test.js"
+```
+
+No API keys, network, or database are needed. Each test writes its numbers to `eval/results/*.json`. Last run: 14 tests, 14 pass (Node 22).
+
+## What is measured, and what it means
+
+### 1. SLA tier routing (`tests/sla.test.js`, real code)
+The confidence-to-SLA mapping used by the crisis feed (`server/routes/violations.js`) was extracted verbatim into `server/services/sla.js` (behaviour-preserving) so it can be tested.
+
+| Check | Result |
+|---|---|
+| 9 boundary confidences (100, 95, 90, 89.99, 85, 80, 79.99, 50, 0) map to 4h / 12h / 24h as documented | 9/9 |
+| Sweep 0-100 in steps of 0.5 (201 points): higher confidence never gets a longer SLA | pass |
+| Status thresholds: ON_TIME up to 80% of SLA, AT_RISK up to 100%, BREACHED beyond | pass |
+
+Observed edge behaviour (documented, not judged): `null`, `undefined`, `NaN`, and a 0-1 fraction such as `0.95` all fall into the 24h tier, because the 0-100 scale is implicit. Also note the crisis feed's candidate filter uses `confidence >= 85` while the 4h tier starts at 90 (`violations.js`), so 85-89 cases are listed but get the 12h SLA.
+
+### 2. AI provider failover (`tests/failover.test.js`, real code, stubbed network)
+Runs the real dispatcher in `server/services/visionAI.js` with dummy keys and a stubbed `fetch` (the Gemini SDK and the Groq call both go through `fetch`). 7 scenarios:
+
+| Scenario | Outcome observed |
+|---|---|
+| Gemini healthy | Gemini answers, Groq called 0 times |
+| Gemini 429 | falls over, Groq answers (1 call) |
+| Gemini 503 | falls over, Groq answers |
+| Gemini hangs past timeout (300 ms in test; 30 s default) | falls over, Groq answers |
+| Gemini 400 (non-retryable) | no failover, error surfaced |
+| Both fail with 429 | error `Both AI providers unavailable...` |
+| 429 with no Groq key configured | no failover, error surfaced |
+
+Finding: after a failover, the parsed result's `_provider`/`_model` metadata still reports `gemini` (it is read from config in `parseVisionResponse`, not from the provider that actually answered). The dispatcher itself returns the correct provider; the metadata attached to the parsed payload does not. Scope: only the `callAI` path in `visionAI.js` fails over. `generateAINotice` in `services/gemini.js` (the `/api/notices/generate-ai` route) uses a single provider chosen by `AI_PROVIDER` (default `groq` in that file) with no failover.
+
+### 3. Citation guardrail (`tests/citations.test.js`, `tests/guardrail_enforcement.test.js`)
+Two different questions, deliberately separated:
+
+**a) Does the app catch fabricated citations in code?** Originally the guardrail was a **prompt instruction only**, and with a stubbed provider returning 20 notices that cite fabricated provisions the real function returned the fabricated text unchanged in **20 of 20** cases (catch rate **0%**). That prompted `server/services/citationGuard.js`, now called from `visionAI.js` and `gemini.js`. Re-running the same test, the app removes the fabricated citation in **20 of 20** cases (catch rate **100%**).
+
+**b) What would a deterministic post-check against the same allowlist catch?** `server/services/citationGuard.js` is the validator, now enforced in the app. Allowlist = the sections named in the app's own prompt (KMC Act 1976 s.308/321/321A/322; Bye-laws 2003 2.2/2.5/3.0/4.6; KTCP Act 1961 s.14/15/76-FF). On 20 hand-written fabricated snippets (non-existent sections, wrong-Act sections, IPC/Constitution/case-law citations, a fake statute, a real-looking fallback phrase followed by a fabricated section) and 20 valid snippets: **20/20 caught (100%), 0/20 false positives**.
+
+Caveats: the fixtures are synthetic and written by the same author as the validator, so 100% is a sanity check, not a generalisation claim. The allowlist comes from the app's prompt and has **not** been verified against the statutes. Neither result says anything about how often a real LLM hallucinates a citation; that needs a live run (see below).
+
+### 4. Vision detection accuracy: PENDING
+Not measured. It needs human-labeled satellite tiles and a Gemini or Groq key. No accuracy numbers are reported anywhere in this repo.
+
+To run once you have data:
+1. Copy `vision/labels.template.csv` to `vision/labels.csv`; add images under `vision/images/`; set `has_violation` to 1/0 from human ground truth (aim for a balanced set, e.g. 50/50).
+2. `export GEMINI_API_KEY=...` (optionally `GROQ_API_KEY`), then `node eval/vision/run.js --limit 100`.
+3. Output `eval/results/vision.json`: confusion matrix, precision, recall, F1, false-positive rate, per-image predictions.
+
+The harness calls `analyzeSingleImage` directly, bypassing the seeded demo detections in `routes/vision.js`. Expect a false-positive bias worth measuring: that prompt tells the model to flag at least one suspicious area whenever buildings are visible. The metric code itself is unit-tested on a toy example (`tests/metrics.test.js`), which is not a detection result.
+
+## Not covered
+Auth/RBAC, notice generation quality, penalty formula correctness, officer assignment, and the client. The README's "round-robin" officer assignment claim could not be found in server code (see the main README).

eval/fixtures/citations.json

@@ -0,0 +1,47 @@
+{
+  "note": "Hand-written synthetic notice snippets. 'valid' = cites only allowlisted provisions (the allowlist is the app's own prompt list, not independently verified against the Acts). 'fabricated' = cites a provision outside the allowlist or an outside authority. Fixtures test the checker, NOT the LLM.",
+  "valid": [
+    "Karnataka Municipal Corporations Act, 1976 - Section 321",
+    "Karnataka Municipal Corporations Act, 1976 - Section 308",
+    "Karnataka Municipal Corporations Act, 1976 - Section 322",
+    "Karnataka Municipal Corporations Act, 1976 - Section 321A",
+    "BBMP Building Bye-laws, 2003 - Bye-law 2.2",
+    "BBMP Building Bye-laws, 2003 - Bye-law 2.5",
+    "BBMP Building Bye-laws, 2003 - Bye-law 3.0",
+    "BBMP Building Bye-laws, 2003 - Bye-law 4.6",
+    "Karnataka Town and Country Planning Act, 1961 - Section 14",
+    "Karnataka Town and Country Planning Act, 1961 - Section 15",
+    "Karnataka Town and Country Planning Act, 1961 - Section 76-FF",
+    "LEGAL PROVISIONS CITED: Section 308 of the Karnataka Municipal Corporations Act, 1976; Bye-law 2.5 of the BBMP Building Bye-laws, 2003",
+    "CONSEQUENCES OF NON-COMPLIANCE: demolition under Sec 321 of the BBMP Act; prosecution under Section 76-FF of the Karnataka Town and Country Planning Act, 1961",
+    "You are directed to comply. Refer to applicable BBMP Building Bye-laws, 2003",
+    "LEGAL PROVISIONS CITED: Refer to applicable BBMP Building Bye-laws, 2003",
+    "NATURE OF VIOLATION: A fourth floor was added without sanction. No statute is cited in this paragraph.",
+    "Stop-work notice under Sec. 322 of the Karnataka Municipal Corporations Act, 1976",
+    "Penalty under BBMP Act Section 321",
+    "Compounding under Karnataka Municipal Corporations Act, 1976 Section 321A where permissible",
+    "Setback per BBMP Building Bye-laws 2003, Bye-law 2.2; FAR per Bye-law 2.5"
+  ],
+  "fabricated": [
+    "Karnataka Municipal Corporations Act, 1976 - Section 999",
+    "Karnataka Municipal Corporations Act, 1976 - Section 412",
+    "Karnataka Municipal Corporations Act, 1976 - Section 320",
+    "Karnataka Municipal Corporations Act, 1976 - Section 323",
+    "Karnataka Municipal Corporations Act, 1976 - Section 308B",
+    "Karnataka Municipal Corporations Act, 1976 - Section 76-FF",
+    "BBMP Building Bye-laws, 2003 - Bye-law 7.9",
+    "BBMP Building Bye-laws, 2003 - Bye-law 2.3",
+    "BBMP Building Bye-laws, 2003 - Bye-law 11.4",
+    "Karnataka Town and Country Planning Act, 1961 - Section 99",
+    "Karnataka Town and Country Planning Act, 1961 - Section 76-FG",
+    "Karnataka Town and Country Planning Act, 1961 - Section 321",
+    "LEGAL PROVISIONS CITED: Section 505 of the Karnataka Municipal Corporations Act, 1976; Bye-law 2.5 of the BBMP Building Bye-laws, 2003",
+    "CONSEQUENCES OF NON-COMPLIANCE: demolition under Sec 321 of the BBMP Act; prosecution under Section 188 of the Indian Penal Code",
+    "Penalty under Section 44 of the Karnataka Building Regulation Act",
+    "As held by the Supreme Court in State of Karnataka v. Sharma, AIR 2011 SC 4021, demolition is mandatory",
+    "Article 300A of the Constitution of India entitles the corporation to act",
+    "Stop-work notice under Sec. 322D of the Karnataka Municipal Corporations Act, 1976",
+    "Refer to applicable BBMP Building Bye-laws, 2003; also Section 512 of the KMC Act, 1976",
+    "Under Section 17 of the Karnataka Municipal Corporations Act, 1976 the owner must pay a fine of Rs 5 lakh"
+  ]
+}

eval/lib/metrics.js

@@ -0,0 +1,21 @@
+/** Binary detection metrics. rows: [{ truth: boolean, pred: boolean }] */
+export function confusion(rows) {
+  const c = { tp: 0, fp: 0, tn: 0, fn: 0 };
+  for (const { truth, pred } of rows) {
+    if (truth && pred) c.tp++;
+    else if (!truth && pred) c.fp++;
+    else if (!truth && !pred) c.tn++;
+    else c.fn++;
+  }
+  return c;
+}
+
+const div = (a, b) => (b === 0 ? null : a / b);
+
+export function summarize(rows) {
+  const c = confusion(rows);
+  const precision = div(c.tp, c.tp + c.fp);
+  const recall = div(c.tp, c.tp + c.fn);
+  const f1 = precision === null || recall === null || precision + recall === 0 ? null : (2 * precision * recall) / (precision + recall);
+  return { n: rows.length, ...c, precision, recall, f1, false_positive_rate: div(c.fp, c.fp + c.tn), accuracy: div(c.tp + c.tn, rows.length) };
+}

eval/lib/results.js

@@ -0,0 +1,11 @@
+import fs from 'node:fs';
+import path from 'node:path';
+import { fileURLToPath } from 'node:url';
+
+const dir = path.join(path.dirname(fileURLToPath(import.meta.url)), '..', 'results');
+
+export function record(name, data) {
+  fs.mkdirSync(dir, { recursive: true });
+  fs.writeFileSync(path.join(dir, `${name}.json`), JSON.stringify(data, null, 2) + '\n');
+  console.log(`# ${name}: ${JSON.stringify(data)}`);
+}

eval/package.json

@@ -0,0 +1,9 @@
+{
+  "name": "infrawatch-eval",
+  "private": true,
+  "type": "module",
+  "scripts": {
+    "test": "node --test \"tests/*.test.js\"",
+    "vision": "node vision/run.js"
+  }
+}

eval/results/app_guardrail_enforcement.json

@@ -0,0 +1,6 @@
+{
+  "fabricated_n": 20,
+  "app_returned_fabricated_text_unchanged": 0,
+  "app_side_catch_rate": 1,
+  "still_flagged_by_validator_after_guard": 0
+}

eval/results/app_guardrail_passthrough.json

@@ -0,0 +1,6 @@
+{
+  "fabricated_n": 20,
+  "app_returned_fabricated_text_unchanged": 20,
+  "app_side_catch_rate": 0,
+  "would_be_flagged_by_reference_validator": 20
+}

eval/results/citation_validator.json

@@ -0,0 +1,10 @@
+{
+  "fabricated_n": 20,
+  "caught": 20,
+  "catch_rate": 1,
+  "valid_n": 20,
+  "false_positives": 0,
+  "false_positive_rate": 0,
+  "missed": [],
+  "false_positive_examples": []
+}

eval/results/failover.json

@@ -0,0 +1,67 @@
+{
+  "scenarios": [
+    {
+      "name": "primary_ok",
+      "ok": true,
+      "provider_reported": "gemini",
+      "calls": {
+        "gemini": 1,
+        "groq": 0
+      }
+    },
+    {
+      "name": "gemini_429",
+      "ok": true,
+      "provider_reported": "gemini",
+      "calls": {
+        "gemini": 1,
+        "groq": 1
+      }
+    },
+    {
+      "name": "gemini_503",
+      "ok": true,
+      "provider_reported": "gemini",
+      "calls": {
+        "gemini": 1,
+        "groq": 1
+      }
+    },
+    {
+      "name": "gemini_timeout",
+      "ok": true,
+      "provider_reported": "gemini",
+      "calls": {
+        "gemini": 1,
+        "groq": 1
+      }
+    },
+    {
+      "name": "gemini_400",
+      "ok": false,
+      "error": "[GoogleGenerativeAI Error]: Error fetching from https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-flash:generateContent: [400 ] HTTP 400",
+      "calls": {
+        "gemini": 1,
+        "groq": 0
+      }
+    },
+    {
+      "name": "both_fail",
+      "ok": false,
+      "error": "Both AI providers unavailable. gemini: [GoogleGenerativeAI Error]: Error fetching from https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-flash:",
+      "calls": {
+        "gemini": 1,
+        "groq": 1
+      }
+    },
+    {
+      "name": "no_secondary_key",
+      "ok": false,
+      "error": "[GoogleGenerativeAI Error]: Error fetching from https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-flash:generateContent: [429 ] HTTP 429",
+      "calls": {
+        "gemini": 1,
+        "groq": 0
+      }
+    }
+  ]
+}

eval/results/sla_tiers.json

@@ -0,0 +1,5 @@
+{
+  "boundary_cases": 9,
+  "sweep_points": 201,
+  "all_pass": true
+}

eval/tests/citations.test.js

@@ -0,0 +1,24 @@
+import test from 'node:test';
+import assert from 'node:assert/strict';
+import fs from 'node:fs';
+import { validateCitations } from '../../server/services/citationGuard.js';
+import { record } from '../lib/results.js';
+
+const fx = JSON.parse(fs.readFileSync(new URL('../fixtures/citations.json', import.meta.url)));
+
+test('reference validator: catch rate on fabricated, false-positive rate on valid', () => {
+  const missed = fx.fabricated.filter((t) => validateCitations(t).ok);
+  const falsePos = fx.valid.filter((t) => !validateCitations(t).ok);
+  record('citation_validator', {
+    fabricated_n: fx.fabricated.length,
+    caught: fx.fabricated.length - missed.length,
+    catch_rate: (fx.fabricated.length - missed.length) / fx.fabricated.length,
+    valid_n: fx.valid.length,
+    false_positives: falsePos.length,
+    false_positive_rate: falsePos.length / fx.valid.length,
+    missed,
+    false_positive_examples: falsePos,
+  });
+  assert.equal(missed.length, 0, `missed: ${JSON.stringify(missed)}`);
+  assert.equal(falsePos.length, 0, `false positives: ${JSON.stringify(falsePos)}`);
+});

eval/tests/failover.test.js

@@ -0,0 +1,99 @@
+/**
+ * Failover eval: runs the REAL dispatcher (server/services/visionAI.js callAI, reached via the
+ * exported analyzeSingleImage) with dummy API keys and a stubbed global fetch. No network, no cost.
+ * The stub stands in for both the Gemini SDK (which uses fetch) and Groq (raw fetch).
+ */
+import test from 'node:test';
+import assert from 'node:assert/strict';
+import { record } from '../lib/results.js';
+
+process.env.AI_TIMEOUT_MS = '300';
+process.env.GEMINI_API_KEY = 'dummy-gemini';
+process.env.GROQ_API_KEY = 'dummy-groq';
+process.env.AI_PROVIDER = 'gemini';
+const { analyzeSingleImage } = await import('../../server/services/visionAI.js');
+
+const GOOD = JSON.stringify({ suspiciousAreasDetected: true, totalAreas: 1, detections: [], overallConfidence: 80 });
+const realFetch = globalThis.fetch;
+let calls;
+
+function stub({ gemini, groq }) {
+  calls = { gemini: 0, groq: 0 };
+  globalThis.fetch = async (url) => {
+    const u = String(url);
+    const which = u.includes('generativelanguage') ? 'gemini' : u.includes('groq.com') ? 'groq' : null;
+    if (!which) throw new Error('unexpected fetch ' + u);
+    calls[which]++;
+    const beh = which === 'gemini' ? gemini : groq;
+    if (beh === 'hang') return new Promise(() => {});
+    if (typeof beh === 'number' && beh !== 200) {
+      return new Response(JSON.stringify({ error: { message: `HTTP ${beh}` } }), { status: beh });
+    }
+    const body = which === 'gemini'
+      ? { candidates: [{ content: { parts: [{ text: GOOD }], role: 'model' }, finishReason: 'STOP' }] }
+      : { choices: [{ message: { content: GOOD } }] };
+    return new Response(JSON.stringify(body), { status: 200, headers: { 'content-type': 'application/json' } });
+  };
+}
+
+const scenarios = [];
+async function scenario(name, cfg) {
+  stub(cfg);
+  let outcome;
+  try {
+    const r = await analyzeSingleImage({ image: null, locationContext: {} });
+    outcome = { ok: true, provider_reported: r._provider, calls: { ...calls } };
+  } catch (e) {
+    outcome = { ok: false, error: String(e.message).slice(0, 160), calls: { ...calls } };
+  }
+  scenarios.push({ name, ...outcome });
+  return outcome;
+}
+
+test.after(() => {
+  globalThis.fetch = realFetch;
+  record('failover', { scenarios });
+});
+
+test('primary healthy: Gemini answers, Groq never called', async () => {
+  const o = await scenario('primary_ok', { gemini: 200, groq: 200 });
+  assert.ok(o.ok, JSON.stringify(o));
+  assert.equal(o.calls.groq, 0);
+  assert.equal(o.calls.gemini, 1);
+});
+test('Gemini 429 -> Groq answers', async () => {
+  const o = await scenario('gemini_429', { gemini: 429, groq: 200 });
+  assert.ok(o.ok, JSON.stringify(o));
+  assert.equal(o.calls.groq, 1);
+});
+test('Gemini 503 -> Groq answers', async () => {
+  const o = await scenario('gemini_503', { gemini: 503, groq: 200 });
+  assert.ok(o.ok, JSON.stringify(o));
+  assert.equal(o.calls.groq, 1);
+});
+test('Gemini hangs past timeout -> Groq answers', async () => {
+  const o = await scenario('gemini_timeout', { gemini: 'hang', groq: 200 });
+  assert.ok(o.ok, JSON.stringify(o));
+  assert.equal(o.calls.groq, 1);
+});
+test('Gemini 400 (non-retryable): no failover, error surfaces', async () => {
+  const o = await scenario('gemini_400', { gemini: 400, groq: 200 });
+  assert.equal(o.ok, false);
+  assert.equal(o.calls.groq, 0);
+});
+test('both providers fail: combined error', async () => {
+  const o = await scenario('both_fail', { gemini: 429, groq: 429 });
+  assert.equal(o.ok, false);
+  assert.match(o.error, /Both AI providers unavailable/);
+});
+test('no Groq key: no failover even on 429', async () => {
+  const k = process.env.GROQ_API_KEY;
+  delete process.env.GROQ_API_KEY;
+  try {
+    const o = await scenario('no_secondary_key', { gemini: 429, groq: 200 });
+    assert.equal(o.ok, false);
+    assert.equal(o.calls.groq, 0);
+  } finally {
+    process.env.GROQ_API_KEY = k;
+  }
+});