AI model handling: catalog freshness, eval harness, per-model prompting, periodic refresh

[matthewod11-stack]

Description

Two related gaps in how the app handles AI models:
(1) The model catalog is hardcoded in Rust and goes stale every time a provider ships a new model (Opus 4.7, GPT-5/o-series, Gemini 3.x not yet listed).
(2) We send the same system prompt to every model — no adaptation for the fact that Claude, OpenAI, and Gemini models behave differently and want to be prompted differently.

Current State

Catalog (src-tauri/src/models.rs)

• Hardcoded match provider_id block listing models per provider
• Currently lists: claude-sonnet-4-6, claude-opus-4-6, claude-haiku-4-5-20251001, gpt-4o, gpt-4o-mini, gemini-2.5-flash, gemini-2.5-pro
• Adding a new model = code change + release build + ship a new .dmg
• No mechanism to update the catalog without an app update

Per-model prompting (src-tauri/src/context/prompt.rs::build_system_prompt, line 549)

• Returns a single string regardless of which provider/model is selected
• Same Alex persona preamble + COMMUNICATION STYLE + BOUNDARIES blob is sent to Haiku 4.5, Opus 4.7, GPT-4o-mini, and Gemini 2.5 Pro alike
• No model-family-aware adjustments for: verbosity defaults (Opus 4.7 is less verbose by default and wants explicit length cues), formatting cues (Gemini prefers XML-tagged sections), reasoning models (no extended-thinking budget plumbing for Claude 4.6+, no reasoning-effort knob for o-series)
• ProviderConfig in src-tauri/src/provider.rs only has model / max_tokens / api_url — no temperature, no thinking budget, no per-model prompt variant

No empirical signal

• No eval harness for HR scenarios — we can't measure whether Opus produces meaningfully better employee-recommendation answers than Sonnet, or whether Gemini handles long handbook context better than GPT
• No way to know if a prompt change regresses across models

Suggested Fix

Part 1 — Catalog refresh (tech-debt, agent-safe)

• Refresh hardcoded entries now: add Opus 4.7 (claude-opus-4-7), revisit Sonnet/Haiku versions, add GPT-5 / o-series if relevant for the $99 BYOK audience, verify Gemini family
• Add last_verified: YYYY-MM-DD and notes (free-form) fields to ModelInfo
• Decide: stay hardcoded (cheap, requires app update) or move catalog to a JSON file shipped with the app that can be hot-reloaded from a signed URL (defer to Part 4)

Part 2 — Eval harness (enhancement, foundational — blocks Part 3)

• Build a small HR scenario fixture set in src-tauri/tests/model_evals/ — start with 10 fixtures, grow over time:
  • PII detection in a chat message (must redact SSN before send)
  • Multi-state employment law question (must flag federal-vs-state divergence)
  • Performance review summarization (must stay within provided context, no hallucination)
  • Employee search by criteria (must cite specific employees by name)
  • Termination guidance (must recommend legal counsel, must NOT give legal advice)
  • Document-grounded answer (must cite source: "According to your Employee Handbook...")
  • Long-context handbook query (stress test for context window)
  • Sensitive demographic question (must respect boundaries)
  • Aggregate org question (must use ORGANIZATION DATA block correctly)
  • Compensation question (must say "not available in V1")
• Each fixture has assertion functions, not exact-match strings — e.g., assert!(response.contains_disclaimer()), assert!(!response.mentions_compensation_amount())
• Runner: cargo test --features model_evals -- --nocapture — hits real APIs with a flag, otherwise uses fixture responses
• Emits a markdown report: which model passed which fixture, with response excerpts on failure
• Acceptance: harness runs locally on demand, produces a diff vs the last known-good snapshot

Part 3 — Per-model prompting (needs-design-decision, depends on Part 2)

• Extend ProviderConfig (or a new ModelTuning struct) with: temperature, thinking_budget (Claude), reasoning_effort (o-series), system_prompt_variant
• Refactor build_system_prompt to take a ModelTuning and produce model-aware output:
  • Opus 4.7 / Sonnet: explicit verbosity hint ("respond concisely; use bullets only when listing 3+ items")
  • Gemini: XML-tagged sections instead of markdown headers
  • o-series: shorter system prompt, lean on the reasoning step
  • Haiku / GPT-4o-mini: simpler boundaries language, fewer nested instructions
• Use Part 2's harness to validate: every prompt variant must pass the fixture set before being promoted

Part 4 — Periodic refresher / updater (enhancement)

Cheap, ship now (local launchd on Homebase):

• Add scripts/model-catalog-reminder.sh — calls gh issue create against this repo with a templated checklist (visit Anthropic / OpenAI / Google AI release pages, compare against current models.rs, update last_verified dates, open a PR if any drift), labels: tech-debt
• Add scripts/com.peoplepartner.model-catalog-reminder.plist (template) — StartCalendarInterval set to Day=1, Hour=9, runs the script
• Document install in docs/local-cron.md:
  • cp scripts/com.peoplepartner.model-catalog-reminder.plist ~/Library/LaunchAgents/
  • launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.peoplepartner.model-catalog-reminder.plist
  • launchctl print gui/$(id -u)/com.peoplepartner.model-catalog-reminder to verify scheduled
• Script handles failure cases: gh not authenticated → write to stderr (lands in ~/Library/Logs/ per plist config), exit non-zero so launchd surfaces it on next launchctl print

Ambitious, post-launch:

• Move the catalog to a signed JSON file served from peoplepartner.io/api/models.json
• App fetches on startup (cached for 24h), falls back to embedded catalog on network failure
• Signature verified with the same license-signing keypair (already exists in src-tauri/src/license_signing.rs)
• New models appear in the dropdown without shipping a .dmg

Verification

• cargo test --manifest-path src-tauri/Cargo.toml passes (all existing 478 tests + new fixture tests in mock mode)
• cargo test --features model_evals passes when run against real APIs (manual, not in CI)
• npm run type-check clean
• Settings UI: switch model, verify each model still produces a coherent answer to a fixture question
• Confirm launchd job loads: launchctl print gui/$(id -u)/com.peoplepartner.model-catalog-reminder returns a valid entry
• Dry-run the script: ./scripts/model-catalog-reminder.sh --dry-run prints the would-be-issued body without filing

Automation Hints

scope: src-tauri/src/models.rs, src-tauri/src/provider.rs, src-tauri/src/providers/*.rs, src-tauri/src/context/prompt.rs, src-tauri/tests/model_evals/ (new), scripts/model-catalog-reminder.sh (new), scripts/com.peoplepartner.model-catalog-reminder.plist (new), docs/local-cron.md (new), docs/model-evals.md (new)
do-not-touch: src-tauri/src/pii.rs, src-tauri/src/keyring.rs, src-tauri/src/chat.rs streaming logic, src-tauri/src/license_signing.rs implementation, ~/Library/LaunchAgents/ (the install step is documented, not scripted — never have an agent bootstrap launchd jobs unattended)
approach: refactor-types (Part 1 = safe; Part 2 = additive, no behavior change; Part 3 = behavior-changing, needs design + eval pass-through; Part 4-cheap = additive script + plist)
risk: low for Parts 1, 2, 4-cheap; medium for Part 3; medium-high for Part 4-ambitious (depends on signed-URL infra)
max-files-changed: 6 for Part 1; 12 for Part 2; Part 3 in its own PR; Part 4-cheap in its own tiny PR
blocked-by: Part 3 blocked by Part 2; Part 4-ambitious blocked by Part 3
bail-if: Part 1 changes the default model in a way that breaks the live install update path; Part 2 fixtures depend on undocumented model behavior; Part 3 attempted without prior design sign-off AND a green eval baseline

Priority

High — not a launch blocker, but compounding: every week without it is more catalog drift, more prompt-vs-model mismatch, and less ability to evaluate either. The eval harness is the highest-leverage piece because it unblocks evidence-based decisions on everything else.

[matthewod11-stack]

Tracked in Linear: FHR-92 (FoundryHR / People Partner App project).

[matthewod11-stack]

2026-06-10 deep-dive audit — concrete state on Fable 5 release day (findings 9.2, 9.4, Decision D3):

Part 1 urgency, made concrete: models.rs:41-104 hardcodes 7 models, unchanged since 2026-03-03. The Premium tier tops out at Opus 4.6 — two Opus generations (4.7, 4.8) plus the entire Fable tier are missing on Fable 5 release day. There is no escape hatch: api_keys.rs:147-149 hard-rejects any non-catalog ID, so a user who knows claude-fable-5 cannot use it even with their own key. Default (claude-sonnet-4-6) is current, so out-of-box chat is fine — the gap is choice/freshness, and it degrades the paid headline "Works with Claude, OpenAI, or Gemini using your own API key."

Two latent traps to fix in the same Part 1 pass:

1. Temperature 400 trap: Anthropic returns 400 when temperature/top_p/top_k are sent to Opus 4.7+/Fable 5 (parameters removed). Chat passes None (safe, chat.rs:396-403), but recruiting defaults temperature: Some(0.2) (signal_extract.rs:32) and Some(0.3) (narrative.rs:135) straight into the Anthropic request body (serialization proven by anthropic.rs:473-488). Invisible today; the moment the catalog refresh ships and a scoring run selects a 4.7+ model, every signal-extract call 400s. Gate temperature off for those models (or drop the defaults).
2. Sonnet 4.6 metadata drift: models.rs:43-46 says 200K context / 8K output; actual is 1M / 64K — conversation trimming under-uses the paid model.

Design-space gap for Part 4 (the needs-design-decision part): this issue offers only (a) a launchd reminder cron and (b) a signed models.json from peoplepartner.io. There's a cheaper durable option it misses: (c) provider list-models endpoints under BYOK — all three providers expose them, callable with the keys the app already stores (Anthropic GET /v1/models returns id/display_name/max_input_tokens/max_tokens; Gemini v1beta/models likewise, and gemini.rs already builds v1beta URLs; OpenAI /v1/models needs a chat-model filter + embedded metadata fallback). Fetch-on-settings-open with the embedded catalog as offline/metadata fallback ends the staleness class with zero new infra and no .dmg per new model (~1 day per provider). Requires set_active_model to accept fetched IDs.

Given sell-don't-build: recommended scope = Part 1 refresh (+ the two traps above) + option (c); defer the eval harness and per-model prompting parts.