model-cascade

Distill a repeated Jev classification task into a local model, on the fly — same answers, your hardware.

⚖️ Kadı — a calibrated judge in front of your frontier LLM. Routine decisions answered locally with a conformal-prediction guarantee (≤ α disagreement with the oracle); everything else escalated and learned from. Zero-dependency Python.

Distil an expensive LLM API call on a narrow task into a small local model: capture traffic, curate, LoRA fine-tune with MLX on Apple Silicon, evaluate against the teacher, and serve an OpenAI-compatible cascade that escalates low-confidence requests.

Cost-aware routing across agentic pipelines. Empirical benchmarks of accuracy, cost, and dangerous rate.

Confidence-gated LLM cascades that measure how often the cheap tier is wrong on the cases it answers alone.

Studying Minimum Sufficient Inference: when objective execution evidence can stop LLM inference without sacrificing reliability.

NSGA-II search framework for CIFAR-10 big/little dynamic inference cascades under embedded memory constraints.

Ask the cheap model first. Pay for the strong one only when the cheap answer fails a check, under a budget, with a cost record for every call.

LLM serving router that prices every request in dollars and milliseconds. Semantic cache plus a confidence-gated cheap-to-GPT-4 cascade: 96.9% lower cost than always calling the strong model, within 1.6 accuracy points, median latency 450ms to sub-ms on repeated traffic. FastAPI, SQLite cost ledger, reproducible benchmark.

Distill a frontier-model task (gpt-oss-120b, banking77 intents) into a 1.5B QLoRA student served on vLLM, with a confidence router that escalates hard queries to the teacher. Measured: 90% of teacher accuracy alone, 99.7% routed at 28% escalation, break-even ~313K req/day. Spend-capped, Cloud Run + Terraform ready.

A cost-aware decision layer for routing text requests: a cheap model first, an LLM only when confidence is low, a person when both are unsure. FastAPI service with a React dashboard, measured for accuracy, cost and latency.

Adaptive Cascade Tuning via Importance Sampling

Route each LLM call to the cheapest model that will get it right. Verifier-gated cascades with an offline threshold optimizer and cost accounting that counts every verifier call at current list prices.

Divergence-aware multi-agent routing that cuts LLM inference cost 10-100x. One signal routes queries, keys a multilingual cache, and detects hallucinations at AUC 0.90. CIKM 2026.