Project Pathfinder — inference-optimization-agent

A daemon-sidecar for Draycott Technologies that speeds up LLM inference with speculative decoding (a distilled draft model proposes tokens; a larger target model verifies them) while keeping the property legal discovery work requires: every decision carries a reviewable audit trail.

The speculative decoding algorithm, the audit trail, the HTTP sidecar, configuration, and the CLI exist and are covered by a 105-test suite that runs offline with no API key. The real model provider (network transport) is the remaining milestone. What the README documents below is what the code actually does today; every command here runs.

Why the design looks the way it does

The hard constraint is auditability. Speculative decoding makes a sequence of accept/reject decisions, and for legal discovery each one has to be reconstructable after the fact. Two choices follow directly:

• DecisionRecord captures a whole speculative step — the drafted tokens, the draft and target probabilities behind each, what was accepted, the correction token, both model identities, and the seed. It serializes to a plain dict for an append-only log.
• All model behaviour goes through the Provider interface, and the shipped DeterministicStubProvider is reproducible from a stable hash. A recorded run can be replayed to the same tokens, and the test suite runs offline with no API key.

Layout

pathfinderinference/
  __init__.py          package exports and __version__
  __main__.py          `python -m pathfinderinference`
  cli.py               argparse entry point: serve / generate / verify
  config.py            defaults + JSON file + PATHFINDER_* env overrides
  service.py           GenerationService (validation) + stdlib HTTP daemon
  observability.py     JSON logging + in-process Metrics
  domain.py            value types: distributions, params, requests, records
  engine.py            SpeculativeDecoder: the speculative sampling algorithm
  audit.py             hash-chained, append-only audit sinks + verify_chain
  providers/
    base.py            Provider ABC + ProviderInfo (the interface + its contract)
    stub.py            DeterministicStubProvider (offline, reproducible)
    real.py            RealProvider (HTTP transport; not wired yet)
tests/                 domain, provider, engine, algorithm, audit, config,
                       service, http, cli, observability, hardening, offline
config.example.json    a full config file to copy and edit

Architecture

One request flows through these components:

HTTP POST /generate                    (service.py: stdlib http.server)
  -> GenerationService._validate        reject bad input as 400
  -> [ single lock ]                     one request decodes at a time
       SpeculativeDecoder.generate       (engine.py) the algorithm
         -> draft Provider.next_distributions   propose k tokens (q)
         -> target Provider.next_distributions  verify k+1 positions (p)
         -> accept/reject + residual resample
         -> AuditSink.record(DecisionRecord)    persist BEFORE emitting tokens
  -> JSON response + audit pointer + latency

Two invariants shape everything:

1. No token without a record. The DecisionRecord for a step is written to the audit sink before the step's tokens are added to the output. If the sink fails, the request fails closed (HTTP 503) rather than returning unaudited output. This is why SpeculativeDecoder takes an AuditSink as a required argument, not an optional one.
2. Reproducibility. Providers are pure functions of (context, params) and the stub derives its distributions from a stable (non-salted) hash, so a recorded run replays to the same tokens — a reviewer can re-run an audited decode and get byte-identical decisions.

The contested calls behind this shape — stdlib HTTP over a framework, a hash-chained log over a database, fail-closed over fail-open, one global chain over per-request concurrency — are recorded in docs/adr/.

The algorithm

SpeculativeDecoder.generate runs standard speculative sampling (Leviathan et al. 2023 / Chen et al. 2023):

1. The draft model proposes k tokens autoregressively from its distribution q.
2. The target model scores all k positions plus one more in a single verify pass, giving distributions p.
3. Each drafted token x is accepted with probability min(1, p(x)/q(x)). On the first rejection, the step resamples from the residual norm(max(0, p - q)) and stops; if all are accepted, a bonus token is sampled from the target.

Step 3 is why the output is distributed exactly as if sampled from the target alone — the draft only affects speed. The test suite checks this three ways: test_greedy_matches_pure_target_argmax (temperature 0 reduces to target greedy decoding), test_identical_draft_and_target_accepts_everything (q == p accepts every token), and test_speculative_sampling_preserves_target_distribution (the empirical output distribution matches the target's over 6000 seeds).

The audit trail

The hard constraint — every decision carries a reviewable audit trail — is enforced structurally: SpeculativeDecoder takes an AuditSink as a required argument, and each speculative step writes one DecisionRecord (drafted tokens, draft and target probabilities, what was accepted, the correction token, both model identities, the seed, and a human-readable rationale) before the tokens are returned. There is no code path that emits a token without recording the decision.

Sinks chain records with a hash: each entry embeds the previous entry's hash, so editing, inserting, or dropping any past record breaks the chain. verify_chain raises AuditChainError at the first broken link. JsonlAuditSink persists one entry per line and fsyncs each write; load_jsonl reads a trail back for verification.

from pathfinderinference import (
    SpeculativeDecoder, DeterministicStubProvider, JsonlAuditSink,
    GenerationRequest, GenerationParams, load_jsonl, verify_chain,
)

with JsonlAuditSink("run.audit.jsonl") as audit:
    decoder = SpeculativeDecoder(
        draft=DeterministicStubProvider(seed=1),
        target=DeterministicStubProvider(seed=2),
        audit=audit,
    )
    result = decoder.generate(GenerationRequest(
        request_id="req-1",
        prompt_token_ids=(1, 2, 3),
        params=GenerationParams(max_tokens=16, temperature=1.0, seed=0),
    ))

print(result.output_token_ids, "acceptance:", result.acceptance_rate)
verify_chain(load_jsonl("run.audit.jsonl"))  # raises if the trail was altered

The provider contract

Provider.next_distributions(context, num_positions, params) returns one next-token distribution per trailing prefix of context: element t is P(next | context[: len(context) - num_positions + 1 + t]). This mirrors a single transformer forward pass, so one method serves both roles — draft a token with num_positions=1, or verify k drafted tokens in one call with num_positions=k+1. Implementations must be pure in (context, params) so audited runs reproduce. See the docstring in pathfinderinference/providers/base.py.

Setup and use

Requires Python 3.12+. Runtime is stdlib-only; pytest is the only dev dependency.

make venv        # create .venv
make install     # editable install with dev extras into .venv
make test        # run the suite
make import      # confirm the package imports: prints "pathfinderinference 0.1.0"

PY overrides the interpreter, e.g. make test PY=python3 to use a system interpreter instead of .venv.

The test suite

105 tests, all offline with no API key. Coverage worth calling out:

• test_algorithm.py pins the accept/reject/residual rule on hand-built distributions where the outcome is known by hand (accept-all with bonus, reject-and-resample, accept-a-prefix-then-reject, ratio-bounded acceptance over 4000 draws), plus edge cases (single-token output, length-one prompt, draft_length clamped to the token budget) and a failure path (a provider that raises mid-decode propagates and leaves no partial audit entry).
• test_engine.py proves distribution correctness three ways (greedy equivalence, q == p accepts everything, empirical match over 6000 seeds).
• tests/conftest.py patches socket.connect for the whole suite to reject any non-loopback address, so the offline requirement is enforced, not assumed. test_offline.py removes socket.socket entirely around a full decode plus audit-to-disk.

Confirm the package imports without the Makefile:

python -c "import pathfinderinference"

Minimal use of what exists today:

from pathfinderinference import DeterministicStubProvider, GenerationParams

provider = DeterministicStubProvider(seed=7)
dist = provider.next_distributions((1, 2, 3), num_positions=1,
                                   params=GenerationParams(temperature=1.0))[0]
print(dist.argmax(), dist.probability(dist.argmax()))

Running the sidecar

The entry point is pathfinder (installed by make install) or, without installing, python -m pathfinderinference. Three subcommands:

This sequence runs from a fresh checkout with no arguments beyond what is shown (no config file or API key needed):

# 1. Decode one request and print JSON. Writes the audit trail to the default
#    path, creating audit-logs/ if needed.
python -m pathfinderinference generate --prompt 1,2,3 --max-tokens 8

# 2. Verify the hash chain of the trail step 1 just wrote.
python -m pathfinderinference verify audit-logs/pathfinder.audit.jsonl

# 3. Start the HTTP sidecar (Ctrl-C or SIGTERM to stop).
python -m pathfinderinference serve --config config.example.json

audit-logs/pathfinder.audit.jsonl is created at runtime by step 1; it is not part of the checkout.

Endpoints:

• GET /healthz — {"status": "ok", "version": "0.1.0"}.
• GET /metrics — a JSON snapshot of counters and summaries (see Observability).
• POST /generate — body {"request_id", "prompt_token_ids", "max_tokens", "temperature", "seed", "draft_length", "eos_token_id"} (only the first two are required). Returns the output tokens, acceptance stats, latency_ms, and an audit pointer (first_seq/last_seq) into the persisted trail.

curl -s localhost:8080/healthz
curl -s -X POST localhost:8080/generate \
  -H 'Content-Type: application/json' \
  -d '{"request_id":"c1","prompt_token_ids":[4,5,6],"max_tokens":5}'

Configuration reference

Precedence, lowest to highest: built-in defaults, a JSON config file (--config), PATHFINDER_* environment variables, then CLI flags. Unknown keys in a config file are rejected rather than ignored. Copy config.example.json and edit it.

Full set of keys (config-file path, type, default):

Key	Type	Default	Meaning
host	str	127.0.0.1	bind host
port	int	8080	bind port (0 = OS-assigned)
audit_path	str	audit-logs/pathfinder.audit.jsonl	JSONL audit log
log_level	str	INFO	DEBUG/INFO/WARNING/ERROR/CRITICAL
draft.kind / target.kind	str	stub	stub or real
draft.model_id / target.model_id	str	stub-model / target-model	identity recorded in the trail
draft.revision / target.revision	str	deterministic-v1	model revision recorded in the trail
draft.seed / target.seed	int	0 / 1	stub determinism seed
draft.vocab_size / target.vocab_size	int	32	stub vocabulary size
draft.endpoint / target.endpoint	str	none	required when kind is real
draft.timeout_s / target.timeout_s	float	30.0	real-provider request timeout
limits.max_prompt_tokens	int	4096	reject longer prompts
limits.max_output_tokens	int	1024	cap on max_tokens
limits.max_draft_length	int	16	cap on draft_length
limits.max_body_bytes	int	1048576	HTTP request body cap
limits.request_timeout_s	float	30.0	per-connection socket timeout

Environment variables map onto a subset (nested keys use _):

Variable	Key
PATHFINDER_HOST, PATHFINDER_PORT	host, port
PATHFINDER_AUDIT_PATH	audit_path
PATHFINDER_LOG_LEVEL	log_level
PATHFINDER_DRAFT_KIND, PATHFINDER_TARGET_KIND	draft.kind, target.kind
PATHFINDER_DRAFT_MODEL_ID, PATHFINDER_TARGET_MODEL_ID	*.model_id
PATHFINDER_DRAFT_ENDPOINT, PATHFINDER_TARGET_ENDPOINT	*.endpoint
PATHFINDER_DRAFT_SEED, PATHFINDER_TARGET_SEED	*.seed
PATHFINDER_MAX_PROMPT_TOKENS, PATHFINDER_MAX_OUTPUT_TOKENS	limits.*

CLI flags override both: --host, --port, --audit-path, --log-level (see python -m pathfinderinference serve --help).

Failure handling

Guarded paths, with the responses they produce:

• Malformed JSON, missing/invalid fields, out-of-range values — 400 with a message naming the field.
• Request body over max_body_bytes (default 1 MiB) — 413; missing Content-Length — 411.
• A configured real provider that isn't wired yet — 503 (dependency unavailable), not a 500.
• Audit trail unwritable — 503 (audit unavailable). This is the constraint-driven degradation: the decision is recorded before any token is returned, so a persistence failure fails the request closed — the daemon refuses to serve output it could not record rather than dropping the audit. An unwritable audit directory is caught at startup (non-zero exit), not at the first request.
• Unreadable or non-JSON config file, missing audit directory — a clear message on stderr and a non-zero exit before the server starts.
• A client that stalls mid-request is dropped after request_timeout_s (default 30s) so it can't pin a worker thread.
• Prompt and output length caps guard against a single request exhausting memory.

Observability

Logs are one JSON object per line on stderr (configure_logging), so each line can be correlated with the audit trail by request_id. The generate app-log line carries acceptance_rate, latency_ms, and the audit_first_seq / audit_last_seq range; a separate access-log line records method, path, status, and latency. Set the level with --log-level or PATHFINDER_LOG_LEVEL.

GET /metrics (and GenerationService.metrics_snapshot()) returns counters (requests.ok, requests.invalid, requests.audit_unavailable, tokens.generated, decode.steps, decode.drafted_total, decode.accepted_total) and timing summaries (request.latency_seconds, and decode.draft_seconds vs. decode.verify_seconds). The draft-vs-verify split and the drafted-vs-accepted counts are the numbers that explain speculative decoding's payoff on a given draft/target pair.

Known limitations

• No real model backend yet. RealProvider constructs without any I/O (loading the sidecar must not emit traffic) and raises NotImplementedError on use. Only the stub provider produces tokens today. See ADR-0005 for the wire tradeoff the transport must resolve.
• No wall-clock speedup to report. The stub is not a fast model, so the work so far verifies correctness and the audit trail, not latency. Acceptance rate (GenerationResult.acceptance_rate, exposed at /metrics) is the proxy that predicts speedup once a real draft/target pair is wired.
• One request decodes at a time. Requests serialize through a single lock so each request's decisions form a contiguous block in one global hash chain. This caps decode throughput; see ADR-0003 for the alternative (per-request chains) and when to revisit.
• Audit storage is availability-critical. By design the daemon fails closed when it cannot persist a decision (ADR-0004), so a full or unwritable disk takes the endpoint down rather than serving unaudited output.
• The HTTP server trusts its clients. It is meant to run as a localhost sidecar; it has no authentication, TLS, or rate limiting, and a peer that lies about Content-Length can stall a worker thread up to request_timeout_s.
• Stub vocabulary is integer token ids only. There is no tokenizer; prompts and outputs are token-id lists. A real deployment pairs this with the tokenizer that its models use.

Architecture decision records

Numbered ADRs for the genuinely contested calls live in docs/adr/. Each records Context, Decision, and Consequences.

---

Draycott Technologies is an illustrative client; this repository is a self-directed reference implementation built to work end to end.