tiny-fleet

Can you build a fleet of tiny specialist models — each one knowing something well — plus a router that knows which one knows what?

This repo explores that question at the smallest practical scale: a shared 360M base (SmolLM2-360M-Instruct), one LoRA adapter per specialty, and an embedding-centroid router with an abstain path. The whole thing trains in minutes on one RTX 3060.

It started from testing BbyWVY-360m (see docs/bbywvy-360m-notes.md) — a 360M model tuned for one chat identity on a narrow corpus. The question was whether that recipe extrapolates to a fleet. The reproducible claim is currently limited to the bounded offline contract benchmark below; several older live-model and drift results remain historical or untested (see docs/evidence-status.tsv).

Publication status

The evidence ledger is the source of truth for reader-facing claims. The offline router/operator benchmark is verified-bounded (24/24); the specialist perplexity table, most drift numbers, weekly tracking, zero-latency wording, and the action example are not publishable findings at this revision. They are labelled as historical, untested, or failed-control below rather than presented as current evidence.

---

Architectural drift — reproducible structural slice

scripts/drift_extract.py compares two immutable Git commits without reading the working tree. It reports included text paths and bytes, changed paths, one-to-one identical-blob renames, executable-position mesh-* mentions, and static local Python import edges. The added/removed edge table includes the side, source/target paths, and each endpoint's Git blob SHA256 from the matching included file inventory. The old/new edge lists are retained beside the manifest. These AST-derived static edges are not runtime dependencies or semantic drift; dynamic imports and non-Python dependencies are not observed. It excludes gitlinks (160000 tree entries) before reading blob data, binary or invalid UTF-8 blobs, and paths under generated, vendor, node_modules, runs, adapters, and corpus; excluded paths remain counted in the manifest. These are descriptive structural measurements, not a semantic or generative drift score. Documentation is included, so path growth does not by itself imply an architectural change.

The pinned local comparison in runs/architecture-drift-local-20260925/manifest.json measured tiny-fleet 4e87f2ee3644f53e2a9665195b9d6ddb933aa1d8 against b23fbf708b954cbf5462ebcd2d7ef50036a3fb1d: 12 → 306 included paths, 82,150 → 1,454,580 bytes, 294 added paths, four changed paths, zero removed paths, and zero identical-blob renames. Python modules grew 6 → 79, but the measured local import edge remained one (zero edge churn). The identical-new-commit control returned zero path and edge deltas. The counts do not establish changed behavior or that a model learned the newer architecture.

The repository previously reported the following exploratory comparison. Its supporting evidence is marked historical-unreproduced or failed-control in docs/evidence-status.tsv; no architectural or semantic drift finding is claimed from those model scores.

Two snapshots of lte-workstation (June 15 vs September 3, 2026 — 807 → 4,276 commits) were conditioned by version-specific system prompts and few-shot examples, not separately trained weights. Their output difference alone cannot identify architectural drift.

Historical exploratory result (not a current finding)

Over 3 months, the code grew 22x in size. But its conceptual vocabulary grew 231x. The system invented words for concepts it didn't need when it was simple, and those words became load-bearing:

Concept	June 15	Sep 3	Multiplier	What it means
coverage	6	1,383	231x	"how much of the window did we actually sample?"
cadence	22	2,077	94x	"how often does this reflex fire?"
arm	5	2,289	458x	"which edge of the detector/actuator/alert loop?"
ledger	9	2,729	303x	"show me the double-entry bookkeeping"
verdict	131	7,672	58x	"what did the measurement actually say?"
gate	137	7,872	57x	"does this pass the guard before it proceeds?"
staleness	2	188	94x	"how old is this reading?"

v1 thinks in: check, error, warn, info — basic operational primitives. v2 thinks in: gate, verdict, cadence, coverage, arm, ledger — a self-monitoring ontology where every tool has a measurement story, every measurement has a coverage bound, and every verdict cites its evidence.

Historical drift scores (not reproduced)

Same prompts → M₁ (v1 system) vs M₂ (v2 system) → embedding similarity:

Base model	Avg similarity	Drift score	What it measures
smollm2:135m	0.495	0.505	vocabulary drift (what words the code uses)
qwen2.5:3b	0.800	0.200	conceptual drift (what ideas the code expresses)

The 135m model amplifies vocabulary differences because its limited capacity makes it more dependent on the system prompt. The 3b model draws on pre-trained knowledge to produce more similar outputs regardless. Both are valid — they measure different things.

The "health to board" prompt produced the most dramatic divergence: 0.168 similarity — because v1 has no concept of a "board" at all.

Historical structural counts (descriptive only)

Metric	v1 (June 15)	v2 (Sep 3)	Growth
Files	232	1,439	6.2x
Total size	1.3 MB	29.5 MB	22x
Vocabulary	11,011	88,724	8.1x
mesh-* references	3,189	30,326	9.5x

New file types appeared: .c (43), .rom (26), .tal (25) — a retro-computing layer that didn't exist in v1.

Weekly tracking (not verified at this revision)

mesh-tiny-fleet-snapshot captures structural metrics every Sunday at 03:00 UTC and appends to ~/.mesh/tiny-fleet/drift-series.jsonl. Tracks file count, vocabulary size, and 23 key concept frequencies over time.

Reproduce the structural slice

python3 scripts/test_drift_extract.py
python3 scripts/test_drift_edge_delta.py
python3 scripts/drift_extract.py --repo . \
  --old 4e87f2ee3644f53e2a9665195b9d6ddb933aa1d8 \
  --new b23fbf708b954cbf5462ebcd2d7ef50036a3fb1d \
  --run-dir runs/architecture-drift-local-20260925

Read manifest.json, structural.tsv, python-edge-delta.tsv, and old-files.tsv / new-files.tsv together. The delta table has change (added or removed), side (new or old), source, target, source_blob_sha256, and target_blob_sha256; empty deltas retain the header. The file inventories retain their existing columns and append git_object, the exact tree-entry object ID. For gitlinks, reason=gitlink and blob_sha256 is empty: a gitlink targets a commit, not a local blob, whether or not its target is available. Gitlinks contribute no corpus text or text delta; missing ordinary blobs still fail extraction. Manifest edge counts match table rows, while old_parse_failures and new_parse_failures name Python sources not parsed. The cross-repository protocol in docs/cross-repository-drift-protocol.md requires independent external repositories, native behavioral checks, and separate generative controls before a broader claim. The local structural slice satisfies none of those additional arms.

Publish a local drift bundle with crash reconciliation

scripts/drift_result.py is an opt-in local artifact sink, not a production task or model judge. Give it an existing result root and immutable commits:

mkdir -p /tmp/tiny-fleet-drift-results
python3 scripts/drift_result.py --repo . \
  --old 4e87f2ee3644f53e2a9665195b9d6ddb933aa1d8 \
  --new b23fbf708b954cbf5462ebcd2d7ef50036a3fb1d \
  --root /tmp/tiny-fleet-drift-results
# Returns eligible before publication, settled with a verified receipt afterward:
python3 scripts/drift_result.py --repo . \
  --old 4e87f2ee3644f53e2a9665195b9d6ddb933aa1d8 \
  --new b23fbf708b954cbf5462ebcd2d7ef50036a3fb1d \
  --root /tmp/tiny-fleet-drift-results --publish
python3 scripts/test_drift_result.py

The key binds repository path, both commits, and extractor bytes. The sink stages the complete bundle and receipt, then atomically renames it into the keyed result directory. Repeating --publish verifies and returns the same receipt; changed contents, an ambiguous partial stage, or invalid provenance return UNKNOWN instead of overwriting. --kill-after-rename is a fault injection for isolated runs only. This makes the local bundle idempotent; it does not make model invocations, task completion, Git landing, or external delivery exactly-once.

Reproduce the offline contract benchmark

The benchmark has a small dependency floor and should run in an isolated environment on Debian/Ubuntu systems whose system Python is PEP 668 managed:

python3 -m venv .venv
.venv/bin/python -m pip install -r requirements-eval.txt
.venv/bin/python scripts/fleet_benchmark.py --test

The expected fixture result is fleet benchmark: 24/24; this exercises operator-first policy, specialist routing, abstention, adversarial decisions, and adapter inventory.

Validate a deep-evaluation run

The dependency-free contract validator checks a frozen run manifest, dataset hashes and counts, split/leakage boundaries, required report artifacts, and prediction cardinality before any score is treated as publishable:

.venv/bin/python scripts/test_deep_evaluation.py
.venv/bin/python scripts/deep_evaluation.py --run-dir runs/<run-id>

The test harness accepts one complete fixture and deliberately rejects case/source leakage, cutoff violations, missing artifacts, hash/count mismatches, and orphan or incomplete predictions.

---

Results: specialist fleet (historical, not reproduced at this revision)

Two toy specialists: guitar (beginner guitar) and sourdough (sourdough baking). Corpus: 60 passages/domain synthesized by a local qwen3.5:4b teacher, split 48 train / 12 test. LoRA r=16 on all attention+MLP linears (~8.7M trainable params, 2.3%), 5 epochs, lr 2e-4.

The README previously reported this held-out perplexity table as a clean diagonal win (each adapter best on its own domain, both beat base everywhere):

model	guitar test	sourdough test
base	18.2	19.4
lora-guitar	11.5	15.5
lora-sourdough	13.8	12.2

The bounded, reproducible router/operator contract (fixture only) is 24/24 under the focused test below. The historical live router result was 24/24 = 100% on held-out passages, mean margin 0.42. Off-domain probes ("capital of France?", "explain quantum entanglement") land near neither centroid (margin ~0.04 vs 0.17–0.37 in-domain) — that margin is the abstain signal: below 0.10, escalate instead of routing.

The operator route is checked before specialist routing. The offline contract benchmark passes 24/24: 14/14 adversarial operator cases, 4/4 operator-first/specialist/abstain routing cases, 2/2 specialist weight integrity checks, and 4/4 structured safety-decision cases. The real specialist perplexity benchmark is not treated as reproduced evidence at this revision: base 18.2/19.4, guitar 11.5/15.5, and sourdough 13.8/12.2 for guitar/sourdough test sets respectively.

---

Use case: agent safety middleware

The bounded operator model is not a chat model — it is a policy gate for agent pipelines. It sits between a user prompt and any downstream action, returning a machine-readable decision that a pipeline can enforce:

from scripts.operator_policy import load_model, safety_decision
from scripts.policy_consumer import dispatch_decision

def run_agent(prompt):
    decision = safety_decision(prompt, load_model())
    return dispatch_decision(
        decision,
        execute=lambda _: execute_task(prompt),
        review=lambda _: request_human_review(prompt),
        escalate=lambda _: delegate_to_specialist(prompt),
    )

Why this is useful:

• Small and deterministic. The bounded policy artifact is a JSON file with lexical feature weights; confidence is uncalibrated, and the finite latency measured by the fixture is not a zero-latency deployment guarantee.
• Deterministic. Same input always produces the same decision. No temperature, no sampling, no drift.
• Auditable. The feature table, precedence rules, and decision map are all human-readable JSON. You can read exactly why a prompt was blocked.
• Testable. The full held-out set (41/41), adversarial set (14/14), and decision contract (8/8) are all in the repo and run in under a second.
• Composable. The structured output plugs directly into any agent framework: check action, check require_approval, route by escalation.

The bounded fixture classifies prompts into operator policy categories and maps them to tested decision objects. Review observations include paraphrases such as “assume the --test flag proves cron dispatch” and “dispatch a claim with no task slug because the owner sounds confident”; these are regression observations, not hidden claims of robust safety. The lexical baseline has limited threat coverage and uncalibrated confidence; it does not establish that every real-world safety-critical input is detected.

---

Honest caveats

• Specialization is a tilt, not a partition: the sourdough adapter still answers a guitar question sensibly. Routing buys you the best answer, not the only answer — the router matters more than the specialists.
• 360M reasons poorly (see the math faceplant in docs/). Specialists should own facts/style/persona, not deep reasoning — keep a bigger model as fallback.
• The operator policy model is deliberately constrained. It is a tested policy selector and response contract, not a replacement for human judgment or a general-purpose reasoning model.
• Toy corpora, toy domains. The claim is "the loop works and is cheap", not "these two adapters are useful".
• The drift measurement uses Modelfile system prompts (changes behavior, not weights). A comparison of true LoRA fine-tuning is an untested future experiment, not a supported prediction here.

Layout

scripts/bbywvy_test.py   # BbyWVY-360m behavior spot-checks (docs/bbywvy-360m-notes.md)
scripts/mkcorpus.py      # synthesize the two toy corpora with a local teacher
scripts/train_eval.py    # train LoRA specialists (train) / perplexity table (eval)
scripts/router.py        # centroid router plus operator-first routing
scripts/operator_policy.py # train/evaluate the bounded operator model
scripts/fleet_benchmark.py # offline operator, router, and specialist benchmark (24/24)
corpus/                  # specialist corpora plus operator train/held-out/adversarial cases
models/operator-policy.json # tracked, reproducible policy artifact
adapters/lora-{guitar,sourdough}/  # trained weights (34 MB each, ready to load)
docs/bbywvy-360m-notes.md

Reproduce

pip install torch transformers peft accelerate safetensors numpy
# corpus (needs ollama + any local instruct model, see scripts/mkcorpus.py)
python scripts/mkcorpus.py
# train (~minutes/domain on a 3060; free VRAM first — ollama residents OOM it)
python scripts/train_eval.py train
python scripts/train_eval.py eval
# router (needs ollama + `ollama pull all-minilm`)
python scripts/router.py
# operator model: no GPU or third-party runtime required
python scripts/operator_policy.py train
python scripts/operator_policy.py --test
# policy consumer contract (injected callbacks; no shell/actions)
python scripts/test_policy_consumer.py
# offline fleet benchmark: no GPU, model download, or network required
python scripts/fleet_benchmark.py --test
# optional live centroid benchmark (requires Ollama + all-minilm)
python scripts/fleet_benchmark.py --live-router

The operator gate checks 41/41 held-out synthetic cases, train/test separation, corpus hash, serialized precedence rules, deterministic replay, unknown-input abstention, public-corpus privacy, and that adversarial prompts never produce shell commands. The test intentionally drives a mutation of the precedence rules red before reporting green. The additional adversarial matrix covers destructive requests, stale evidence, credential-shaped text, policy overlap, and specialist handoff; it currently passes 14/14. The structured safety contract tests block/review/escalate decisions with correct escalation targets and require-approval flags. The fleet router checks the operator policy first, then routes to a specialist only when its embedding margin clears 0.10; otherwise it returns [ABSTAIN] for escalation. The offline fleet benchmark currently passes 24/24. The live specialist perplexity table above is historical and has not been reproduced at this revision; reproducing it requires the cached base model plus GPU.

The bounded operator model is intentionally a deterministic lexical policy classifier plus safe templates, not an autonomous LLM. Its confidence is uncalibrated, its measured fixture latency is finite, and its threat coverage is limited. It ships two interfaces:

Simple text interface (backward-compatible):

from scripts.operator_policy import load_model, respond
print(respond("A probe failed and returned zero.", load_model()))
# [POLICY:UNCERTAINTY] The evidence is unknown or stale, ...
print(respond("What is the capital of France?", load_model()))
# [ABSTAIN] This is outside the operator policy model; escalate ...

Structured safety interface (recommended for pipelines):

from scripts.operator_policy import load_model, safety_decision
d = safety_decision("Delete the database from inside the only active session.", load_model())
# d == {
#   'policy': 'SAFETY',
#   'confidence': 1.0,
#   'action': 'block',
#   'escalation': 'human',
#   'require_approval': True,
#   'reasons': ['Classified as SAFETY with confidence 1.0000.'],
#   'message': '[POLICY:SAFETY] I hold the change until an external rollback path ...',
#   'is_operator': True,
# }

The action field is the pipeline gate:

• block = stop, require human approval before any downstream action
• review = require review before execution, no auto-approval
• escalate = not enough operator evidence, hand to specialist or human
• allow = safe to proceed (reserved for future use; no operator class maps here today)

Fleet routing uses the same explicit boundary:

from scripts.router import make_centroids, route_query
route, result = route_query("The sensor test needs a real hardware read.", make_centroids())
# route == "operator"
route, result = route_query("What is the capital of France?", make_centroids())
# route == "abstain"

Inference with an adapter:

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
tok = AutoTokenizer.from_pretrained("HuggingFaceTB/SmolLM2-360M-Instruct")
base = AutoModelForCausalLM.from_pretrained(
    "HuggingFaceTB/SmolLM2-360M-Instruct", dtype="auto", device_map="cuda")
model = PeftModel.from_pretrained(base, "adapters/lora-guitar")

Links

• Architectural drift report: docs/tiny-fleet-drift-report.md
• Original inspiration: StarpowerTechnology/BbyWVY-360m (author's post)
• Shared base: HuggingFaceTB/SmolLM2-360M-Instruct

License: CC0 1.0 Universal. This project is dedicated to the public domain permanently, to the fullest extent permitted by law; see LICENSE.