Geometry Lab: fine-tuning ignores a scratch-trained model's own vocabulary

[jeremymanning]

Found by an adversarial review of the feature-005 documentation: a panel note claimed
fine-tuning "keeps the existing vocabulary". It does not, and the underlying behaviour is
wrong rather than the note being merely sloppy.

What happens

train from scratch mints a model with a fresh 1000-word vocabulary built from the
user's text, so its token ids mean different words than the shipped checkpoint's.
Fine-tuning that model then encodes the fine-tuning text with the shipped vocabulary:

• code/backend/src/llm_geometry/geo/finetune.py:157 — ids = get_tokenizer().encode_stream(text)
  (the module-level canonical tokenizer, never tokenizer_for(base))
• code/frontend/src/lib/staticClient/geo.ts — same, and its own docstring says
  "Fine-tuning keeps the shipped vocabulary"

So gradients land on rows whose ids mean something else entirely.

It also loses the base's vocabulary on the way out:
finetune.py:207 calls save_weight_set(new_ws, source="finetuned", store=store) with no
vocab_json, so tokenizer_for() falls back to the canonical tokenizer for the result —
and every dot on the sphere is relabelled with the shipped words.

Repro

1. Train from scratch on any text with ≥1000 distinct types.
2. Fine-tune the resulting model on a paragraph.
3. Hover the dots: they are labelled with the shipped checkpoint's words, not yours.

Fix sketch

• finetune.py: tokenize with tokenizer_for(base_token, store), and pass the base's
  vocab_json through to save_weight_set so the result keeps its own words.
• Mirror in staticClient/geo.ts via the engine's tokenizerFor(base).
• Test: scratch-train, fine-tune, assert the fine-tuned model's vocabulary equals the
  scratch model's and that a known word round-trips to the same id.

Documented in the meantime: the Info tab and the fine-tune panel both now say outright that
fine-tuning uses the shipped vocabulary, and link here.