A curated example keeps the same id across deploys, and the caller can see it - #233
Merged
Merged
Conversation
…n see it The id column was in the primary key and useless three ways over: absent an authored id the seed minted a uuid4, nothing wrote it back so the next deploy minted a different one for the same example, and the read never selected the column so no id reached the caller even when one held something. So no curated example had a durable identity — nothing could name one, count one, or tell whether the one returned yesterday was the one returned today. The id is now derived from the example's own content: question + sql, 12 hex characters, the construction compute_model_hash already uses. Derived rather than minted because a minted id changes every deploy, and derived rather than authored because authoring gives the id two homes that can disagree. area is excluded deliberately — it is not in the example file, the deploy injects it from the subject-area directory name — so two examples sharing question and sql now collapse to one id, which is correct rather than a collision. An authored id in the YAML still wins; derivation is only the fallback. That collapse is also why write_examples dedups: area is not in the primary key, so the second insert of a repeated id would raise and abort the whole deploy. The rows are deleted before the batch, so a within-call repeat is the only collision reachable, and first wins keeps the surviving row's area stable across re-seeds. example_by_id ships with the derivation rather than after it. The hash is one-way, so an id is inert without a lookup, and there was none; adding it in a later release would leave the id write-only until that release landed. Spec: ACE-109
vishalkalbi27
requested review from
ashwin-agami and
sandeep-agami
as code owners
August 17, 2026 11:17
There was a problem hiding this comment.
Pull request overview
This PR makes prompt_example identities durable across deploys by deriving a stable id from an example’s own content, returning that id to callers, and adding a reverse lookup to resolve an id back to the stored example (ACE-109).
Changes:
- Add deterministic
example_id(ex)and use it as the fallback id inwrite_examples, with within-batch dedup to avoid PK collisions. - Update
select_examplesto select/return theidcolumn (column wins over any staleidinsidedoc) and addexample_by_id(...). - Add new and expanded tests to pin hash construction and ensure served examples include ids.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| tests/test_prompt_examples_serving.py | Adds an integration-style test asserting served examples include id and it matches the derivation. |
| tests/test_ace109_example_identity.py | New test suite covering id derivation properties, reseeding stability, dedup behavior, and example_by_id. |
| packages/agami-core/src/model_store.py | Implements deterministic example ids, dedup in seeding, returns ids from reads, and adds lookup by id. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+413
to
+415
| # Keep a stable id across re-seeds when the example carries one (so per-example identity | ||
| # survives a redeploy); mint one only when absent. | ||
| ex_id = str(ex.get("id") or uuid4().hex) | ||
| # survives a redeploy); derive one from its content when absent. | ||
| ex_id = str(ex.get("id") or example_id(ex)) |
`ex.get("id") or example_id(ex)` reads any falsy value as "no id was authored". An
unquoted `id: 0` in YAML parses to an int, so an example that carries an id gets a
derived one instead — and a hand-numbered library starts at exactly that value.
The failure is silent. The example keeps working, under an identity its author did
not choose, and the escape hatch this seed documents does not hold for the one value
most likely to be written first.
Absent now means absent: `None`, or a key present but empty. Everything else is an
authored id, including `0`.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
prompt_examplehas anidcolumn in its primary key and it was unusable three ways over: absent anauthored
idthe seed minted auuid4(), nothing wrote it back so the next deploy minted adifferent one for the same example, and the read selected
question, doc— never theidcolumn —so no id reached the caller even when one held something.
The effect is that a curated example has no durable identity. Nothing can name one, count one, or
say whether the one returned yesterday is the one returned today. Three things already work around
its absence:
agami-save-correctioncannot tell a correction that created a new example from onethat should have replaced an existing one; ranking returns a subset that is not recordable, so
example-set behaviour is not reproducible across deploys; and the library only grows, because no two
observations of an example can be tied together.
This derives the id from the example's own content, returns it on the served path, and adds the
lookup that turns a one-way hash back into an example.
Changes
example_id(ex)— a 12-character SHA-256 prefix overquestion+sql, each part UTF-8encoded and NUL-terminated, copied from
snapshot.py::_hash_and_manifest(whatcompute_model_hashuses). Terminating rather than joining is what keeps("ab", "c")and("a", "bc")distinct. Byte-exact — no strip, no case-fold — matching that same precedent.write_examplesderives instead of minting. An authoredidin the YAML still wins; theexisting escape hatch and its comment are untouched. Derivation is the fallback for the case that
is universal in practice.
write_examplesdedups within the batch, first-wins.areais deliberately not hashed — it isnot carried in the example file, the deploy injects it from the subject-area directory name — so
two examples sharing
questionandsqlnow collapse to one id.areais not inPRIMARY KEY (org_id, datasource, id)either, so without the dedup the secondINSERTwould raiseand abort the whole deploy. The rows are deleted before the batch, so a within-call repeat is the
only collision reachable.
select_examplesselectsidand merges it into each returned dict, column-wins, beforemeasuring against the char budget so the accounting reflects what is actually returned.
example_by_id(store, *, org_id, datasource, example_id)— the derivation is one-way, so an idis inert without a lookup and there was none. Scoped by org and datasource like every other read in
the module;
Nonewhen the id was never seeded.No schema change, no migration, no tool-schema change. The local file path is untouched: it returns
markdown-wrapped raw YAML rather than dicts, so an id there means parsing and re-rendering YAML in a
path that has no parser today.
Two behaviour changes worth calling out
served for both areas. Collapsed to one id it keeps the first area, and
select_examplesfiltersarea = ? OR area IS NULL, so the second area's scoped query no longer finds it. Writing thesurvivor into the cross-area (
area IS NULL) bucket would preserve both, but would also serve itto areas it was never filed under. Narrowing a rare deliberate duplicate beats polluting every
unrelated few-shot set.
IntegrityErrorand take thedeploy down; they now skip. That is a strict improvement, and it is noted in the comment.
Checklist
tests/test_ace109_example_identity.pyplus one intests/test_prompt_examples_serving.py. No existing test modified or removed (git diff -- tests/is additions only), which is a stated success criterion.that it is the same one next release, so the test that matters is the one that fails when
someone changes the hash.
uv run dev.py checkgreen — ruff lint + format, 4570 passed, gitleaks.uv run dev.py cover— 100% diff coverage (18 lines).Zero must-fix on the diff. Collision risk quantified: 3.7e-12 at the ~46-example scale, 1.8e-5
at 100k.
sales/assetsareas.Spec: ACE-109