Skip to content

A curated example keeps the same id across deploys, and the caller can see it - #233

Merged
vishalkalbi27 merged 2 commits into
mainfrom
ACE-109-stable-prompt-example-id
Aug 18, 2026
Merged

A curated example keeps the same id across deploys, and the caller can see it#233
vishalkalbi27 merged 2 commits into
mainfrom
ACE-109-stable-prompt-example-id

Conversation

@vishalkalbi27

Copy link
Copy Markdown
Collaborator

Summary

prompt_example has an id column in its primary key and it was unusable three ways over: absent an
authored id the seed minted a uuid4(), nothing wrote it back so the next deploy minted a
different one for the same example, and the read selected question, doc — never the id column —
so no id reached the caller even when one held something.

The effect is that a curated example has no durable identity. Nothing can name one, count one, or
say whether the one returned yesterday is the one returned today. Three things already work around
its absence: agami-save-correction cannot tell a correction that created a new example from one
that should have replaced an existing one; ranking returns a subset that is not recordable, so
example-set behaviour is not reproducible across deploys; and the library only grows, because no two
observations of an example can be tied together.

This derives the id from the example's own content, returns it on the served path, and adds the
lookup that turns a one-way hash back into an example.

Changes

  • example_id(ex) — a 12-character SHA-256 prefix over question + sql, each part UTF-8
    encoded and NUL-terminated, copied from snapshot.py::_hash_and_manifest (what
    compute_model_hash uses). Terminating rather than joining is what keeps ("ab", "c") and
    ("a", "bc") distinct. Byte-exact — no strip, no case-fold — matching that same precedent.
  • write_examples derives instead of minting. An authored id in the YAML still wins; the
    existing escape hatch and its comment are untouched. Derivation is the fallback for the case that
    is universal in practice.
  • write_examples dedups within the batch, first-wins. area is deliberately not hashed — it is
    not carried in the example file, the deploy injects it from the subject-area directory name — so
    two examples sharing question and sql now collapse to one id. area is not in
    PRIMARY KEY (org_id, datasource, id) either, so without the dedup the second INSERT would raise
    and abort the whole deploy. The rows are deleted before the batch, so a within-call repeat is the
    only collision reachable.
  • select_examples selects id and merges it into each returned dict, column-wins, before
    measuring against the char budget so the accounting reflects what is actually returned.
  • example_by_id(store, *, org_id, datasource, example_id) — the derivation is one-way, so an id
    is inert without a lookup and there was none. Scoped by org and datasource like every other read in
    the module; None when the id was never seeded.

No schema change, no migration, no tool-schema change. The local file path is untouched: it returns
markdown-wrapped raw YAML rather than dicts, so an id there means parsing and re-rendering YAML in a
path that has no parser today.

Two behaviour changes worth calling out

  • An example authored under two subject-area directories previously seeded two rows and was
    served for both areas. Collapsed to one id it keeps the first area, and select_examples filters
    area = ? OR area IS NULL, so the second area's scoped query no longer finds it. Writing the
    survivor into the cross-area (area IS NULL) bucket would preserve both, but would also serve it
    to areas it was never filed under. Narrowing a rare deliberate duplicate beats polluting every
    unrelated few-shot set.
  • Two examples carrying the same authored id used to raise an IntegrityError and take the
    deploy down; they now skip. That is a strict improvement, and it is noted in the comment.

Checklist

  • Tests added — 24 new tests in tests/test_ace109_example_identity.py plus one in
    tests/test_prompt_examples_serving.py. No existing test modified or removed (git diff -- tests/ is additions only), which is a stated success criterion.
  • The construction is pinned by a golden constant, not just a round-trip — the point of the id is
    that it is the same one next release, so the test that matters is the one that fails when
    someone changes the hash.
  • uv run dev.py check green — ruff lint + format, 4570 passed, gitleaks.
  • uv run dev.py cover — 100% diff coverage (18 lines).
  • Reviewed with the Agami panel (correctness/tests, security + runtime cost, structural rubric).
    Zero must-fix on the diff. Collision risk quantified: 3.7e-12 at the ~46-example scale, 1.8e-5
    at 100k.
  • Public-repo safe — no customer, dataset, or internal naming; fixtures are the repo's synthetic
    sales/assets areas.

Spec: ACE-109

…n see it

The id column was in the primary key and useless three ways over: absent an
authored id the seed minted a uuid4, nothing wrote it back so the next deploy
minted a different one for the same example, and the read never selected the
column so no id reached the caller even when one held something. So no curated
example had a durable identity — nothing could name one, count one, or tell
whether the one returned yesterday was the one returned today.

The id is now derived from the example's own content: question + sql, 12 hex
characters, the construction compute_model_hash already uses. Derived rather
than minted because a minted id changes every deploy, and derived rather than
authored because authoring gives the id two homes that can disagree. area is
excluded deliberately — it is not in the example file, the deploy injects it
from the subject-area directory name — so two examples sharing question and sql
now collapse to one id, which is correct rather than a collision. An authored
id in the YAML still wins; derivation is only the fallback.

That collapse is also why write_examples dedups: area is not in the primary
key, so the second insert of a repeated id would raise and abort the whole
deploy. The rows are deleted before the batch, so a within-call repeat is the
only collision reachable, and first wins keeps the surviving row's area stable
across re-seeds.

example_by_id ships with the derivation rather than after it. The hash is
one-way, so an id is inert without a lookup, and there was none; adding it in a
later release would leave the id write-only until that release landed.

Spec: ACE-109

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR makes prompt_example identities durable across deploys by deriving a stable id from an example’s own content, returning that id to callers, and adding a reverse lookup to resolve an id back to the stored example (ACE-109).

Changes:

  • Add deterministic example_id(ex) and use it as the fallback id in write_examples, with within-batch dedup to avoid PK collisions.
  • Update select_examples to select/return the id column (column wins over any stale id inside doc) and add example_by_id(...).
  • Add new and expanded tests to pin hash construction and ensure served examples include ids.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File Description
tests/test_prompt_examples_serving.py Adds an integration-style test asserting served examples include id and it matches the derivation.
tests/test_ace109_example_identity.py New test suite covering id derivation properties, reseeding stability, dedup behavior, and example_by_id.
packages/agami-core/src/model_store.py Implements deterministic example ids, dedup in seeding, returns ids from reads, and adds lookup by id.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread packages/agami-core/src/model_store.py Outdated
Comment on lines +413 to +415
# Keep a stable id across re-seeds when the example carries one (so per-example identity
# survives a redeploy); mint one only when absent.
ex_id = str(ex.get("id") or uuid4().hex)
# survives a redeploy); derive one from its content when absent.
ex_id = str(ex.get("id") or example_id(ex))
`ex.get("id") or example_id(ex)` reads any falsy value as "no id was authored". An
unquoted `id: 0` in YAML parses to an int, so an example that carries an id gets a
derived one instead — and a hand-numbered library starts at exactly that value.

The failure is silent. The example keeps working, under an identity its author did
not choose, and the escape hatch this seed documents does not hold for the one value
most likely to be written first.

Absent now means absent: `None`, or a key present but empty. Everything else is an
authored id, including `0`.
@vishalkalbi27
vishalkalbi27 merged commit a8398b7 into main Aug 18, 2026
8 checks passed
@vishalkalbi27
vishalkalbi27 deleted the ACE-109-stable-prompt-example-id branch August 18, 2026 11:03
@github-actions github-actions Bot locked and limited conversation to collaborators Aug 18, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants