Skip to content

fix(gp): bless live response oracles for 9 unblessed scripts + gate future regrowth - #229

Open
mikemcdougall wants to merge 2 commits into
trunkfrom
fix/202-gp-response-oracles
Open

fix(gp): bless live response oracles for 9 unblessed scripts + gate future regrowth#229
mikemcdougall wants to merge 2 commits into
trunkfrom
fix/202-gp-response-oracles

Conversation

@mikemcdougall

Copy link
Copy Markdown
Contributor

Stacked on #228 (needed so ephemeral-server-smoke is green against a working baseline instead of the currently-broken nightly-aot pin) — the diff will shrink to just this PR's commit once #228 merges and this branch is rebased.

Problem

ephemeral-server-smoke graded no response oracle for 9 of the 25 supported honua-gp eval scripts (describe_segments[_fields], list_fields_segments[_filtered|_wildcard], insert_cursor_append_rows, update_cursor_close_status, update_cursor_delete_closed, make_table_view). An un-oracled supported script graded as a silent "unblessed" pass, so a response-parsing regression in schema introspection (Describe/ListFields) or the write-cursor round trip (InsertCursor/UpdateCursor) could sail through this lane undetected.

What I found while blessing them

Blessing the two write-cursor scripts against a fresh seed didn't just fill in a missing value — it surfaced a real bug. honua_gp.da._values_for_row / _extract_oid only recognized a feature's .attributes, but the live SDK's honua_sdk.models.QueryFeature (what Source.iter_features actually yields against a real server) exposes GeoJSON-shaped .properties / .id instead — there is no .attributes. Every live SearchCursor/UpdateCursor field read (OID included) silently degraded to None. Stub-mode CI never caught this because the eval stub's _StubFeature happens to define .attributes.

Fixed both helpers to read the real feature shape, preferring the SDK's already-resolved .id for OID@ (the seed's object-id field is the lower-case objectid, which never matched the existing OBJECTID/oid/FID key list either). Added two regression tests against the actual QueryFeature type (verified they fail against the pre-fix code, restored, verified they pass with the fix). Re-blessed against a fresh stack: update_cursor_close_status now genuinely proves an update round-trip (update_count: 1) instead of masking the bug behind a vacuous update_count: 0.

Describe/ListFields (schema fetch, a different code path) and InsertCursor (write-only) were already correct — those simply needed a schema / apply-edits fingerprint oracle.

Non-capturable case: make_table_view

MakeTableView/MakeFeatureLayer resolve to honua_gp.management._make_layer_handler, which only registers a client-side session alias — no honua-server HTTP round trip at all. There's no response to fingerprint. Documented in a new eval/UNBLESSED_ALLOWLIST.json with the reason, per the issue's acceptance criteria ("say so explicitly rather than leaving it implicit").

Anti-regrowth gate

run_eval.py now fails a live-mode supported script that has no response oracle and no UNBLESSED_ALLOWLIST.json entry, instead of passing it as "unblessed". Verified by emptying the allow-list and confirming make_table_view.py fails with the expected reason, then restoring it.

Acceptance criteria (from #202)

  • Decided per script whether a response oracle is capturable (8 capturable, 1 genuinely not, documented).
  • Blessed the capturable ones against a fresh client-compat seed and committed the oracles.
  • Made the remaining unblessed script explicit and non-silent via UNBLESSED_ALLOWLIST.json + a gate that fails a new/unblessed supported script.

Test plan

  • python3 -m pytest packages/honua-gp/tests -q: 197 passed (added regression tests confirmed to fail pre-fix, pass post-fix).
  • python3 -m honua_gp._cli matrix --check (no drift).
  • Stub-mode run_eval.py: 50/50.
  • Live-mode run_eval.py (HONUA_GP_EVAL_USE_STUB=0) against three independent fresh seeded stacks: 50/50 each, confirming the blessed oracles are deterministic.
  • Confirmed the anti-regrowth gate fails make_table_view.py when its allow-list entry is removed, with the documented reason.

Closes #202

…seed revision

The gp-eval live smoke job defaulted to the moving `nightly-aot` tag and
checked out the honua-server seed fixtures with no `ref:` (default-branch
HEAD at run time), so a commit's verdict here depended on whatever
honua-server merged that day and could flip red/green across re-runs with
no change in this repo.

Pin both halves as one identity, reusing the already-reviewed baseline
conformance.yml pins (honua-server@sha256:78e3088... / f897700...) instead
of promoting a second independently-moving baseline. Confirmed live: today's
nightly-aot (sha256:cd12f3c...) actually 500s against the static seed SQL --
honua-server added a migration-owned-table journal guard
(df25cba60, 2026-09-02) the seed does not satisfy -- while the pinned
2026-08-19 baseline starts cleanly and the full eval suite passes 50/50
against it (docs/golden-eval.md walks the exact repro).

Closes #201
…uture regrowth

Live smoke graded no response oracle for 9 of 25 supported honua-gp eval
scripts (Describe x2, ListFields x3, InsertCursor, UpdateCursor x2), so a
response-parsing regression in schema introspection or write-cursor round
trips would pass the lane silently as an "unblessed" pass.

Blessing the two write-cursor scripts against a fresh seed surfaced a real
bug rather than a missing oracle: honua_gp.da._values_for_row/_extract_oid
only recognized a feature's .attributes, but the live SDK's
honua_sdk.models.QueryFeature exposes GeoJSON-shaped .properties/.id instead
-- SearchCursor/UpdateCursor silently returned None for every field (OID
included) against a real server. Stub-mode CI never caught this because the
eval stub's _StubFeature happens to have .attributes. Fixed both helpers to
read the real feature shape (preferring the SDK's already-resolved .id for
OID@, since the seed's object-id field is lower-case objectid and never
matched the existing OBJECTID/oid/FID key list), added regression tests
against the actual QueryFeature type, and re-blessed against a fresh stack
-- update_cursor_close_status now genuinely proves an update round-trip
(update_count: 1) instead of masking the bug behind a vacuous zero.

Describe/ListFields (schema fetch) and InsertCursor were already correct;
those got a schema/apply-edits fingerprint oracle. MakeTableView makes no
server round trip at all (a client-side session alias registration) -- no
response to fingerprint, so it goes in a new UNBLESSED_ALLOWLIST.json with
the reason instead of a silent pass.

run_eval.py now fails a live-mode supported script that has no response
oracle and no UNBLESSED_ALLOWLIST.json entry, so this set cannot regrow
unnoticed (verified: an emptied allow-list correctly fails make_table_view.py
with the new reason).

Closes #202
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for security reviews. Please try again later.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Live smoke grades no response oracle for 9 of 25 supported eval scripts (unblessed passes silently)

1 participant