feat(v2): Phase 3 — LanceDB index - #228
Merged
Merged
Conversation
…erge-insert upsert
…n-scalar guard; strengthen evolution test
…; remove dead identifier-char set
Single list_tables() call only returns first page; iterate until page_token is None to avoid missing tables beyond the first page in ensure_table, drop_table and search existence checks.
- table_columns: add ORDER BY version ASC so earliest-version dtype wins deterministically (prevents dtype divergence between successive table_columns calls on the same schema). - rebuild_schema_index: add V2Record.id as tiebreaker to the OFFSET pagination sort key so rows with equal inserted_at cannot be skipped across page boundaries.
…ild; add response_model - RecordFilter: pydantic model_validator rejects op='in' with a scalar value (str/non-iterable) raising ValueError → 422 instead of an unmapped TypeError → 500 - LanceIndexEngine.upsert: add optimize=False kwarg; new optimize_table() method - rebuild_schema_index: upsert all batches with optimize=False, call optimize_table() once at the end — avoids O(batches) FTS rebuilds during a full reindex - rebuild_schema_index endpoint: add response_model=dict[str,int] and timeout advisory docstring - Test: assert op='in' + scalar → 422
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
JonnyTran
added a commit
that referenced
this pull request
Jul 21, 2026
…ests Addresses roborev job #228 (auto-review of 2a1ac44). Medium — the table score exemption was total, accepting arbitrary list scores. That contradicts its own rationale (a table suggestion's score is whole-suggestion confidence) and is not inert: the fan-out repeats the whole list onto every cell, surfacing a multi-value score no consumer can read. Narrowed to `score is None or isinstance(score, (int, float))`; a per-row score is a distinct future feature needing indexed fan-out. Flipped the two list-accepting tests to assert rejection; mutation-verified they fail without the guard. Low — the two non-table negative tests paired type=multi_label_selection with LABEL_SETTINGS (settings type label_selection), a combination the API cannot persist that only worked because both settings models expose `.options`. Added "no" to MULTI_LABEL_SETTINGS and use it, so the type/settings pair matches what QuestionSettingsValidator enforces. Low — test_non_ascii_names_and_bindings_resolve keyed its response envelope on ASCII "pays", so the responses-side ensure_ascii=False was untested. Renamed the scalar question to "país" and key the envelope with it, exercising the response-envelope join (rc.question_name = q.question_name) against a non-ASCII key. Server 138 passed; ruff clean.
JonnyTran
added a commit
that referenced
this pull request
Jul 22, 2026
…#233) * docs(spec): extraction table design — workspace projection, Perspective grid, review-page retirement * docs(spec): add Perspective/Vue3/Nuxt4 integration refs + single-row table-value constraint * plan * docs(plan): update extraction table plan with DuckDB integration and architectural details - Enhanced the architecture section to detail the use of DuckDB for in-memory denormalization, replacing previous methods. - Clarified the tech stack to include DuckDB as a denormalization engine. - Updated Phase 1 description to specify the addition of DuckDB as a new dependency. - Added new integration tests for the workspace projection functionality. * feat(server): accept list[dict] table values additively (spec §3.4) * feat(server): enrich ProjectionCell with record_id/agent/score provenance * feat(server): workspace projection denormalized via in-process DuckDB * fix(server): escape JSON paths in workspace projection denormalization * fix(server): join sub-column bindings instead of interpolating JSON paths * feat(server): GET /api/v2/projection workspace endpoint * test(server): pin projection authz and query-bounds validation * feat(v2-ui): workspace projection contract, domain types, repository method * feat(v2-ui): pure grid adapter — flat rows, banding, click-URL contract * test(v2-ui): pin server column/cell ordering in getWorkspaceProjection * test(v2-ui): pin grid-adapter key order and encode annotation URL schema id - toPerspectiveData: add a reverse-alphabetical fixture asserting Object.keys() equals [reference, ...manifest order] so a stray alphabetical sort regresses the test, not just the comment. - Cover empty-rows for toPerspectiveData/bandParity, a cells key absent from the manifest, and a reference recurring after a gap ([0,1,0]). - buildAnnotationUrl: encodeURIComponent the schemaId path segment too (not just the reference), plus tests for &/# and a slashed schema id. - Correct the toPerspectiveData comment: the integer-like-key hazard is avoided today because column names always contain a dot, not because of anything the function does — and note Phase 2 should pass projection.columns as Perspective's explicit column config rather than rely on inferred key order. * feat(v2-ui): workspace-projection use-case, extractions storage, DI wiring * test(v2-ui): pin manifest aggregation and zero-progress guard Pins the two hazards the paging fix relies on but that were previously unasserted: (1) columns are captured once from the first page, in server order, never sorted or duplicated across pages; (2) an empty page with a huge totalReferences stops after one call instead of hammering the endpoint. Also captures totalReferences from the first page only (matching columns), and documents the server spine-CTE invariant the zero-progress guard depends on. * fix(v2): exempt table values from suggestion score cardinality; close review gaps Addresses medium-severity roborev findings on the Phase 1 branch. Behavior fix: - V2SuggestionValidator._validate_score now takes the question type and skips the list-cardinality rules for table questions. Those rules assume a list value is a list of answer choices (multi_label_selection, ranking); a table value's list is N rows (spec 3.4), and a suggestion's score applies to the suggestion as a whole. Before this, a multi-row table suggestion with one confidence score -- the exact shape Task 3's fan-out consumes -- was rejected by upsert_suggestion. Hardening: - Serialize DuckDB inputs with ensure_ascii=False. The projection joins Python strings against keys DuckDB parses from JSON text; emitting non-ASCII names as \uXXXX made that join depend on DuckDB decoding them back. It does decode them correctly today, so this removes a dependency rather than fixing a live break. Test gaps closed: - table suggestions: single score over multi-row value, mismatched score list, and score list over a bare dict; plus non-table types still enforcing cardinality - multi-question response envelope: pins the positional key/value zip-unnest that every other test exercised with only one key - non-ASCII schema/question/sub-column names resolve end to end - response-over-suggestion cell: seed agent/score on the losing suggestion so the `is None` assertions prove suppression instead of passing vacuously - repository fixture: suggestion-sourced table sub-column with agent/score and a list-valued score, so a transposition or dropped sub_column now fails - use-case: first-page totalReferences capture (growing total mid-sweep) and the empty-workspace path; manifest test pages now return distinct manifests Server 138 passed; frontend 871 passed, typecheck/lint/gen:api drift all clean. * chore: update VSCode settings for cursorpyright analysis paths and clean up SQL denormalization logic - Added extraPaths for cursorpyright analysis to match Python analysis settings. - Simplified the SQL denormalization logic by replacing the detailed SQL statement with a placeholder, indicating that the exact SQL is to be defined by the implementer. * fix(server): narrow table score exemption to scalar; tighten review tests Addresses roborev job #228 (auto-review of 2a1ac44). Medium — the table score exemption was total, accepting arbitrary list scores. That contradicts its own rationale (a table suggestion's score is whole-suggestion confidence) and is not inert: the fan-out repeats the whole list onto every cell, surfacing a multi-value score no consumer can read. Narrowed to `score is None or isinstance(score, (int, float))`; a per-row score is a distinct future feature needing indexed fan-out. Flipped the two list-accepting tests to assert rejection; mutation-verified they fail without the guard. Low — the two non-table negative tests paired type=multi_label_selection with LABEL_SETTINGS (settings type label_selection), a combination the API cannot persist that only worked because both settings models expose `.options`. Added "no" to MULTI_LABEL_SETTINGS and use it, so the type/settings pair matches what QuestionSettingsValidator enforces. Low — test_non_ascii_names_and_bindings_resolve keyed its response envelope on ASCII "pays", so the responses-side ensure_ascii=False was untested. Renamed the scalar question to "país" and key the envelope with it, exercising the response-envelope join (rc.question_name = q.question_name) against a non-ASCII key. Server 138 passed; ruff clean. * fix(v2-ui): regenerate OpenAPI snapshot under py3.12 to match CI (422 phrase) The v2 OpenAPI drift gate dumps the schema under CI's Python (<=3.12), where HTTPStatus(422).phrase is 'Unprocessable Entity'. The committed snapshot was generated under Python 3.13, which renamed the phrase to 'Unprocessable Content', so the full-file diff -u failed across all 28 occurrences. Regenerated openapi.json + v2-api.ts under Python 3.12; no schema/type changes beyond the phrase.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 3 of 6 — LanceDB index engine (v2)
Adds a LanceDB-backed index derived from Postgres, exposing full-text (BM25) + scalar-filter search over v2 records via
POST /api/v2/schemas/{id}/records:search, kept in sync best-effort on write and rebuildable. Additive only — the v1search_engine/(ES/OpenSearch) package is untouched (branch diff is empty over that path); v1 retires in Phase 6.Spec:
docs/superpowers/specs/2026-06-27-schema-centric-data-model-design.md§15 (design) + new §16 (implementation resolutions).Plan:
docs/superpowers/plans/2026-07-07-v2-lancedb-index.md.What's included
index/package — pure Arrow/rowmapping.py(no LanceDB/DB import); v2-shapedIndexEngineABC + result/filter models (base.py);LanceIndexEngineon the async LanceDB API (lancedb_engine.py);get_index_engineDI provider mirroringget_search_engine.contexts/v2/index_sync.py) — ensure/evolve on schema publish, upsert after bulk-upsert, delete on delete. Any engine error is logged at WARNING and swallowed; the API request still succeeds. Only:rebuild-index,rebuild_schema_index, and the reindex CLI surface errors.POST /schemas/{id}/records:search— Lance supplies matchingrecord_ids + scores; payloads are hydrated from Postgres (source of truth), returned in Lance hit order, tolerant of stale-index rows.POST /schemas/{id}:rebuild-indexrecovery endpoint +extralit_server index reindex|listCLI.EXTRALIT_LANCEDB_URIsetting (defaults to{home_path}/lance).Design guarantees
schema_{id.hex}): system columns + typed columns (superset across versions viaadd_columns) + a derivedtextcolumn carrying the FTS index.similarity_search, no litellm this phase.Testing / review
search_engine+ v2indexcoexist on import;create_server_app()boots;ruff check/formatclean.ge/le+None→ always-false>= NULL;lancedb_uriannotation).🤖 Generated with Claude Code