UN-4232 [FEAT] Agent-KV API serving the table extractor only

[vishnuszipstack]

What

• Stands the Agent-KV API (POST /agent-kv/) up on a branch cut from main, routing the table extractor and nothing else.
• Ships the agent_kv Django app, its URL/settings mounts, the AGENT_KV storage type, the agent_kv_callback queue and its ide_callback tasks, the scheduler's sweep/TTL beat tasks, the internal API client, the webhook notifier, and the compose/worker wiring for all of it.
• Makes four deliberate subtractions, each revertible in one line:
  1. kv is out of EXTRACTOR_ROUTES, so a kv submit returns 400 at the serializer.
  2. celery_executor_agentic_kv is out of both compose fleets and run-worker.sh.
  3. /validate is unregistered (the view class and unstract/agent-kv-schema both stay).
  4. No codegen sandbox: not the worker, its compose service, PG_ROLE_SANDBOX, its SANDBOX_* env block, its registry/enum entries, or its coverage target.
• The table extractor runs on the CALLER's platform adapters, not operator env credentials. The submit names llm / lite_llm / x2text adapter ids and the API validates each against the Bearer key's organization, its AdapterTypes, and whether it is usable. This reverses spec D6 and is the largest functional change on the branch — see Why.
• Records which adapters a run spent on the job row and reports them in the status document, because adapter choice is now a per-request, caller-controlled, cost-bearing decision.
• Re-points the backend and e2e suites from kv to table, and inverts two guards rather than deleting them.
• Adds the browser page the keys were missing. Settings → Agent-KV API Keys, admin-only, a thin wrapper over the ApiKeyManager screen that already backs Platform API keys and Global API deployment keys. Until now the only way to obtain a key was to POST to the management endpoint by hand, so the honest answer to "how does a user get one?" was "open devtools".
• Adds the other control an API caller was missing: Copy ID on the adapter row. The submit names llm / lite_llm / x2text by instance id, and that id was in no screen — the card shows the adapter name, so the first call answered Must be a valid UUID three times with no way forward but the network tab. ResourceTable gains an opt-in handleCopyId, wired by ToolSettings.
• An empty result now says why it is empty (detected / reason, cloud side). A run that matches no page is a successful run and it bills; tables: [] with row_count: 0 previously looked identical whether detection found nothing or extraction returned nothing, and the only way to tell was which keys were missing from timing.
• The key list no longer returns plaintext keys. One serializer answered every action, so that table would have printed every key in the organization in full. list now masks to ****-<last 4> and adds created_by_email; retrieve/create/rotate carry the full value, each being a deliberate request for one key. Mirrors platform_api, which settled this first.
• Locks two cross-repo constants (the stage name and the stage-status allowlist) and corrects docs/agent-kv-api.md.

Why

Table extraction on the API is ready, but it currently rides unstract#2309 / unstract-cloud#1816 — a 338-file change set across two repos that also carries the KV extraction engine, the schema codegen path and a hardened sandbox worker. Those need time to stabilise. This PR carves out the subset customers actually need now so it can ship on its own.

The kv extractor is left in the tree, dormant and still tested, so the larger PRs re-enable it by restoring one dict entry rather than merging content back into the files they rewrite most.

Why adapters replaced env credentials (spec D6 reversed). D6 specified the blind API would have "no platform adapters by design", with models and OCR credentials env-configured and "no end-user model control". That was right while the consumer was hypothetical. It is wrong now that the consumers are existing customers with accounts: they already own adapters, and there was no operator-level LLM credential anywhere in the platform to reuse. Confirmed empirically rather than assumed — the only LLMWhisperer credential in a dev namespace is LLMWHISPERER_PORTAL_KEY, an X-PORTAL-KEY admin provisioning key (wrong type and over-privileged), and there is no operator LLM key in env or in DB adapters at all. docs/agent-kv-api.md §3a documents the contract and the reversal.

How

• Copy-forward, not cherry-pick. The source branch carries 100+ interleaved commits; the subset was reconstructed by copying files and then subtracting, verified by a full branch-vs-main file sweep.
• The load-bearing line is EXTRACTOR_ROUTES. SUPPORTED_EXTRACTORS is derived from its keys, so dropping kv is what turns a kv submit into a 400. Leaving it routable with no agentic_kv plugin deployed would return 202, dispatch to celery_executor_agentic_kv, and leave the job in DISPATCHED forever — no error at the producer, nothing in any log.
• Guards inverted, not removed:
  • test_queue_consumer_wiring.py used to assert the KV queue has a consumer; it now asserts no fleet advertises it. Mutation-checked — re-adding the queue to compose fails the guard.
  • shared/infrastructure/config/tests/test_registry.py was entirely sandbox assertions; it now asserts the ide_callback worker subscribes to agent_kv_callback and that both terminal callbacks route there.
• KVOptionsSerializer is now tested directly rather than through a submit. With kv refused at validate_name, driving those rules through SubmitSerializer would have asserted nothing while still passing.
• The adapter gate is in the view, not the serializer, and runs before anything is spent. Shape validation (three required roles, UUID-shaped) stays in the serializer and needs no database; tenancy, type, is_usable and is_available need both a DB and the Bearer key, so they live in the view — and run before the job row, the concurrency slot and the staging write, so a bad submit is a 400 that costs nothing. is_usable is the one with billing teeth: it is how billing cuts off a frictionlessly onboarded org's operator-funded trial, and the platform service does not check it, so without this gate an exhausted trial that the IDE, workflows and Prompt Studio all refuse would still run here on credentials the operator pays for.
• "Not yours" and "no such adapter" are reported identically, deliberately — distinguishing them turns the endpoint into an oracle for which adapter UUIDs exist in other organizations. Pinned from two genuinely different database states.
• The gate is defence in depth, not the only tenancy check. platform-service re-scopes every adapter lookup as WHERE id=%s and organization_id=%s, against the org of the platform key dispatch._platform_api_key(job) mints from the job's own org — so org A cannot spend org B's credential even with this gate removed. Correction to an earlier version of this description and to four code comments: they claimed the submit-time check was the only one. What the gate actually buys is a clean 400 naming the role instead of a mid-run SdkError, plus the two refusals the platform service does not make (is_usable, is_available).
• The cross-repo stage name is pinned on both sides. TABLE_STAGE_NAMES[0] here must equal STAGE_TABLE_EXTRACTION in the cloud repo's api_binding.py; StageReportView persists whatever name arrives and _status_document filters through the list here, so drift yields jobs that complete and bill normally while every status response returns an empty stages array. Asserted as a literal, not an import — the repos are separate checkouts that meet only in the merged tree.
• worker-unified.Dockerfile is reverted to main: its UID pin exists so the sandbox pod can assert runAsNonRoot with a numeric UID, and its comments cite a chart template this PR does not ship.

Can this PR break any existing features. If yes, please list possible items. If no, please explain why. (PS: Admins do not merge the PR without this section filled)

No existing feature is affected. Everything here is additive to OSS main, which has no agent_kv app today — there is no prior Agent-KV deployment to regress.

Risks that do exist, stated plainly:

• A kv submit now 400s. Intentional and the central point of the PR, but it is a visible difference from #2309. It cannot be a regression for anyone, because kv has never been available on main.

• /validate is not routed. Same reasoning — never shipped on main.

• The generated-code trust boundary is unchanged but more reachable. agentic_table executes LLM-generated Python in the executor pod on every run, exactly as the IDE table path does in production today. This PR does not add that risk, but it does widen reach from Prompt Studio users in an org to any holder of an API key. UN-4215 is the agreed fast-follow.

• Quota admission is not wired. The entitlement gate ships (402 on expired trial or inactive subscription); an org inside a valid subscription but over its allowance can still run and bill. Tracked as UN-4216 and consciously deferred for this release.

• ⚠️ DEPLOY ORDER: the cloud worker must go out BEFORE this backend. The two repos deploy independently and nothing on the wire marks the capability. This backend sends validated adapters in executor_params; an old worker has no ParamKeys.ADAPTERS, ignores the key, and calls ExtractionConfig.from_env(). Where the six AGENT_KV_* values are set it runs on operator credentials, bills the agent_kv:{provider}/{model} sentinel, and returns a 202 with a correct-looking result — while the adapters the caller named, and is paying for, were never used. Nothing in either repo's logs reveals it. The reverse order is safe by construction: the new worker hard-fails "Missing required adapters". Mitigation beyond ordering: leave global.sharedConfigs.agentKv unset in every table-serving namespace, so an old worker fails loudly instead. Documented in §12 of the API doc and at the chart group.

• The sweep and TTL cleanup have exactly one owner, and it is the cloud CronJob. Migration 0004 seeded two PgPeriodicTask rows on the premise that nothing fired these commands; that premise was false about its own branch — the cloud chart already ran them. The rows were seeded inert (pg_owned: False), and 0007 scoped its delete to that flag so a self-hosted operator who had adopted them was not stranded — on the premise that only a human ever sets it. That premise is false and the collision was live. The cloud chart's backend.pgScheduleMirror hook runs converge_pg_scheduler --periodics on every install, upgrade and rollback; that calls mirror_pg_periodic_tasks --adopt with no names, which is PgPeriodicTask.objects.all(); and validate-pg-worker-fleet.yaml refuses to render with the hook off. Deploying snapshot.415 found agent_kv_sweep at pg_owned=t, last_run 11:00:02 / next_run 11:10, firing every 10 minutes beside the CronJob's */15 — meeting at :30, with ttl-cleanup daily on one owner and hourly on the other. 0007's guard spares precisely the rows that double-fire, so 0009 repeats the delete by name alone; it has to be a new migration because 0007 is recorded as applied wherever the problem exists. Self-hosted OSS operators must schedule manage.py agent_kv_sweep and manage.py agent_kv_ttl_cleanup themselves — see §12. Without them a stranded job holds its concurrency slot until the 6 h Redis TTL and AGENT_KV_RESULT_TTL_DAYS is advisory, which is a retention failure rather than a disk-usage one. (Note a pure-OSS install cannot dispatch an Agent-KV job at all, since table needs the cloud plugin, so there is nothing to reap until an engine plugin ships.)

• The page cap now bounds pages_selected, not the document's length. Checked after page_start/page_end, so pages 1–50 of a 400-page PDF is accepted under a 100-page cap while the same document with no range is refused. Intended — the cap bounds paid work per job — but it is a behaviour change that was previously undocumented. Now in §6.1 and the env table.

• Not yet validated outside compose with real adapters. The branch is deployed to vishnu-unstract-dev at snapshot.382 (Synced/Healthy, migrations applied), but no job has run end to end on real adapters — that needs an LLM adapter, a cheaper LLM adapter and an LLMWhisperer X2TEXT adapter created in the dev org.

• Key-management response shapes changed (list masks the key and gains created_by_email; create requires a sanitised description; a duplicate name per org is now a 400 rather than the model's unique constraint surfacing as a 500). No external breakage: these endpoints are new in this PR and had no consumer before the page landed.

• The table result gains detected and reason (always present; reason is "" on a successful extraction). Additive, and the companion change is in unstract-cloud#1828 — the backend serves the stored engine output verbatim, so no OSS code change was needed. Three whole-dict assertions in the engine's own suite needed the new keys and were updated rather than loosened.

Database Migrations

Nine, all for the new agent_kv app — no existing table is touched, with one exception noted below:

• 0001_initial — AgentKVKey, AgentKVJob
• 0002_agentkvjob_extractor — adds extractor, defaulting existing rows to kv (there are none on a fresh deploy; new jobs always set it explicitly from the serializer)
• 0003_agentkvjob_cleanup_failed_at
• 0004_pg_periodic_tasks — superseded by 0007/0009; left seeding deliberately. Django records a migration as applied and never re-runs it, so rewriting this one would have cleaned nothing in the environments that actually have the rows while silently changing what a fresh install does.
• 0005_agentkvjob_extractor_choices — adds choices and drops the default, so an unknown extractor name cannot reach the column
• 0006_agentkvjob_adapters — adds adapters (JSONB, {} default). Additive; existing rows predate caller-supplied adapters and genuinely ran on env credentials, which {} honestly represents
• 0007_remove_agent_kv_periodic_tasks — deletes 0004's rows, scoped by name and by pg_owned=False. Incomplete; see 0009. Left exactly as applied: it is recorded in every environment that has the problem, so widening its predicate here would reach none of them.
• 0008_agentkvjob_files_purged — adds files_purged (bool, default=False). Records purge intent directly instead of inferring it from input_ref/result_ref, which mean "never set", "purged" and "purge attempted but the delete failed" at different times. A plain AddField with a non-volatile default is metadata-only on PG 11+ — no rewrite, no backfill scan
• 0009_drop_agent_kv_periodic_tasks — repeats 0007's delete without the ownership scope, because the chart's schedule-mirror hook adopts every row in pg_periodic_task on every sync, so 0007 spared exactly the rows that were double-firing. Touches pg_queue's shared pg_periodic_task table; the name scope is the load-bearing one now and the five dashboard_metrics periodics are untouched. Reverse is a documented no-op — 0007's reverse already restores the pair for a rollback that goes back past it, and restoring them here would put the second owner back. The self-hosted operator 0007's guard protected cannot exist at the point either migration runs: 0004 and its removals ship in one release, and an operator who creates their own rows afterwards is untouched because 0009 is already recorded applied

Env Config

New, all AGENT_KV_* and all documented in docs/agent-kv-api.md §12 plus backend/sample.env, workers/sample.env and docker/sample.env:

• Models and credentials: AGENT_KV_LLM_PROVIDER, AGENT_KV_LITE_MODEL, AGENT_KV_ADVANCED_MODEL, AGENT_KV_LLM_API_KEY, AGENT_KV_LLMWHISPERER_API_KEY, AGENT_KV_LLMWHISPERER_BASE_URL — none of these six is read on the table path, and they must be left UNSET in any namespace serving it. They are the operator's credentials, read only by the executor's ExtractionConfig.from_env(); table resolves the caller's adapters instead. Leaving them set is what makes the deploy-order skew above silent rather than loud. They remain the credential model for the dormant kv extractor.
• Limits: AGENT_KV_MAX_TOKENS, AGENT_KV_PARALLEL_PAGES, AGENT_KV_MAX_JOB_TOKENS, AGENT_KV_MAX_PAGES, AGENT_KV_MAX_FILE_SIZE_MB, AGENT_KV_MAX_SCHEMA_BYTES, AGENT_KV_MAX_TIMEOUT_SECONDS
• Storage and retention: AGENT_KV_STORAGE_DIR_PREFIX, AGENT_KV_FILE_STORAGE_CREDENTIALS, AGENT_KV_RESULT_TTL_DAYS
• Feature gates (both default off): AGENT_KV_CALCULATIONS_ENABLED, AGENT_KV_STRUCTURED_OUTPUT_ENABLED

Two operator knobs are read on the table path, independently of the credential group: AGENT_KV_STORAGE_DIR_PREFIX (must match the backend's — the executor derives the job's expected directory from it to enforce the tenant boundary on the staged input) and AGENT_KV_PARALLEL_PAGES.

The e2e lane gains three required and one optional variable, none of them product config: AGENT_KV_E2E_LLM_ADAPTER, AGENT_KV_E2E_LITE_LLM_ADAPTER, AGENT_KV_E2E_X2TEXT_ADAPTER (adapter ids in the key's org; without them every accepted-submit scenario skips rather than fails) and AGENT_KV_E2E_BAD_LLM_ADAPTER (an adapter carrying a deliberately invalid credential, for the one scenario that needs a dispatched job to end FAILED). This replaces AGENT_KV_E2E_BAD_KEY_JOB, which the table path no longer reads — an operator following the old instructions got a job that ran on a valid credential, reached completed, and failed the test's own assertion.

No SANDBOX_* variables — this PR ships no sandbox worker.

Relevant Docs

• docs/agent-kv-api.md — updated in this PR: name now documents table as the only supported value, with a note explaining why kv returns 400; the /validate section and all references removed; sections 7–13 renumbered to 6–12 with anchors and cross-links. Added since: §3a (the adapter contract — the three roles, the four 400 cases, and the D6 reversal stated as a reversal), the adapters field in the §7.2 status document, the deploy-order warning and the self-hosted scheduling requirement in §12, the corrected page-cap semantics (pages_selected, not document length), and the rewritten e2e instructions in §11.
• docs/superpowers/plans/2026-10-06-table-extractor-api-carveout.md — the plan this PR implements, included here.

Related Issues or PRs

• UN-4232 — this work
• Parent epic: UN-4044
• Companion PR (merge this one FIRST): Zipstack/unstract-cloud#1828 — this PR owns the routing and the executor_params contract the cloud operation is dispatched through.
  Note this is the merge order; the DEPLOY order is the opposite — cloud worker first (see the risks section). Merging this one first is safe because the new worker refuses a submit with no adapters rather than falling back to env.
  Correction to an earlier version of this description: it said the cloud test groups would be red by design until this lands. They are not — all three pass on the cloud PR today (118 / 162 / 698), because the shared-seams move left those plugins with no OSS dependency. Treat any red in them as a real failure.
• Carved out of #2309 (OSS) and Zipstack/unstract-cloud#1816 (cloud), both of which stay open.
• Fast-follows: UN-4215 (sandbox), UN-4216 (quota admission), UN-4213 (staging deploy)

Dependencies Versions

• Adds the OSS workspace package unstract-agent-kv-schema (new, in-repo, zero third-party dependencies) and wires it into workers/pyproject.toml.
• backend/pyproject.toml, workers/pyproject.toml and the matching uv.lock files updated accordingly.
• No new third-party dependency, and no version bump to an existing one.

Notes on Testing

Automated, all green on this branch:

• backend — agent_kv/tests → 366 passed, 1 skipped
• frontend — vitest → 627 passed (40 files), biome clean on the new files
• workers — ide_callback/tests, shared/infrastructure/config/tests, shared/tests/test_webhook_notify.py, shared/tests/test_agent_kv_client.py, tests/test_agent_kv_scheduler_tasks.py, tests/test_queue_consumer_wiring.py, tests/test_llm_bearing_ops.py → 74 passed, 1 skipped
• unstract/agent-kv-schema → 143 passed; unstract/sdk1 → 766 passed
• Migrations 0001–0009 applied against a real Postgres from a fresh database and from one pre-seeded with 0004's rows; reversed and re-applied; makemigrations --check clean
• The double-fire reproduced before it was fixed, on a real database rather than in the fakes that missed it: roll a fresh schema back to 0006, UPDATE pg_periodic_task SET pg_owned = true as the hook does, migrate to 0007 (both rows survive — the bug), then to 0009 (both gone, the five dashboard_metrics periodics intact). The in-process replay test fails if the ownership scope comes back, and nothing else does
• ruff 0.3.4 (the pinned pre-commit version) clean on check and format; the full pre-commit suite passes

Mutation-checked, because each is silent when broken. Every fix in the round-2 review pass was verified by reverting it and confirming the new test fails:

• The adapter lookup uses _base_manager, not .objects. AdapterInstance's default manager auto-filters by UserContext.get_organization(), a request-local thread-local — and /agent-kv/ is deliberately whitelisted past the tenant middleware, so that value is unset and the ambient filter became organization=None. Every valid submit would have 400'd. Thirteen passing view tests could not see it because they all mocked the manager. tests/test_adapter_scoping.py is DB-backed with the thread-local unset, which is the only arrangement that reproduces it.
• A cancel that raced a dispatch does not release the slot. _never_dispatched now reads two durable markers from the database (dispatched_at and stage, written by the first stage report) rather than a snapshot taken before mark_terminal. Both a stale-row read and a RUNNING-with-NULL-dispatched_at row are covered.
• is_usable / is_available refuse at submit — an exhausted trial or a deprecated adapter 400s before the slot and the staging write.
• "Not yours" and "no such adapter" are byte-identical, asserted from two different DB states rather than by running one configuration twice.
• Queue wiring is derived from EXTRACTOR_ROUTES, so re-enabling kv flips the expectation automatically instead of requiring a guard to be deleted. Also covers run-worker-docker.sh, which was guarded by nothing — dropping agent_kv_callback there passed every test in the tree while a docker-launched fleet drained no terminal callbacks.
• Every routed operation is in _LLM_BEARING_OPS — derived, because the two hand-maintained literals could only fail when they disagreed, never when both were wrong.
• format: "Enum:paid,unpaid" is refused. _parse_format matches the prefix case-sensitively, so the mis-cased form slipped the typo guard and validation was silently off for that key. Scoped to the two argument-taking formats, so "number: of items" stays valid free text.
• A swallowed result-backend failure is logged with its traceback — the fallback value is identical either way, so the only observable difference is the log line.

Worth verifying by hand on review:

• Settings → Agent-KV API Keys lists keys with a masked value, creates one (full key copied to the clipboard), rotates, disables and deletes

• A non-admin cannot reach that page (RequireAdmin on the route, IsOrganizationAdmin on the viewset)

• Settings → LLMs → a row's Copy ID button puts the adapter's instance UUID on the clipboard, and that value is accepted by a submit

• A table submit returns 202, polls to completed, and the result is filed under extractors.table

• GET /agent-kv/{job} returns a non-empty stages: [{"name": "table_extraction", …}] — this is the assertion that catches cross-repo constant drift from the outside

• A kv submit returns 400, not 202 — the single most important subtraction in this PR

• A table submit naming adapters from another organization is refused, and refused identically to one naming an id that does not exist

• A submit with no adapters block is a 400 naming extractors, and GET /agent-kv/{job} echoes the adapter ids the job ran on

• Cancel mid-run stops the run and releases the concurrency slot

• An .xlsx submit extracts and is capped post-OCR

• Page usage and token/cost land on the standard Audit() path

• No queue is advertised without a consumer — check against the PG fleet guard, not by eyeballing compose

Run end to end on real adapters. snapshot.415 on vishnu-unstract-dev (Synced/Healthy, all migrations applied): a table submit returned 202, completed in ~44 s with 6 rows off a 2-page rent roll, and billed 33,386 in / 1,338 out tokens at $0.144095 across 2 pages. Also verified there, none of it reachable from a test: an unauthenticated POST /agent-kv/ returns 403 rather than the frontend's HTML shell; the three adapter-gate refusals (missing roles, unknown id, X2TEXT supplied in the llm slot) each 400 with the documented message; the status document echoes adapters and carries a non-empty stages; the PageUsage row carries file_name='input.pdf'; flipping is_usable=False on the LLM adapter refuses the submit with the trial-exhausted message before any spend (flag restored afterwards); and cancel returns the documented {"success": false, "status": "cancelled"}.

That deploy is also what found the double-fire above — no unit test could, since it only exists once the chart's hook has run against a database.

Re-deployed since as snapshot.428. snapshot.426 added the keys page and the
masking split. Verified on that deployment: both page modules compile and are
served by Vite, all three key routes resolve to AgentKVKeyViewSet, and a real
key created in the dev org serialises as ****-b1a8 in list (the real value
absent from the whole payload) and in full from retrieve. The page itself was
checked in the browser by the author.

The same deploy confirmed migration 0009: the two pg_periodic_task rows that
were live at pg_owned=true before it are gone, with the five
dashboard_metrics rows untouched, so the CronJobs are the only owner again.

A Postman collection covering submit / status / result / cancel / delete and
the six documented refusals is attached to the ticket.

Still unverified by hand: an .xlsx submit, and a kv submit returning 400
(asserted in tests, and refused live during the snapshot.425 run).

Two commits here are not this branch's subject, called out so a reviewer can skip them.

[MISC] Remove the dead ListOfItems adapter card deletes a card component that nothing imports — every adapter screen renders ToolSettings, which uses ResourceTable. It was briefly its own PR (#2337, now closed) and folded in here on request. It matters to this branch because the Copy ID action was first added to that dead file and shipped unreachable; the file reads exactly like the adapter list and will mislead the next person too. Verified by building: vite build resolves every import and succeeds.

And: test_wfe_status_created_idx_planner.py (UN-3883 / #2276, untouched by this feature) backdates rows with (random() * n || ' days')::interval, which round-trips the float through its own text form — and Postgres renders a small enough double in scientific notation, which ::interval rejects: invalid input syntax for type interval: "7.418860519248227e-05 days". It failed CI twice on this PR and was the only thing red, so it is fixed here rather than left to make the PR look broken: random() * n * INTERVAL '1 day' never formats anything. Verified against a real Postgres rather than by re-running, since at ~1-in-10k a green run proves nothing — the exact value from the CI log raises with the old form and gives 00:03:12.296865 with the new one, and the midpoint and maximum still give 15 and 30 days. One SQL string in one fixture; happy to pull it onto its own branch if you would rather.

Screenshots

N/A

Checklist

I have read and understood the Contribution Guidelines.

🤖 Generated with Claude Code

[greptile-apps]

via Greptile

Confidence Score: 5/5

[Critical impact] The latest documentation correction appears safe to merge and fixes the remaining numbered finding.

Summary

This PR adds the table-only Agent-KV API, caller-selected adapters, job cleanup, terminal webhooks, and an admin page for API keys.

• Table extraction jobs use adapter IDs supplied by the API caller.
• Admins can manage Agent-KV keys and copy adapter IDs in Settings.
• Workers finish, notify, and clean up Agent-KV jobs.
• Removes an unused adapter card and updates a flaky test expression.

Previous accepted risks and deferrals:

• vishnuszipstack accepted manual maintenance scheduling for OSS/compose because the cloud deployment supplies CronJobs and pure OSS cannot dispatch this extractor.
• vishnuszipstack deferred duplicate cancellation webhooks because delivery can repeat after a worker stops before acknowledging a message. The same risk applies to other terminal webhooks.
• vishnuszipstack accepted the remaining enqueue-to-bookkeeping cancellation window; closing it needs a dispatch marker written before enqueueing.
• vishnuszipstack accepted the remaining storage-cleanup race when a result is written after cleanup has finished and its immediate deletion also fails. The current preflight check blocks ordinary callbacks for already-purged jobs.
• vishnuszipstack dismissed duplicate webhooks between mismatched intermediate branch builds because their responses cannot distinguish who owns delivery, those builds are not released versions, and receivers already need to tolerate repeats.
• Earlier deferrals around missing subscription checks, cancellation notifications, failed finalization, and discarded cancelled results have since been replaced by explicit checks, queued notifications, retries, and guarded result attachment. The page cap is now documented as a whole-document limit.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  Caller[API caller] --> Submit[Check key, subscription, adapters and limits]
  Submit --> Storage[Stage document]
  Storage --> Queue[Queue table extraction]
  Queue --> Worker[Table worker]
  Worker --> Finalize[Save result and finish job]
  Finalize --> Notify[Queue terminal webhook]
  Caller --> Read[Read status or result]

Loading

_{Reviews (52) · Last reviewed commit: "UN-4232 [FIX] Keep the /api/v1 prefix of..." · Reviewed by Greptile}

[vishnuszipstack]

Multi-agent review — unstract#2317

Reviewed with the PR-review toolkit (code quality, silent failures, test coverage, type design, comments), then every load-bearing claim re-verified by hand against the PR branch. Inline comments below; CONFIRMED = traced in code, PLAUSIBLE = suspected, not fully traced.

Cross-repo defects — green in both PRs until merged together

1. Stage status "failed" is rejected and the rejection is swallowed. OSS internal_views.py:25 accepts only {"running","done"} (400 otherwise). Cloud api_binding.py:392 posts "failed". StageReporter.report catches the 400 into a logger.warning, so every over-cap Excel job finalizes failed beside a stage permanently reading running. The cloud PR's own progress.py:42-44 documents this constraint. One-line fix on the cloud side.

2. Token metering has no backstop for the new operation. workers/executor/tasks.py:45-52 lists four _LLM_BEARING_OPS; the new op is table_extract_api, which is absent. So the elif at :161 — the guard whose purpose is logging "LLM-bearing op emitted no usage" — never fires for table extraction. Combined with flush() returning [] on a silent hasattr skip, total loss of billing rows produces zero log lines. tasks.py is not in this diff, so flagging here: add "table_extract_api" to that set.

Verified good

Multi-tenancy is clean on every path traced — _get_job filters on organization_id, mark_terminal carries it in the WHERE, and both internal views match a body-supplied org_id against the row. The concurrency acquire is genuinely atomic (one Lua script, not check-then-act). write_result's per-attempt nonce correctly fixes the duplicate-finalize race. storage.py's confirmed-clear delete contract is a well-reasoned inversion. The jsonb || stage merge avoids the read-modify-write clobber. Both limiters now fail closed, and the test pins the inversion in both directions. test_cross_repo_stage_name.py is a model cross-repo lock — literal, not import, with a named twin on the cloud side; I verified both sides match.

Test counts are real: backend agent_kv/tests is exactly 193 test functions, and the worker numbers all check out. Both deliberately-inverted guards hold mechanically — test_queue_consumer_wiring.py parses the real compose and shell files, and re-adding the queue does fail it.

Note on CI

e2e was still pending when I reviewed. One e2e assertion will fail when it runs — see the comment on test_agent_kv_e2e.py.

[athul-rs]

Approving. The page-cap blocker and the other findings are fixed, and the two deferrals (lost webhook on a finalize replay, and the header-trickle residual) are reasonable follow-ups. Please open tickets for both: enqueue the terminal webhook in the same transaction as the terminal write, and move delivery to notifications_webhook.

Two small things, neither blocking:

• Prefix guard gap. CustomAuthMiddleware matches with request.path.startswith(path) (custom_auth_middleware.py:63), but validate_agent_kv_path_prefix only refuses prefix == other or other.startswith(f"{prefix}/"). So AGENT_KV_PATH_PREFIX=ap passes, becomes /ap, and exempts all of /api/v1/.... Refusing when other.startswith(prefix) closes it without changing the whitelist entry, so the POST /agent-kv → 301 redirect you wanted to keep still works. It does reject deploy alongside deployment, which is correct under startswith matching.
• Docstring nit: the AGENT_KV_ABANDONED_SLOT_GRACE_SECONDS docstring in settings/base.py opens with "How long after DISPATCH" and then says the grace is measured from the cancel.

[github-actions]

Frontend Lint Report (Biome)

✅ All checks passed! No linting or formatting issues found.

[sonarqubecloud]

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[github-actions]

Unstract test results

Per-group results

Status	Group	Tier	Passed	Failed	Errors	Skipped	Duration (s)
⚪	e2e-agent-kv	e2e	0	0	0	1	0.1
✅	e2e-api-deployment	e2e	3	0	0	0	8.3
✅	e2e-coowners	e2e	1	0	0	0	1.4
✅	e2e-etl	e2e	1	0	0	0	12.5
✅	e2e-login	e2e	2	0	0	0	1.3
✅	e2e-prompt-studio	e2e	1	0	0	0	8.9
✅	e2e-smoke	e2e	2	0	0	0	1.4
✅	e2e-workflow	e2e	1	0	0	0	12.6
✅	frontend	unit	629	0	0	0	14.2
✅	integration-backend	integration	705	0	0	26	69.1
✅	integration-connectors	integration	1	0	0	7	10.1
✅	integration-workers	integration	164	0	0	1	56.9
❌	ui	e2e	0	1	0	0	0.0
✅	unit-agent-kv-schema	unit	143	0	0	0	1.2
✅	unit-backend	unit	1918	0	0	1	44.3
✅	unit-connectors	unit	82	0	0	0	10.6
✅	unit-core	unit	281	0	0	0	3.0
✅	unit-filesystem	unit	1	0	0	0	3.9
✅	unit-platform-service	unit	18	0	0	0	2.3
✅	unit-rig	unit	120	0	0	0	3.5
✅	unit-runner	unit	12	0	0	0	2.4
✅	unit-sdk1	unit	847	0	0	0	42.4
✅	unit-tool-sidecar	unit	8	0	0	0	1.4
✅	unit-workers	unit	1513	0	0	2	118.6
	TOTAL		6453	1	0	38	430.3

Critical paths

⚠️ Critical paths not yet covered

• workflow-execution-fan-out — Multi-file workflow execution fans out to file-processing workers and rejoins. (declared coverage: no groups declared)

✅ Covered critical paths

• auth-login — covered by e2e-login
• adapter-register-llm — covered by integration-backend
• workflow-author — covered by integration-backend
• co-owner-manage — covered by integration-backend, e2e-coowners
• workflow-create-execute — covered by e2e-workflow
• api-deployment-provision — covered by integration-backend
• api-deployment-auth — covered by integration-backend
• api-deployment-run — covered by e2e-api-deployment
• mcp-server-auth — covered by integration-backend
• mcp-platform-auth — covered by integration-backend
• platform-key-whoami — covered by integration-backend
• prompt-studio-author — covered by integration-backend
• prompt-studio-fetch-response — covered by e2e-prompt-studio
• connector-register-test — covered by integration-backend
• pipeline-etl-execute — covered by e2e-etl
• usage-aggregate-read — covered by integration-backend
• usage-token-tracking — covered by e2e-api-deployment
• callback-result-delivery — covered by e2e-api-deployment