feat(audience): add locate / line / transcription qualification examples#578
Open
RapidPoseidon wants to merge 1 commit intomainfrom
Open
feat(audience): add locate / line / transcription qualification examples#578RapidPoseidon wants to merge 1 commit intomainfrom
RapidPoseidon wants to merge 1 commit intomainfrom
Conversation
Match the dashboard's audience-creation flow — `Audience` now exposes a typed helper for each of the new audience example types backed by rapidata-backend#3992 and rapidata-backend#4051. Public API on `RapidataAudience`: - `add_locate_example(instruction, truth: list[Box], datapoint, ...)` — labelers click inside the correct region(s) on an image. Random baseline is the union-area of `truth`. - `add_line_example(instruction, datapoint, truth: list[Box] | None, ...)` — labelers draw a line. When `truth` is omitted the example is scored by annotator consensus (empty `LineExampleTruth` on the wire); when provided the line points are graded against the union of the boxes exactly like Locate. - `add_transcription_example(instruction, sentence, truth: list[int], datapoint, ...)` — labelers pick the correct words out of a spoken transcription. `truth` is a list of word indices (0-based). All three forward to the existing `AudienceExampleHandler`, mirror the existing `add_classification_example` / `add_compare_example` signature shape (same context / media_context / explanation / settings keyword args), and run through `_try_start_recruiting` once at least one example exists. Refactors: - Extracted `RapidsManager._calculate_boxes_coverage` to a module-level `calculate_boxes_coverage` in `validation.rapids.box` so the audience example handler can compute the same union-area random baseline that the rapid layer does. The original method on `RapidsManager` is kept as a thin alias for backwards compatibility. - Regenerated openapi schemas + clients against prod so the new optional fields on `IExampleTruthLineExampleTruth` (boundingBoxes, requiredPrecision, requiredCompleteness) are typed correctly. The generated model file was patched by hand because the local generator-cli run can't materialise Java; the next CI regen will produce the same code. Docs: - `docs/audiences.md`: added a method-pick-table covering all five example types and a runnable snippet showing Locate / Line / Transcription. Updated the settings note to reflect that all `add_*_example` helpers accept `settings`. Tests: - pyright on `rapidata_client` reports no new errors over the main baseline (the 3 remaining errors are pre-existing missing-import warnings for pandas / pydantic at lint time). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Co-Authored-By: manuel <manuel@rapidata.ai>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The Rapidata dashboard now offers Locate / Line / Transcription audience types alongside the existing Classify and Compare (app-frontend#2342, with the LineExampleTruth multi-box backend support landing in rapidata-backend#3992 + rapidata-backend#4051). The Python SDK only had typed helpers for Classify and Compare — adding the missing three so the SDK matches the dashboard's surface and the dashboard's "Create it with Python" snippet on those forms can point at a real public method.
What's new
Three typed methods on
RapidataAudience, each calling through to a matching method onAudienceExampleHandler:All three:
add_classification_example/add_compare_example(samecontext,media_context,explanation,settings),randomCorrectProbability(calculate_boxes_coverage(truth)for Locate, same for Line when boxes are given,0.1flat for Line when not,len(correct) / len(total)for Transcription),_try_start_recruitingafter the example is added.Refactors / housekeeping
validation.rapids.box: extractedcalculate_boxes_coveragefrom the privateRapidsManager._calculate_boxes_coverageso both the rapid and audience layers can reuse the union-area sweep-line. The old method onRapidsManageris kept as a thin alias so any external callers (and any docs that reference it) keep working.IExampleTruthLineExampleTruthcodegen now carriesboundingBoxes/requiredPrecision/requiredCompleteness(matching prod after rapidata-backend#4051). The local generator-cli can't materialise Java in this dev environment, so the model file was hand-patched in the same shape the generator would produce; the next CI regen will overwrite it idempotently.bash openapi/generate-schema.sh.Docs
docs/audiences.md: added a method-picker table for all five example types and a runnable snippet showing the three new helpers. Updated the existing 'settings' note to reflect that alladd_*_examplehelpers acceptsettings, not just classify/compare.Test plan
pyright src/rapidata/rapidata_client— no new errors above main's baseline (the 3 remaining errors are pre-existing missing-import warnings forpandas/pydanticin this isolated environment).audience.add_locate_example(...)against a staging audience, verify the resulting rapid carries aLocateBoxTruthwith the expected boxes andrequiredPrecision/requiredCompletenesspropagated.audience.add_line_example(..., truth=None)andaudience.add_line_example(..., truth=[Box(...)])and verify both round-trip — the first produces an emptyLineTruth, the second aBoundingBoxTruth(per the rapidata-backend#4051 mapping).audience.add_transcription_example(...)and verifycorrectWordsreach the rapid asTranscriptionTruth.correctWords.🔗 Session: session-2de1ad62