feat: 可选 TwelveLabs 视频理解 provider(Pegasus 问答 + Marengo 向量)/ opt-in TwelveLabs video understanding#12
Open
mohit-twelvelabs wants to merge 1 commit into
Conversation
- 新增 TwelveLabsVideoQAService:通过 Pegasus 直接理解视频画面进行问答, 接口与现有 VideoQAService 一致,可作为视频问答的可选 provider - 新增 Marengo 512 维多模态向量接口,用于视频片段语义检索 - 通过 VIDEO_QA_PROVIDER=twelvelabs + TWELVELABS_API_KEY 显式启用, 默认关闭,不配置时现有转录文本 + OpenAI 路径完全不受影响 - twelvelabs 作为可选依赖(uv sync --extra twelvelabs) - 补充 .env.example、README 文档与单元/实时契约测试
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hi! I'm Mohit, I work at TwelveLabs (@mohit-twelvelabs).
简介(中文)
为 ViNote 新增可选的 TwelveLabs 视频理解 provider,让「视频问答」可以直接理解视频画面(动作、场景、画面中的物体等转录文本无法表达的信息),而不仅仅依赖转录文本。
TwelveLabsVideoQAService,接口与现有VideoQAService完全一致,可无缝替换)。完全可选、非破坏性:通过
VIDEO_QA_PROVIDER=twelvelabs+TWELVELABS_API_KEY显式启用;不配置时,现有「转录文本 + OpenAI」路径完全不受任何影响。依赖以可选 extra 形式提供(uv sync --extra twelvelabs)。可在 https://twelvelabs.io 免费获取 API key,有较慷慨的免费额度。
What this adds (English)
This PR adds an opt-in TwelveLabs video-understanding provider so ViNote's video Q&A can reason over the actual video (actions, scenes, on-screen objects — things a transcript can't capture), not just the transcript text.
TwelveLabsVideoQAServicemirrors the existingVideoQAServiceinterface (answer_question_stream/is_available), so it slots in as a drop-in QA provider.Why it helps ViNote
ViNote already does transcript-based Q&A. For visually-rich videos (tutorials, demos, sports, anything where what's on screen matters), transcript-only answers miss a lot. Pegasus closes that gap, and Marengo gives a foundation for "find the clip that matches this query" semantic search.
Opt-in / non-breaking
VIDEO_QA_PROVIDER=twelvelabsandTWELVELABS_API_KEYare set.get_video_qa_service()returns the existingVideoQAServiceunchanged — zero behavior change.twelvelabsis an optional dependency ([project.optional-dependencies]→uv sync --extra twelvelabs), so existing installs are untouched.How it was tested
twelvelabs>=1.2.8):marengo3.0text embedding returns a 512-dim vector.pegasus1.5analyze(...)returns a text answer (verified via theasset_idpath on an uploaded clip).backend/tests/test_twelvelabs_service.py): no-network unit tests for config toggles, provider selection, sentence-chunking, and the empty-URL guard, plus a live Marengo 512-dim contract test that auto-skips withoutTWELVELABS_API_KEY.Notes / verified gotchas baked in
video_id— this uses a public URL or an uploadedasset_id(direct upload capped at 200MB, documented in code + README).max_tokens≥ 512 (clamped in config).You can grab a free API key at https://twelvelabs.io — there's a generous free tier.