Skip to content

feat: 可选 TwelveLabs 视频理解 provider(Pegasus 问答 + Marengo 向量)/ opt-in TwelveLabs video understanding#12

Open
mohit-twelvelabs wants to merge 1 commit into
zrt-ai-lab:mainfrom
mohit-twelvelabs:feat/twelvelabs-integration
Open

feat: 可选 TwelveLabs 视频理解 provider(Pegasus 问答 + Marengo 向量)/ opt-in TwelveLabs video understanding#12
mohit-twelvelabs wants to merge 1 commit into
zrt-ai-lab:mainfrom
mohit-twelvelabs:feat/twelvelabs-integration

Conversation

@mohit-twelvelabs

Copy link
Copy Markdown

Hi! I'm Mohit, I work at TwelveLabs (@mohit-twelvelabs).

简介(中文)

为 ViNote 新增可选TwelveLabs 视频理解 provider,让「视频问答」可以直接理解视频画面(动作、场景、画面中的物体等转录文本无法表达的信息),而不仅仅依赖转录文本。

  • Pegasus —— 直接对视频内容进行问答(TwelveLabsVideoQAService,接口与现有 VideoQAService 完全一致,可无缝替换)。
  • Marengo —— 生成 512 维多模态向量,用于对抽取出的视频片段做语义检索。

完全可选、非破坏性:通过 VIDEO_QA_PROVIDER=twelvelabs + TWELVELABS_API_KEY 显式启用;不配置时,现有「转录文本 + OpenAI」路径完全不受任何影响。依赖以可选 extra 形式提供(uv sync --extra twelvelabs)。

可在 https://twelvelabs.io 免费获取 API key,有较慷慨的免费额度。


What this adds (English)

This PR adds an opt-in TwelveLabs video-understanding provider so ViNote's video Q&A can reason over the actual video (actions, scenes, on-screen objects — things a transcript can't capture), not just the transcript text.

  • Pegasus — answers questions directly over the video. TwelveLabsVideoQAService mirrors the existing VideoQAService interface (answer_question_stream / is_available), so it slots in as a drop-in QA provider.
  • Marengo — produces 512-dim multimodal embeddings for semantic search over extracted clips.

Why it helps ViNote

ViNote already does transcript-based Q&A. For visually-rich videos (tutorials, demos, sports, anything where what's on screen matters), transcript-only answers miss a lot. Pegasus closes that gap, and Marengo gives a foundation for "find the clip that matches this query" semantic search.

Opt-in / non-breaking

  • Disabled by default. Only active when VIDEO_QA_PROVIDER=twelvelabs and TWELVELABS_API_KEY are set.
  • With no key configured, get_video_qa_service() returns the existing VideoQAService unchanged — zero behavior change.
  • twelvelabs is an optional dependency ([project.optional-dependencies]uv sync --extra twelvelabs), so existing installs are untouched.

How it was tested

  • Live-verified the TwelveLabs contracts against the real API with the official SDK (twelvelabs>=1.2.8):
    • Marengo marengo3.0 text embedding returns a 512-dim vector.
    • Pegasus pegasus1.5 analyze(...) returns a text answer (verified via the asset_id path on an uploaded clip).
  • 5 pytest tests pass (backend/tests/test_twelvelabs_service.py): no-network unit tests for config toggles, provider selection, sentence-chunking, and the empty-URL guard, plus a live Marengo 512-dim contract test that auto-skips without TWELVELABS_API_KEY.

Notes / verified gotchas baked in

  • Pegasus 1.5 doesn't accept a bare video_id — this uses a public URL or an uploaded asset_id (direct upload capped at 200MB, documented in code + README).
  • Pegasus requires the analyzed window ≥ 4s, and max_tokens ≥ 512 (clamped in config).

You can grab a free API key at https://twelvelabs.io — there's a generous free tier.

- 新增 TwelveLabsVideoQAService:通过 Pegasus 直接理解视频画面进行问答,
  接口与现有 VideoQAService 一致,可作为视频问答的可选 provider
- 新增 Marengo 512 维多模态向量接口,用于视频片段语义检索
- 通过 VIDEO_QA_PROVIDER=twelvelabs + TWELVELABS_API_KEY 显式启用,
  默认关闭,不配置时现有转录文本 + OpenAI 路径完全不受影响
- twelvelabs 作为可选依赖(uv sync --extra twelvelabs)
- 补充 .env.example、README 文档与单元/实时契约测试
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant