feat(ai-twelvelabs): add TwelveLabs Pegasus video understanding adapter

.changeset/twelvelabs-adapter.md

@@ -0,0 +1,5 @@
+---
+'@tanstack/ai-twelvelabs': minor
+---
+
+Add `@tanstack/ai-twelvelabs`, a TwelveLabs provider adapter for video understanding. `twelvelabsText('pegasus1.5')` analyzes a video supplied as a URL, inline base64, or a pre-uploaded asset id and streams prompt-guided text via Pegasus through the standard `chat()` activity. Supports native streaming (`analyzeStream`), structured output (`json_schema` response format), clip windowing, and token usage. Opt-in and non-breaking.

README.md

@@ -195,6 +195,7 @@ Official adapters include:
 | [`@tanstack/ai-groq`](https://tanstack.com/ai/latest/docs/adapters/groq)             | Groq low-latency inference                                                     |
 | [`@tanstack/ai-elevenlabs`](https://tanstack.com/ai/latest/docs/adapters/elevenlabs) | ElevenLabs realtime voice, speech, transcription, music, and sound effects     |
 | [`@tanstack/ai-fal`](https://tanstack.com/ai/latest/docs/adapters/fal)               | fal.ai image, video, audio, speech, and transcription models                   |
+| [`@tanstack/ai-twelvelabs`](https://tanstack.com/ai/latest/docs/adapters/twelvelabs) | TwelveLabs Pegasus video understanding — summaries, Q&A, and structured output |
 
 The adapter system is tree-shakeable by activity. Import `openaiText` for chat,
 `openaiImage` for images, `falVideo` for video, `geminiSpeech` for TTS, and so

docs/adapters/twelvelabs.md

@@ -0,0 +1,153 @@
+---
+title: TwelveLabs
+id: twelvelabs-adapter
+order: 9
+description: "Understand video with TwelveLabs Pegasus in TanStack AI — summaries, Q&A, chapters, and structured output via the @tanstack/ai-twelvelabs adapter."
+keywords:
+  - tanstack ai
+  - twelvelabs
+  - pegasus
+  - video understanding
+  - multimodal
+  - adapter
+---
+
+The TwelveLabs adapter brings video understanding to TanStack AI. [TwelveLabs](https://twelvelabs.io) builds video-native foundation models; **Pegasus** reasons over a video and returns prompt-guided text — summaries, Q&A, chapters, and highlights. The adapter plugs Pegasus into the standard `chat()` and `summarize()` activities: include a video content part in a message, ask a question, and stream back text.
+
+## Installation
+
+```bash
+npm install @tanstack/ai-twelvelabs
+```
+
+Set `TWELVELABS_API_KEY` in your environment, or pass the key explicitly. You can grab a free API key at [twelvelabs.io](https://twelvelabs.io) — there's a generous free tier.
+
+## Basic Usage
+
+```typescript
+import { chat } from "@tanstack/ai";
+import { twelvelabsText } from "@tanstack/ai-twelvelabs";
+
+const stream = chat({
+  adapter: twelvelabsText("pegasus1.5"),
+  messages: [
+    {
+      role: "user",
+      content: [
+        { type: "text", content: "Summarize this video in one paragraph." },
+        {
+          type: "video",
+          source: { type: "url", value: "https://example.com/clip.mp4" },
+        },
+      ],
+    },
+  ],
+});
+
+for await (const chunk of stream) {
+  if (chunk.type === "TEXT_MESSAGE_CONTENT") {
+    process.stdout.write(chunk.delta);
+  }
+}
+```
+
+## Supplying the Video
+
+A video can be supplied three ways:
+
+- **URL** — `{ type: "video", source: { type: "url", value } }`. Use a direct link to a raw media file; video-hosting-platform and cloud-storage sharing links are not supported.
+- **Inline base64** — `{ type: "video", source: { type: "data", value, mimeType } }`. Max 30 MB.
+- **Pre-uploaded asset** — set `modelOptions.assetId`. It takes precedence over any inline video part in the messages.
+
+## Custom API Key
+
+```typescript
+import { chat } from "@tanstack/ai";
+import { createTwelveLabsText } from "@tanstack/ai-twelvelabs";
+
+const adapter = createTwelveLabsText("pegasus1.5", process.env.TWELVELABS_API_KEY!);
+
+const stream = chat({
+  adapter,
+  messages: [
+    {
+      role: "user",
+      content: [
+        { type: "text", content: "What happens in this clip?" },
+        { type: "video", source: { type: "url", value: "https://example.com/clip.mp4" } },
+      ],
+    },
+  ],
+});
+```
+
+## Provider Options
+
+```typescript
+import { chat } from "@tanstack/ai";
+import { twelvelabsText } from "@tanstack/ai-twelvelabs";
+
+const stream = chat({
+  adapter: twelvelabsText("pegasus1.5"),
+  messages: [
+    {
+      role: "user",
+      content: [
+        { type: "text", content: "Describe what happens." },
+        { type: "video", source: { type: "url", value: "https://example.com/clip.mp4" } },
+      ],
+    },
+  ],
+  modelOptions: {
+    temperature: 0.2,
+    maxTokens: 2048,
+    // Analyze only seconds 10–30 of the video (Pegasus 1.5).
+    startTime: 10,
+    endTime: 30,
+  },
+});
+```
+
+| Option        | Description                                                       |
+| ------------- | ----------------------------------------------------------------- |
+| `temperature` | Sampling temperature, `0`–`1`. Default `0.2`.                     |
+| `maxTokens`   | Maximum response length, in tokens.                               |
+| `startTime`   | Start of the analysis window, in seconds (Pegasus 1.5).           |
+| `endTime`     | End of the analysis window, in seconds (Pegasus 1.5).             |
+| `assetId`     | Analyze a previously uploaded TwelveLabs asset instead of inline. |
+
+## Structured Output
+
+Pegasus supports a `json_schema` response format. Pass an `outputSchema` to `chat()` and the adapter constrains the model's output to it:
+
+```typescript
+import { chat } from "@tanstack/ai";
+import { twelvelabsText } from "@tanstack/ai-twelvelabs";
+import { z } from "zod";
+
+const result = await chat({
+  adapter: twelvelabsText("pegasus1.5"),
+  messages: [
+    {
+      role: "user",
+      content: [
+        { type: "text", content: "Extract the summary and topics." },
+        { type: "video", source: { type: "url", value: "https://example.com/clip.mp4" } },
+      ],
+    },
+  ],
+  outputSchema: z.object({
+    summary: z.string(),
+    topics: z.array(z.string()),
+  }),
+});
+```
+
+## Models
+
+| Model        | Use                                                               |
+| ------------ | ----------------------------------------------------------------- |
+| `pegasus1.5` | Video understanding with clip windowing and a larger token budget |
+| `pegasus1.2` | General video understanding (legacy)                              |
+
+`TWELVELABS_EMBEDDING_MODELS` (`marengo3.0`) is also exported for reference — Marengo produces 512-dim multimodal embeddings over a shared text/image/audio/video space. TanStack AI does not yet expose an embeddings activity, so this adapter focuses on the Pegasus video-understanding path.

docs/config.json

@@ -458,6 +458,11 @@
           "to": "adapters/fal",
           "addedAt": "2026-04-15"
         },
+        {
+          "label": "TwelveLabs",
+          "to": "adapters/twelvelabs",
+          "addedAt": "2026-06-25"
+        },
         {
           "label": "OpenRouter Adapter",
           "to": "adapters/openrouter",

packages/ai-twelvelabs/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Tanner Linsley
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

packages/ai-twelvelabs/README.md

@@ -0,0 +1,120 @@
+# @tanstack/ai-twelvelabs
+
+TwelveLabs adapter for TanStack AI — video understanding with Pegasus.
+
+[TwelveLabs](https://twelvelabs.io) builds video-native foundation models.
+**Pegasus** reasons over a video and returns prompt-guided text (summaries,
+Q&A, chapters, highlights). This adapter exposes Pegasus through the standard
+TanStack AI `chat()` / `summarize()` activities: put a video content part in a
+message, ask a question, and stream back text.
+
+## Installation
+
+```bash
+npm install @tanstack/ai-twelvelabs @tanstack/ai
+```
+
+Set `TWELVELABS_API_KEY` in your environment (or pass the key explicitly). You
+can grab a free API key at [twelvelabs.io](https://twelvelabs.io) — there's a
+generous free tier.
+
+## Basic Usage
+
+```typescript
+import { chat } from '@tanstack/ai'
+import { twelvelabsText } from '@tanstack/ai-twelvelabs'
+
+const stream = chat({
+  adapter: twelvelabsText('pegasus1.5'),
+  messages: [
+    {
+      role: 'user',
+      content: [
+        { type: 'text', content: 'Summarize this video in one paragraph.' },
+        {
+          type: 'video',
+          source: { type: 'url', value: 'https://example.com/clip.mp4' },
+        },
+      ],
+    },
+  ],
+})
+
+for await (const chunk of stream) {
+  if (chunk.type === 'TEXT_MESSAGE_CONTENT') process.stdout.write(chunk.delta)
+}
+```
+
+The video can be supplied three ways:
+
+- **URL** — `{ type: 'video', source: { type: 'url', value } }` (direct link to
+  a raw media file; hosting-platform links are not supported).
+- **Inline base64** — `source: { type: 'data', value, mimeType }` (max 30 MB).
+- **Pre-uploaded asset** — `modelOptions: { assetId }`, which takes precedence
+  over any inline video part.
+
+## Custom API Key
+
+```typescript
+import { createTwelveLabsText } from '@tanstack/ai-twelvelabs'
+
+const adapter = createTwelveLabsText(
+  'pegasus1.5',
+  process.env.TWELVELABS_API_KEY!,
+)
+```
+
+## Provider Options
+
+```typescript
+import { chat } from '@tanstack/ai'
+import { twelvelabsText } from '@tanstack/ai-twelvelabs'
+
+const stream = chat({
+  adapter: twelvelabsText('pegasus1.5'),
+  messages: [
+    /* ... */
+  ],
+  modelOptions: {
+    temperature: 0.2,
+    maxTokens: 2048,
+    startTime: 10, // analyze only seconds 10–30 (Pegasus 1.5)
+    endTime: 30,
+  },
+})
+```
+
+## Structured Output
+
+Pegasus supports a `json_schema` response format. Pass a schema to `chat()` and
+the adapter constrains the output:
+
+```typescript
+import { chat } from '@tanstack/ai'
+import { twelvelabsText } from '@tanstack/ai-twelvelabs'
+import { z } from 'zod'
+
+const result = await chat({
+  adapter: twelvelabsText('pegasus1.5'),
+  messages: [
+    /* ... a video + prompt ... */
+  ],
+  outputSchema: z.object({ summary: z.string(), topics: z.array(z.string()) }),
+})
+```
+
+## Models
+
+| Model        | Use                                                             |
+| ------------ | --------------------------------------------------------------- |
+| `pegasus1.5` | Video understanding with clip windowing and larger token budget |
+| `pegasus1.2` | General video understanding (legacy)                            |
+
+`TWELVELABS_EMBEDDING_MODELS` (`marengo3.0`) is also exported for reference —
+Marengo produces 512-dim multimodal embeddings over a shared text/image/audio/video
+space. TanStack AI does not yet expose an embeddings activity; this adapter
+focuses on the Pegasus video-understanding path.
+
+## License
+
+MIT

packages/ai-twelvelabs/package.json

@@ -0,0 +1,68 @@
+{
+  "name": "@tanstack/ai-twelvelabs",
+  "version": "0.0.1",
+  "description": "TwelveLabs adapter for TanStack AI — video understanding with Pegasus (analyze/summarize) and multimodal embeddings with Marengo.",
+  "author": "Tanner Linsley",
+  "license": "MIT",
+  "homepage": "https://tanstack.com/ai",
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/TanStack/ai.git",
+    "directory": "packages/ai-twelvelabs"
+  },
+  "bugs": {
+    "url": "https://github.com/TanStack/ai/issues"
+  },
+  "funding": {
+    "type": "github",
+    "url": "https://github.com/sponsors/tannerlinsley"
+  },
+  "keywords": [
+    "ai",
+    "ai-sdk",
+    "typescript",
+    "tanstack",
+    "twelvelabs",
+    "adapter",
+    "video",
+    "video-understanding",
+    "multimodal",
+    "pegasus",
+    "marengo",
+    "embeddings"
+  ],
+  "type": "module",
+  "module": "./dist/esm/index.js",
+  "types": "./dist/esm/index.d.ts",
+  "exports": {
+    ".": {
+      "types": "./dist/esm/index.d.ts",
+      "import": "./dist/esm/index.js"
+    }
+  },
+  "files": [
+    "dist",
+    "src"
+  ],
+  "scripts": {
+    "build": "vite build",
+    "clean": "premove ./build ./dist",
+    "lint:fix": "eslint ./src --fix",
+    "test:build": "publint --strict",
+    "test:eslint": "eslint ./src",
+    "test:lib": "vitest --passWithNoTests",
+    "test:lib:dev": "pnpm test:lib --watch",
+    "test:types": "tsc"
+  },
+  "dependencies": {
+    "@tanstack/ai-utils": "workspace:*",
+    "twelvelabs-js": "^1.2.8"
+  },
+  "peerDependencies": {
+    "@tanstack/ai": "workspace:^"
+  },
+  "devDependencies": {
+    "@tanstack/ai": "workspace:*",
+    "@vitest/coverage-v8": "4.0.14"
+  }
+}