- [2026-08-28] 🎬 Upgraded the multimodal editing workspace with reusable content-search results, person/speaker workflows, voiceprint-based clipping, and an editable fine-cut timeline; also streamlined CPU/GPU deployment.
- [2026-08-20] 🎯 Added Face-Matched Editing and Topic-Based Editing, enabling target-person retrieval and topic-driven clip extraction.
- [2026-08-12] 🚀 Optimized the timeline display and added support for switching to vertical creation mode.
- [2026-07-20] ✨ Added the Highlight Editing Agent for automatic highlight clip extraction.
- [2026-07-10] 🎉 ClipTalk is now open-source!
ClipTalk is actively evolving toward a more powerful conversational video editing experience.
- 💬 Conversational Editing Infrastructure
- ⚡ Highlight Editing Agent
- 👤 Face-Matched Editing — Find a target person and extract the segments where they appear on screen.
- 🔊 Voiceprint-Based Editing — Identify a target speaker and extract the segments where they are speaking.
- 🧭 Topic-Based Editing — Understand the content and extract segments around a specific topic.
- 🎞️ Editable Timeline — Further edit and fine-tune AI-generated results directly on the timeline.
Real editing tasks completed by ClipTalk with a single instruction.
📰 News & Broadcast Highlights |
🧵 DIY & Craft Tutorials |
📦 Product Demos |
🏢 Lifestyle Vlogs |
🔥 Live PK Reaction Highlights |
⚽ Sports Highlights |
ClipTalk is an AI video-editing agent. You don't drag clips on a timeline or scrub through hours of footage — you just describe what you want in natural language, and the agent handles the entire process:
understanding the footage → locating the target content → planning the edit → executing cuts → delivering the final clips.
“Make a highlight from the best moments.” — ClipTalk analyzes the footage, presents the event timeline, and delivers multiple AI-edited versions.
Upload a 1-hour video, describe the moments you want, and let ClipTalk understand the footage, plan the edit, and deliver the finished clips.
Edit videos simply by describing what you want.
From “make a highlight of the best moments” to “cut out the part where they introduce Product X”, ClipTalk turns natural-language instructions directly into editing actions.
You can also refine the result through follow-up instructions such as “make it shorter” or “start from the part about pricing”.
-
⚡ Highlight Extraction — Automatically identify and extract the most valuable moments from long-form footage to create concise highlight clips.
-
👤 Face-Matched Editing — Find a target person in the video and extract the segments where that person appears on screen.
-
🧭 Topic-Based Editing — Locate and extract clips around a specific topic, such as “cut out the parts where they explain Product X.”
-
🔊 Voiceprint-Based Editing — Identify a specific speaker by voiceprint and extract the segments where that person is speaking.
Requires Linux x86_64 (or WSL2), Python 3.10–3.11, FFmpeg/ffprobe, curl, and a CJK font. On Debian/Ubuntu:
sudo apt-get update
sudo apt-get install -y ffmpeg curl fonts-wqy-zenheiInstall the complete CPU runtime and start ClipTalk:
git clone https://github.com/GML-MMGroup/ClipTalk.git
cd ClipTalk
python3 -m venv .venv
source .venv/bin/activate
python3 -m pip install --upgrade pip
python3 -m pip install -r requirements-cpu.txt
# Recommended before the first person/content recognition task
python3 tools/prepare_recognition_models.py --data-root data
python3 tools/doctor.py --profile cpu
./start.shFor an NVIDIA host with a compatible driver, install requirements-gpu.txt instead of requirements-cpu.txt, then run python3 tools/doctor.py --profile cuda.
- VLM · Required — understands frames, finds events, and refines shot boundaries.
- LLM · Optional — plans shot selection, ordering, and alternative edits; it can reuse the VLM.
- Local multimodal stack · Included — SenseVoice, OCR, visual/audio embeddings, and anonymous-person recognition; model weights are prepared locally or downloaded on first use.
Open the URL printed in the terminal. In Settings, configure the VLM and optionally a separate LLM, then upload a video and describe the edit you want.
git clone https://github.com/GML-MMGroup/ClipTalk.git
cd ClipTalk
cp .env.example .env
docker compose up --buildOpen the URL printed in the terminal, then configure the vision and planning models in Settings.
- NVIDIA GPU: install
requirements-gpu.txtfor a native environment. For Docker, usedocker compose -f docker-compose.yml -f docker-compose.gpu.yml up --buildwith NVIDIA Container Toolkit installed. - Local models: SenseVoice weights download on first use; run
python3 tools/prepare_recognition_models.py --data-root datato prepare person/content recognition assets. - Environment variables: see
.env.example. Never commit.envor API keys.
Remote deployment and security
For remote access, set HIGHLIGHT_HOST=0.0.0.0 for a native install or CLIPTALK_BIND_ADDRESS=0.0.0.0 for Docker. Configure a strong HIGHLIGHT_ACCESS_TOKEN, allow the selected port, and use an authenticated HTTPS reverse proxy for internet-facing deployments.
Pull Requests are always welcome. Contribute code, new features, bug fixes, or other improvements to AdCraft and become a project contributor.
For questions, feedback, collaboration, or other inquiries, feel free to contact us:
Ma Fei — mafei@gml.ac.cn Xu Hongbo — xuhongbo@gml.ac.cn
⭐ If you find ClipTalk useful, please give us a star!
Made with ❤️ by GML-MMGroup