From bddb5462ff3aa9009a9869631f844d4f6ff05a30 Mon Sep 17 00:00:00 2001 From: Caspar Addyman Date: Wed, 8 Jul 2026 14:10:51 +0000 Subject: [PATCH 1/4] feat: bundle Video Annotation Viewer at /viewer, fix person-tracking contract gap, prep JOSS resubmission Modularity & integration release (v1.4.4) responding to JOSS pre-review feedback (joss-reviews#10182/#10183) on install footprint and the VideoAnnotator/VAV relationship: - FastAPI now serves Video Annotation Viewer's static build at /viewer, same-origin, zero-config (VIDEOANNOTATOR_ENABLE_VIEWER to disable). VAV remains fully independent. - New tests/contract/test_viewer_contract.py models VAV's actual Zod schemas in Python and validates VideoAnnotator's real exporter output against them. First run caught a live gap: person_tracking COCO output was missing person_id/person_label/label_confidence/ labeling_method whenever identity labeling was disabled, which VAV requires as non-optional. Fixed in person_pipeline.py with sensible fallback defaults; also corrected the pipeline's self-declared schema (image_id documented as integer, actually always a string). - Added roadmap_v1.6.0.md (plugin ecosystem + local LLM/Ollama backend) and rewrote roadmap_v1.5.0.md around the modularity spec (specs/003-modular-pipeline-architecture); archived the superseded UX wishlist rather than deleting it. - Doc sweep: README's install flow now leads with the bundled viewer; fixed several stale version pins (INSTALLATION.md still titled "v1.2.0", Docker.md, output_naming_conventions.md). - Added the interface figure to the combined JOSS paper and strengthened the AI-usage disclosure with a concrete, checkable pointer to the specs/roadmap design trail, addressing the "evidence of iterative design process" concern raised during pre-review. - Version bumped to 1.4.4; CITATION.cff was still stale at 1.4.2, also corrected. Co-Authored-By: Claude Sonnet 5 --- CITATION.cff | 4 +- README.md | 35 +- docs/README.md | 4 +- .../development/roadmap_v1.5.0_ux_wishlist.md | 496 +++++++++++++++ docs/deployment/Docker.md | 6 +- docs/development/output_naming_conventions.md | 4 +- docs/development/roadmap_v1.5.0.md | 565 ++++-------------- docs/development/roadmap_v1.6.0.md | 99 +++ docs/installation/INSTALLATION.md | 9 +- docs/usage/GETTING_STARTED.md | 14 +- paper/cover_letter.md | 33 +- paper/figure1.png | Bin 0 -> 456081 bytes paper/paper.bib | 46 ++ paper/paper.md | 55 +- paper/release_note_v1.4.2.md | 45 ++ pyproject.toml | 5 +- .../checklists/requirements.md | 51 ++ .../003-modular-pipeline-architecture/spec.md | 120 ++++ src/videoannotator/api/database.py | 9 +- src/videoannotator/api/main.py | 34 ++ src/videoannotator/cli.py | 8 + src/videoannotator/config_env.py | 5 + .../person_tracking/person_pipeline.py | 48 +- src/videoannotator/storage/base.py | 8 + src/videoannotator/storage/sqlite_backend.py | 7 + src/videoannotator/version.py | 8 +- src/videoannotator/viewer_static/404.html | 41 ++ .../viewer_static/VideoAnnotationViewer.png | Bin 0 -> 1489153 bytes .../viewer_static/apple-touch-icon.png | Bin 0 -> 33445 bytes .../assets/VideoAnnotationViewer-CS09kCW8.png | Bin 0 -> 456081 bytes .../viewer_static/assets/index-BmyNzjfh.css | 1 + .../viewer_static/assets/index-Du5mhplO.js | 145 +++++ .../assets/openface3Parser-Bf49xR9R.js | 1 + .../assets/v-a-v.icon-CddpxgV4.png | Bin 0 -> 1754753 bytes src/videoannotator/viewer_static/favicon.ico | Bin 0 -> 11737 bytes .../viewer_static/icon-16x16.png | Bin 0 -> 764 bytes .../viewer_static/icon-32x32.png | Bin 0 -> 2019 bytes .../viewer_static/icon-64x64.png | Bin 0 -> 5144 bytes src/videoannotator/viewer_static/index.html | 41 ++ .../viewer_static/placeholder.svg | 1 + src/videoannotator/viewer_static/robots.txt | 14 + tests/conftest.py | 17 +- tests/contract/test_viewer_contract.py | 210 +++++++ 43 files changed, 1656 insertions(+), 533 deletions(-) create mode 100644 docs/archive/development/roadmap_v1.5.0_ux_wishlist.md create mode 100644 docs/development/roadmap_v1.6.0.md create mode 100644 paper/figure1.png create mode 100644 paper/release_note_v1.4.2.md create mode 100644 specs/003-modular-pipeline-architecture/checklists/requirements.md create mode 100644 specs/003-modular-pipeline-architecture/spec.md create mode 100644 src/videoannotator/viewer_static/404.html create mode 100644 src/videoannotator/viewer_static/VideoAnnotationViewer.png create mode 100644 src/videoannotator/viewer_static/apple-touch-icon.png create mode 100644 src/videoannotator/viewer_static/assets/VideoAnnotationViewer-CS09kCW8.png create mode 100644 src/videoannotator/viewer_static/assets/index-BmyNzjfh.css create mode 100644 src/videoannotator/viewer_static/assets/index-Du5mhplO.js create mode 100644 src/videoannotator/viewer_static/assets/openface3Parser-Bf49xR9R.js create mode 100644 src/videoannotator/viewer_static/assets/v-a-v.icon-CddpxgV4.png create mode 100644 src/videoannotator/viewer_static/favicon.ico create mode 100644 src/videoannotator/viewer_static/icon-16x16.png create mode 100644 src/videoannotator/viewer_static/icon-32x32.png create mode 100644 src/videoannotator/viewer_static/icon-64x64.png create mode 100644 src/videoannotator/viewer_static/index.html create mode 100644 src/videoannotator/viewer_static/placeholder.svg create mode 100644 src/videoannotator/viewer_static/robots.txt create mode 100644 tests/contract/test_viewer_contract.py diff --git a/CITATION.cff b/CITATION.cff index 309ea2a..d5c2f4b 100644 --- a/CITATION.cff +++ b/CITATION.cff @@ -28,5 +28,5 @@ authors: orcid: "https://orcid.org/0000-0001-5846-3444" license: "MIT" repository-code: "https://github.com/InfantLab/VideoAnnotator" -version: "1.4.2" -date-released: "2026-03-04" +version: "1.4.4" +date-released: "2026-07-08" diff --git a/README.md b/README.md index 6ca3869..a0efa64 100644 --- a/README.md +++ b/README.md @@ -24,7 +24,7 @@ VideoAnnotator automatically analyzes videos of human interactions and extracts ## 🖥️ Complete Solution: Processing + Visualization -VideoAnnotator provides both **automated processing** and **interactive visualization**: +VideoAnnotator provides both **automated processing** and **interactive visualization** — and installing this repository gets you both, no separate setup required. ### 📹 **VideoAnnotator** (this repository) @@ -34,8 +34,9 @@ VideoAnnotator provides both **automated processing** and **interactive visualiz - REST API for integration with research workflows - Supports batch processing and custom configurations - Outputs standardized JSON data +- Bundles **[Video Annotation Viewer](https://github.com/InfantLab/video-annotation-viewer)**, served at `/viewer` -### 🌐 **[Video Annotation Viewer](https://github.com/InfantLab/video-annotation-viewer)** (paired repository) +### 🌐 **[Video Annotation Viewer](https://github.com/InfantLab/video-annotation-viewer)** (companion project, bundled here) **Interactive web-based visualization tool** @@ -43,8 +44,9 @@ VideoAnnotator provides both **automated processing** and **interactive visualiz - Synchronized video playback with annotation overlays - Timeline scrubbing with pose, face, and audio data - Export tools for further analysis +- Also fully independent: usable standalone with output from any tool that produces COCO/WebVTT/RTTM/scene-JSON files, no VideoAnnotator install required -**Complete workflow**: `Your Videos → [VideoAnnotator Processing] → Annotation Data → [Video Annotation Viewer] → Interactive Analysis` +**Complete workflow**: `Your Videos → [VideoAnnotator Processing] → Annotation Data → [Video Annotation Viewer, at /viewer] → Interactive Analysis` ## 🚀 Get Started in 60 Seconds @@ -82,19 +84,34 @@ curl -X POST "http://localhost:18011/api/v1/jobs/" \ ### 3. Visualize Results +No extra install needed — VideoAnnotator bundles [Video Annotation Viewer](https://github.com/InfantLab/video-annotation-viewer) and serves it directly: + +``` +Open http://localhost:18011/viewer +``` + +It's pre-configured to talk to this server, so your job results are one click away. Set `VIDEOANNOTATOR_ENABLE_VIEWER=false` if you'd rather not serve it. + +**🎉 That's it!** You now have both automated video processing and interactive visualization. + +
+Running Video Annotation Viewer standalone instead + +Video Annotation Viewer also works as an independent tool — useful if you want to review output from +other pipelines, or run the viewer without VideoAnnotator at all: + ```bash -# Install the companion web viewer git clone https://github.com/InfantLab/video-annotation-viewer.git cd video-annotation-viewer npm install npm run dev ``` -Note: Ensure Node and NPM are installed. On macOS with Homebrew: `brew install node` +Note: Ensure Node and NPM are installed. On macOS with Homebrew: `brew install node`. Open +http://localhost:3000 and load your files, or connect it to a VideoAnnotator server via `.env` or +the in-app Settings page — see that repo's README for details. -Open http://localhost:3000 and load your VideoAnnotator results. - -**🎉 That's it!** You now have both automated video processing and interactive visualization. +
## 🧠 AI Pipelines & Capabilities @@ -223,7 +240,7 @@ VideoAnnotator produces machine-readable outputs (primarily JSON files and API r - **Python**: Load JSON into pandas / numpy for analysis (see [examples/](examples/)) - **R / MATLAB**: Not currently supported with official helper packages, but the JSON outputs can be consumed using standard JSON readers -- **Visualization**: Use the companion [Video Annotation Viewer](https://github.com/InfantLab/video-annotation-viewer) for interactive playback + overlays +- **Visualization**: Bundled [Video Annotation Viewer](https://github.com/InfantLab/video-annotation-viewer) at `/viewer` for interactive playback + overlays; also runs standalone against output from other tools ## 🛠️ Installation Options diff --git a/docs/README.md b/docs/README.md index 6383b42..3c90157 100644 --- a/docs/README.md +++ b/docs/README.md @@ -55,7 +55,9 @@ For more copy-pasteable CLI workflows, see `usage/demo_commands.md`. - **[Roadmap Overview (Archived)](archive/development/roadmap_overview.md)** - Historical release strategy notes - [v1.4.0 Roadmap](development/roadmap_v1.4.0.md) - Roadmap for the v1.4.0 cycle -- [v1.5.0 Roadmap](development/roadmap_v1.5.0.md) - Roadmap for the v1.5.0 cycle +- [v1.5.0 Roadmap](development/roadmap_v1.5.0.md) - Modularity & integration release (JOSS resubmission response) +- [v1.6.0 Roadmap](development/roadmap_v1.6.0.md) - Plugin ecosystem & local LLM (Ollama/llama.cpp) backends +- [Modular Pipeline Architecture Spec](../specs/003-modular-pipeline-architecture/spec.md) - Full design spec underlying v1.5.0/v1.6.0 - [Examples CLI Update Plan](development/EXAMPLES_CLI_UPDATE_CHECKLIST.md) - CLI modernization checklist ## 🧪 Testing & QA diff --git a/docs/archive/development/roadmap_v1.5.0_ux_wishlist.md b/docs/archive/development/roadmap_v1.5.0_ux_wishlist.md new file mode 100644 index 0000000..4e1543f --- /dev/null +++ b/docs/archive/development/roadmap_v1.5.0_ux_wishlist.md @@ -0,0 +1,496 @@ +# Archived: Original v1.5.0 UX Wishlist (Deferred, Unscheduled) + +**Archived**: 2026-07-08 +**Reason**: The JOSS pre-review of VideoAnnotator (#10182) and Video Annotation Viewer (#10183) raised +concerns about install footprint, monolithic packaging, and evidence of active cross-project integration. +Responding to that feedback took priority over this wishlist, so the v1.5.0 and v1.6.0 slots were +reassigned to the modular-pipeline-architecture work and the FastAPI/VAV integration +(see [`roadmap_v1.5.0.md`](../../development/roadmap_v1.5.0.md)) and to plugin ecosystem + local LLM +backend support (see [`roadmap_v1.6.0.md`](../../development/roadmap_v1.6.0.md)). + +The content below is the original v1.5.0 plan, preserved as-is. None of it has been done; it has no +committed release slot. Revisit and re-prioritize into a future roadmap (v1.7.0+) once the modularity +and integration work lands. + +--- + +# 🚀 VideoAnnotator v1.5.0 Development Roadmap + +## Release Overview + +VideoAnnotator v1.5.0 is the **Feature Enhancement Release** - bringing all the wishlist items that were intentionally deferred from v1.4.0 to keep the JOSS release focused. This release adds usability improvements, advanced features, and quality enhancements that make VideoAnnotator more powerful and easier to use. + +**Target Release**: Q3 2026 (3-4 months after v1.4.0) +**Current Status**: Planning Phase +**Main Goal**: Enhanced usability and advanced capabilities +**Duration**: 12-14 weeks + +**Prerequisites**: v1.4.0 delivered JOSS publication foundation: +- ✅ JOSS paper submitted/accepted +- ✅ PyPI package published +- ✅ 4 research workflow examples +- ✅ Docker images (CPU/GPU) +- ✅ Core documentation complete +- ✅ Multi-platform testing + +--- + +## 🎯 Core Principles + +This release focuses on enhancing the user experience and adding advanced capabilities: + +- ✅ **Usability First** - Progress indicators, wizards, intuitive workflows +- ✅ **Quality & Performance** - Assessment pipelines, batch optimization +- ✅ **Integration** - FiftyOne, Label Studio, flexible exports +- ✅ **Developer Experience** - Better logging, debugging, analysis tools +- ✅ **Advanced Features** - Quality metrics, comparison tools, smart scheduling + +**Still OUT OF SCOPE** (deferred to v1.6.0+): +- ❌ Enterprise features (SSO, RBAC, multi-tenancy) +- ❌ Plugin system architecture +- ❌ Real-time streaming +- ❌ GraphQL API +- ❌ Cloud provider integration +- ❌ Microservice architecture + +--- + +## 📋 Feature Categories + +All items in v1.5.0 were deferred from v1.4.0 to maintain JOSS focus. They are organized into logical feature groups: + +### Category A: Installation & Setup +- Model auto-download with progress +- Setup wizard for first-run +- Enhanced health metrics + +### Category B: Progress & Feedback +- Real-time progress indicators +- Resource usage monitoring +- Job notifications + +### Category C: Configuration +- Interactive config wizard +- Config templates library +- YAML validation improvements + +### Category D: Logging & Debugging +- Structured logging (JSON) +- Log analysis tools +- Enhanced version info + +### Category E: Export & Integration +- FiftyOne integration +- Label Studio integration +- Custom CSV templates + +### Category F: Quality & Performance +- Quality assessment pipeline +- Batch processing optimization +- Pipeline comparison tools + +--- + +## 📋 v1.5.0 Deliverables + +### Phase 1: Installation & Setup Improvements (Weeks 1-2) + +#### 1.1 Model Auto-Download System + +**Problem**: Users must manually download large model files before first use. + +**Solution**: +- [ ] Automatic model download on first use (not during pip install) +- [ ] Progress bar with download speed and ETA +- [ ] Configurable cache directory (`VIDEOANNOTATOR_MODEL_CACHE`) +- [ ] Offline mode (use cached models only) +- [ ] Manual download script for air-gapped systems + +**Deliverables**: +- `src/videoannotator/models/downloader.py` - Auto-download logic +- Progress indicators using `tqdm` +- Offline mode flag and detection +- Documentation in `docs/installation/models.md` + +**Effort**: 16 hours + +#### 1.2 First-Run Setup Wizard + +**Problem**: New users don't know what to configure. + +**Solution**: +- [ ] `videoannotator setup` command for first-run configuration +- [ ] Interactive prompts: + - Detect GPU availability (CUDA/ROCm) + - Set storage directory + - Generate API token + - Select common pipelines to cache + - Test installation with sample video +- [ ] Non-interactive mode with flags +- [ ] Skip wizard option (for CI/automated deployments) + +**Deliverables**: +- `src/videoannotator/cli/setup.py` - Setup wizard +- ASCII-safe prompts for Windows compatibility +- Documentation in `docs/usage/setup_wizard.md` + +**Effort**: 12 hours + +#### 1.3 Enhanced Health & Version Endpoints + +**Problem**: Limited diagnostic information available via API. + +**Solution**: +- [ ] `/api/v1/system/version` endpoint +- [ ] Enhanced health endpoint with GPU memory, storage, uptime +- [ ] `videoannotator version --detailed` CLI command + +**Deliverables**: +- New version endpoint +- Enhanced health metrics +- CLI version command + +**Effort**: 12 hours + +**Checkpoint**: Installation experience improved (40 hours total) + +--- + +### Phase 2: Progress & Feedback (Weeks 3-4) + +#### 2.1 Real-Time Progress Indicators + +- [ ] Progress tracking (percentage, stage, ETA, frames processed) +- [ ] CLI progress bar using `rich` library +- [ ] API progress endpoint: `GET /api/v1/jobs/{id}/progress` +- [ ] WebSocket support for real-time updates (optional) + +**Effort**: 20 hours + +#### 2.2 Resource Usage Monitoring + +- [ ] Real-time CPU, GPU, RAM, disk I/O monitoring +- [ ] Include in health endpoint +- [ ] `videoannotator monitor` CLI command +- [ ] Configurable warning thresholds + +**Effort**: 16 hours + +#### 2.3 Job Completion Notifications + +- [ ] Pluggable notification system: + - Email (SMTP) + - Webhook (HTTP POST) + - Desktop notification (CLI mode) + - Slack integration + - Discord integration +- [ ] Per-job notification preferences +- [ ] Notification templates + +**Effort**: 20 hours + +**Checkpoint**: Progress and feedback complete (56 hours total) + +--- + +### Phase 3: Configuration & Validation (Weeks 5-6) + +#### 3.1 Interactive Config Wizard + +- [ ] `videoannotator config init --interactive` command +- [ ] Step-by-step prompts for common scenarios +- [ ] Hardware detection (GPU, memory, storage) +- [ ] Scenario selection (fast/balanced/high-quality) +- [ ] Validation and preview before saving + +**Effort**: 16 hours + +#### 3.2 Config Templates Library + +- [ ] Pre-built configuration templates: + - `templates/fast.yaml` - Quick processing (lower accuracy) + - `templates/balanced.yaml` - Default settings + - `templates/high-quality.yaml` - Best accuracy (slower) + - `templates/cpu-only.yaml` - No GPU required + - `templates/classroom.yaml` - Multi-person + audio + - `templates/clinical.yaml` - Face + emotion focus +- [ ] `videoannotator config list-templates` command +- [ ] `videoannotator config use