feat: add webcam background effect (blur, color, image)#438
feat: add webcam background effect (blur, color, image)#438praburajasekaran wants to merge 1 commit intowebadderallorg:mainfrom
Conversation
Wraps the webcam MediaStream with a canvas-based pipeline that uses MediaPipe Selfie Segmentation to apply a background blur, solid color, or replacement image. Effect applies to both the live preview and the recorded -webcam.webm sidecar. UI lives in the camera dropdown with a mode selector, intensity slider for blur, color picker for solid color, and image upload for image mode. Choice is persisted in localStorage. MediaPipe model and WASM runtime are bundled under public/mediapipe so it works offline.
|
|
2 similar comments
|
|
|
|
📝 WalkthroughWalkthroughThis PR introduces a configurable webcam background effects system using MediaPipe segmentation. It adds a canvas-based processor supporting blur, solid color, and image backgrounds; a state-management hook with localStorage persistence; UI controls in LaunchWindow; integration with the screen recorder; and localized UI labels across eight languages. ChangesWebcam Background Effects
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Possibly related PRs
Suggested labels
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Tip 💬 Introducing Slack Agent: The best way for teams to turn conversations into code.Slack Agent is built on CodeRabbit's deep understanding of your code, so your team can collaborate across the entire SDLC without losing context.
Built for teams:
One agent for your entire SDLC. Right inside Slack. Comment |
|
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (1)
src/lib/camera/backgroundEffect.ts (1)
251-279: 💤 Low valueModule-level scratch canvas is shared across instances.
personScratchis a module-level singleton that would be shared if multiplecreateProcessedStreampipelines run concurrently (e.g., preview and recording). While RAF callbacks are serialized within a single thread, having instance-local scratch canvases would be more defensive.In the current usage pattern (preview stops before recording), this is unlikely to cause issues, so flagging as optional.
♻️ Optional: Move scratch canvas into Internals state
type Internals = { cancelled: boolean; rafHandle: number | null; video: HTMLVideoElement; canvas: HTMLCanvasElement; ctx: CanvasRenderingContext2D; maskCanvas: HTMLCanvasElement; maskCtx: CanvasRenderingContext2D; bgImage: HTMLImageElement | null; bgImageUrl: string | null; + personScratch: HTMLCanvasElement; + personScratchCtx: CanvasRenderingContext2D; };Then initialize per-instance in
createProcessedStreamand pass todrawPersonOnTop.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/lib/camera/backgroundEffect.ts` around lines 251 - 279, The module-level scratch canvas (personScratch/personScratchCtx) is shared across instances and should be made instance-local to avoid cross-pipeline interference; move initialization out of the module scope into the per-stream state created by createProcessedStream, pass the instance-specific canvas/context into drawPersonOnTop (replace references to module globals inside drawPersonOnTop with the passed-in canvas/context), and ensure sizing/clearing logic remains the same but operates on the per-instance scratch canvas.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@src/lib/camera/backgroundEffect.ts`:
- Around line 251-279: The module-level scratch canvas
(personScratch/personScratchCtx) is shared across instances and should be made
instance-local to avoid cross-pipeline interference; move initialization out of
the module scope into the per-stream state created by createProcessedStream,
pass the instance-specific canvas/context into drawPersonOnTop (replace
references to module globals inside drawPersonOnTop with the passed-in
canvas/context), and ensure sizing/clearing logic remains the same but operates
on the per-instance scratch canvas.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: fd145452-bf08-4f46-869d-247be90eeae9
⛔ Files ignored due to path filters (3)
package-lock.jsonis excluded by!**/package-lock.jsonpublic/mediapipe/wasm/vision_wasm_internal.wasmis excluded by!**/*.wasmpublic/mediapipe/wasm/vision_wasm_nosimd_internal.wasmis excluded by!**/*.wasm
📒 Files selected for processing (16)
package.jsonpublic/mediapipe/selfie_segmenter.tflitepublic/mediapipe/wasm/vision_wasm_internal.jspublic/mediapipe/wasm/vision_wasm_nosimd_internal.jssrc/components/launch/LaunchWindow.tsxsrc/hooks/useBackgroundEffect.tssrc/hooks/useScreenRecorder.tssrc/i18n/locales/en/launch.jsonsrc/i18n/locales/es/launch.jsonsrc/i18n/locales/fr/launch.jsonsrc/i18n/locales/ko/launch.jsonsrc/i18n/locales/nl/launch.jsonsrc/i18n/locales/pt-BR/launch.jsonsrc/i18n/locales/zh-CN/launch.jsonsrc/i18n/locales/zh-TW/launch.jsonsrc/lib/camera/backgroundEffect.ts
|
There is a good chance adding mediapipe to the main thread, while also recording, will result in heavy lag, maybe consider webworkers with mediapipe. They can work with GPU too. |
5 participants
Wraps the webcam MediaStream with a canvas-based pipeline that uses MediaPipe Selfie Segmentation to apply a background blur, solid color, or replacement image. Effect applies to both the live preview and the recorded -webcam.webm sidecar.
UI lives in the camera dropdown with a mode selector, intensity slider for blur, color picker for solid color, and image upload for image mode. Choice is persisted in localStorage. MediaPipe model and WASM runtime are bundled under public/mediapipe so it works offline.
Pull Request Template
Description
Motivation
Type of Change
Related Issue(s)
Screenshots / Video
Screenshot (if applicable):
Video (wherever possible):
Testing Guide
Checklist
Thank you for contributing!
Summary by CodeRabbit