Client-side media: Add video transcoding to web-safe formats - #79375
Client-side media: Add video transcoding to web-safe formats#79375adamsilverstein wants to merge 28 commits into
Conversation
github-actions
Bot
added
[Package] Block library
adamsilverstein
added
[Feature] Client Side Media
|
The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message. To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook. |
|
Size Change: +130 kB (+1.65%) Total Size: 8.05 MB 📦 View Changed
|
|
Flaky tests detected in 34667e1. 🔍 Workflow run URL: https://github.com/WordPress/gutenberg/actions/runs/33114972868 three users concurrently edit a large post with diverse blocks in
|
Detect uploaded videos that are not web-safe (non-MP4/WebM container, non-web-safe codec, oversized, or over a bitrate budget) and transcode them to MP4/H.264 (or WebM/VP9) in the browser before they are served, mirroring the GIF-to-video flow. The original upload is preserved as the attachment (consistent with how WordPress keeps the `-scaled` and HEIC originals); the transcoded, web-safe version is sideloaded as a companion file and is what plays. A `videoKeepOriginal` setting flips the pipeline to replace-primary so only the optimized file is stored. Transcoding runs off the main thread in the existing @wordpress/video-conversion worker via mediabunny's Conversion API, with hardware acceleration where available and graceful fallback to the original when the browser cannot encode. Video encoding shares the single-concurrency WebCodecs gate with GIF conversion. Part of #76756. Implements #79363.
…g-mediabunny # Conflicts: # lib/media/class-gutenberg-rest-attachments-controller.php
…g-mediabunny # Conflicts: # lib/media/class-gutenberg-rest-attachments-controller.php # packages/block-editor/src/components/provider/use-media-upload-settings.js # packages/block-library/src/video/edit.js # packages/upload-media/src/store/private-actions.ts # packages/video-conversion/README.md # packages/video-conversion/src/index.ts # packages/video-conversion/src/video-conversion-worker.ts # packages/video-conversion/src/worker.ts
Merging trunk brought in stricter `@wordpress/dependency-group` and `import/order` rules these branch-owned files predate. The `waitForFunction` call also passed its options as the second argument, where Playwright expects the polling arg, so the 120s timeout was silently falling back to `actionTimeout`.
github-project-automation
Bot
moved this to 🔎 Needs Review
in WordPress 7.2 Editor Tasks
WebCodecs treats 'prefer-hardware' as a requirement rather than a hint, so the browser rejects the encoder configuration outright when no hardware encoder is available for the codec and output size. Headless browsers, VMs and CI runners have none, which made every transcode fail there. Probe encoder support at the real output dimensions and bitrate before encoding, and fall back to 'no-preference' so the browser can reach its software encoder. Only give up, gracefully, when no hint works. Also read the source dimensions unconditionally: they are needed for the probe, since support is parameter-specific.
Production code changed in block-editor, block-library, upload-media and video-conversion needs an entry in each package's changelog.
The sideload endpoint only accepted image and PDF attachments, because a sideloaded file is a sub-size of the attachment it extends. The web-safe transcode of an uploaded video is a companion of the video attachment itself, so every transcode was rejected with "Only images and PDFs can be sideloaded" and the original video was silently kept instead. Accept a video parent, scoped to the 'optimized-video' size so a video is still rejected for the sub-sizes it cannot have.
|
Hey 👋 I wanted to give you a heads-up since this pull request adds new files containing JSX. #80123 enables an ESLint rule that requires that files containing JSX must use the What you'll need to do: You'll need to rename any new files containing JSX to use the This comment is automated, based on pull requests with recent activity that contain affected |
Trunk now rejects JSX in `.js` files, and new source files in the repository are TypeScript. Move the control to `.tsx` and type its props and the attachment record it reads.
A failed 'optimized-video' sideload was not in the optional-companion allowlist, so cancelItem took the total-failure branch and deleted the already-uploaded original video from the server. Share the companion size list between the GIF and video flows. Also re-kick items waiting on the shared video-encoding gate when a TranscodeVideo operation is cancelled, not only a TranscodeGif; without it a failed video transcode left every queued video stuck.
Name the audio codec browsers play for each container (AAC for MP4, Opus for WebM) and treat an audio track mediabunny discards as an unsupported conversion, so the original is kept instead of a mute companion playing by default. Drop computeDuration() from getVideoMetadata: no caller reads it, and on a container without an index it walks every cluster to EOF before the transcode decision is even made.
get_sideloaded_file_names() did not include the optimized_video metadata key, so the companion's sideload provenance row was never deleted. Also move the block-library changelog entry back under Unreleased after the 10.5.0 release bump.
The Media Library selection object has no media_details, so a video chosen from the library played the non-web-safe original while the toolbar offered the optimized version. Look the companion up on the attachment record when the selection does not carry it, as the animated-GIF transform already does.
Replace the window.__videoTranscodingKeepOriginal inline script, which the WordPress-agnostic block-editor package read directly, with a video_keep_original field on the REST API root index that flows through the editor settings like the image processing settings do.
|
Thanks for the review and testing @andrewserong - I went through your points with Claude and it put together this summary:
|
Every other special sideload size uses underscores (animated_video, source_original), and the Core backport follows that convention, so the token the editor sends must match.
The setting is read from the REST index into the block editor settings, but the base entity only requests an explicit _fields list, so it was never returned. Add it to the entity fields and the matching preload path.
…ideo upload. Add server-side support for the client-side video transcoding flow. When client-side media processing is enabled, the editor transcodes an uploaded video that is not already web-safe to MP4/WebM in the browser and sideloads the result as a companion of the original video attachment, which remains the attachment itself. - Accept a video attachment as the sideload parent for the new `optimized_video` companion size, and only for that size; every other size still requires an image or PDF parent. - Record the companion under the `optimized_video` attachment metadata key on finalize, and treat it as a name the request is allowed to store. - Delete the companion file alongside the attachment in `wp_delete_attachment_files()`. - Expose the new `wp_video_transcoding_keep_original` filter (default `true`) on the REST API index as `video_keep_original`, so the editor knows whether to keep the original upload or transcode before uploading, and preload it with the other media processing settings. See WordPress/gutenberg#79375.
|
Core backport: WordPress/wordpress-develop#13323 (Trac https://core.trac.wordpress.org/ticket/65998) Claude prepared the backport and pushed two small fixes here along the way:
|
|
Thanks for summarizing, Adam! It's been a while since I've looked at this, so just sharing some thoughts after having not looked at it for a while. Please know that I think as a long-term goal being able to support video conversion is a great idea, so my thoughts here are in the spirit of supporting the effort! Now, onto the approach. I think there are a lot of assumptions here, and this problem space is more complex than a single PR can capture succinctly. Transcoding can take a lot of time, and depending on the computer, is computationally expensive. Will it slow down the editor for some users while it's happening? What if a user saves their post and closes the window, or closes their laptop lid. Where will they be in the transcoding process? What if they're in the site editor and switch to a different view from the post they were working on? What is the baseline configuration that we should use for a single video? How can a user preview what it looks like and make choices about how they want it to look? What if the video is too compressed, or not compressed enough? The differences for an individual image can be marginal, but on a bigger video can be significant. And what about audio codecs? What if the video looks fine, but an indie artist's music video doesn't sound right? If we're using the current image pipeline as an example, users can currently choose from a range of sub-sizes or select "full" to show the original, after the conversion has happened. This is easy to do because generating sub-sizes for images is quick, so the UI to make that selection can kind of happen after the conversion has happened. Whereas transcoding video is typically slow. All of this is why I'm skeptical of transcoding behind the scenes being the right path for this feature. Especially if one of the goals is to ultimately not upload the original file for the case of very large media. It isn't clear to me in the linked issue (#79363) if the goal is to solve the storage problem or not, as it's used as one of the reasons we should do this feature, but it's also mentioned that we're preserving the file that the user uploaded. There are good arguments either way, but in its current form, I wasn't sure where we're trying to land. So, my recommendation here would be to consider (and maybe this is more a discussion for the linked issue): what's the best UX we'd like to provide a user for this feature? Unlike client-side image processing which can fall back to a server-side approach, this is a new feature with no fallback. Once it's announced "WordPress can automatically convert your videos", we're committed to supporting it, so there's a bit less wriggle room IMO than for the still image processing. Alrighty, those are my first impressions. I think overall it's a good feature idea, but also a project in and of itself to make it feel seamless and polished. Would it be worth introducing it behind an experiment to begin with so that many of these questions can be explored after the initial foundations have landed? I like asking all these questions, but I'm sure there's much of this PR that is infrastructural that doesn't needed to be blocked by thinking through all the particulars. (I.e. I'm trying to figure out how I can best be supportive here without needlessly blocking good work!) |
4 participants
What?
Adds client-side video transcoding to web-safe formats for the client-side media feature, implementing #79363. When a video that is not already web-safe is uploaded, it is transcoded in the browser (via mediabunny / WebCodecs) to MP4/H.264 (or WebM/VP9) before it is served, mirroring what we already do for images and the GIF-to-video conversion in #78410.
Fixes #79363.
Note
This PR is stacked on #78410 (
add/gif-to-video-mediabunny) and reuses its@wordpress/video-conversionpackage, worker boundary, and companion-file infrastructure. It targets that branch and should be rebased ontotrunkonce #78410 lands. Review the last 6 commits.Why?
WordPress accepts video uploads in any format (
.mov,.mkv, high-bitrate.mp4) and never transcodes them, because server-side transcoding needs FFmpeg, which most hosts lack. The result is slow page loads, high bandwidth/storage cost, and clips that won't play in some browsers. This is one of the capabilities called out for the 7.1 cycle and was only a roadmap bullet on #76756.How?
Keep the original, serve a web-safe companion. WordPress always preserves the original upload (so it can be linked to or used to regenerate later, like the
-scaledand HEIC originals). So the original video is stored as the attachment and the transcoded, web-safe version is sideloaded as a companion file (recorded in attachment metadata underoptimized_video). Thecore/videoblock points its playbacksrcat the companion; a toolbar control lets authors switch back to the original.Pipeline. Detection and transcoding reuse the
@wordpress/upload-mediaqueue:prepareItemprobes the video metadata (getVideoMetadata, a cheap header read) andneedsVideoTranscodedecides eligibility (non-web-safe container/codec, oversized past the 1920px threshold, or over an optional bitrate budget) — already-optimized small files are skipped.Upload → GenerateVideoCompanion → Finalize; the companion is transcoded off the main thread via mediabunny's high-levelConversionAPI withhardwareAcceleration: 'prefer-hardware', then sideloaded.Developer opt-out. The
gutenberg_video_transcoding_keep_originalPHP filter (defaulttrue) flips the pipeline to transcode-before-upload, storing only the optimized file. Since video files can be very large, this affords the ability to reduce storage requirements with the tradeoff that the original video is no longer available for download or regeneration.Testing Instructions
.mov, an.mkv, or a large/high-bitrate.mp4)..mp4once the upload finishes, while the Media Library still holds the original upload..mp4and confirm it is left untouched.add_filter( 'gutenberg_video_transcoding_keep_original', '__return_false' );and confirm only the optimized file is stored.Automated tests
npm run test:unit packages/upload-media packages/video-conversionvendor/bin/phpunit phpunit/media/video-transcoding-test.phpnpm run test:e2e -- test/e2e/specs/editor/various/video-transcoding.spec.jsPart of #76756.
AI use
This PR was written by Claude after careful prompting, planning and review from me. I also plan to manually review the code and test the feature directly.