Fix cut boundaries: timeline drift, audio-guided edges, phrase cutter - #8
Open
duncsdownunder wants to merge 3 commits into
Open
Fix cut boundaries: timeline drift, audio-guided edges, phrase cutter#8duncsdownunder wants to merge 3 commits into
duncsdownunder wants to merge 3 commits into
Conversation
Both apply-cuts.mjs and cut-silences.mjs computed retimed transcript timestamps by exact float subtraction, then fed those same floats to ffmpeg's trim/atrim filters, which can only cut on real frame boundaries. Each cut silently lost a fraction of a frame versus the math, and the error compounded linearly with cut count (measured up to ~1.5s drift across 163 cuts on a real 48-minute podcast render), so any downstream tool trusting the retimed transcript against the actual rendered video would land on the wrong word once enough cuts had accumulated. Snap every cut boundary to the nearest real video frame (detected via ffprobe) before computing the retimed transcript, when --video is given. Testing on real footage shows this substantially reduces drift but does not fully eliminate it (real frame spacing isn't perfectly uniform), so both SKILL.md files now document the residual limitation and require re-transcribing the final render to diff against intended text before calling any such cut "done" -- frame strips alone can't catch a uniform time offset on a locked-off single camera. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PbLt6aLpY5gruwSHS2AXaC
…holds
The previous commit snapped cut boundaries to real frame times but then
wrote them into the ffmpeg filtergraph with toFixed(3). A 30fps frame
boundary is 33.333...ms, so millisecond rounding pushed the value past the
frame's real timestamp, ffmpeg dropped the boundary frame, video ran one
frame short of its sample-exact audio, and concat shifted every later
segment a few ms in the same direction. Net effect: the snap did nothing.
Controlled A/B on one identical 45-cut clip, drift measured by
re-transcribing each render:
pre-fix 0.27s at end of file
snap + ms-precision (previous) 0.31s -- no improvement
snap + us-precision (this) 0.01s, rendered duration matches the
transcript's claim exactly
The previous commit message and both SKILL.md files overclaimed a
substantial reduction that was never measured against a controlled
baseline; the docs now state the measured comparison and keep the
"re-transcribe the final render and diff" requirement as defense in depth.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PbLt6aLpY5gruwSHS2AXaC
…bugs Second cut-boundary fix from the podcast cutdowns project. The first (frame snapping at microsecond precision) fixed timeline drift; this one fixes cuts landing on the first syllable of the next word. Cause: ASR word timestamps are 50-150ms coarse and speakers usually start the next word inside that window, so a cut at word.end (plus any pad) clips the following onset. apply-cuts.mjs now moves each delete edge to the quietest 30ms nearby (up to 150ms outward, 30ms inward, on a 16 kHz mono extract) and never past the neighbouring kept words, before frame snapping. On by default with --video; --no-snap-audio restores the old behaviour. The helper is scripts/lib/audio-edges.mjs with its own tests. scripts/cut-phrases.mjs is the phrase-driven cutter used to build the nine ads: name each segment's first and last words, resolve them against short ground-truth transcripts of the source file itself, snap, cut with dense keyframes, assemble, then re-transcribe the assembly and diff it against the intended text. Skill and workflow docs updated: audio-guided edges section in cut-mistakes, cut-phrases + keyframe + strict-render guidance in short-form-edit, apply-cuts flag in WORKFLOW.md. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PbLt6aLpY5gruwSHS2AXaC
duncsdownunder
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three related fixes for cut boundaries, found while cutting nine short ads out of a 52-minute podcast with this kit.
1. Timeline drift (commits 2db07f9, 4e9dd06)
cut-silencesandapply-cutscomputed the retimed transcript from exact float subtraction while ffmpeg cut on frame boundaries, and the filtergraph wrote trim times at millisecond precision, which rounds a 33.333ms boundary past the frame and drops it. Across a few hundred cuts the transcript drifted 1–3s from the render. Fix: snap delete edges to the source frame grid (only with--video) and write trim times at microsecond precision.Controlled A/B on one identical 45-cut clip:
2. Audio-guided cut edges (this commit)
ASR word timestamps are 50–150ms coarse and speakers usually start the next word inside that window, so a cut at
word.endlands on the first syllable of the following word.apply-cuts.mjsnow moves each delete edge to the quietest 30ms nearby (up to 150ms outward, 30ms inward, on a 16 kHz mono extract of the source) and never past the neighbouring kept words, before frame snapping. On by default with--video;--no-snap-audiokeeps the old behaviour. The EDL records both requested and snapped times. Helper:scripts/lib/audio-edges.mjs, with its own tests (tests/audio-edges.test.mjs).Measured on a 41-cut set: last-80ms energy before each cut went from speech level (≈-30 dB) to -31…-78 dB.
3.
scripts/cut-phrases.mjsThe phrase-driven cutter that produced the ads: name each segment's first and last words, resolve them against short ground-truth transcripts of the source file itself (never a retimed transcript), snap, cut with dense keyframes (
-g 30, the renderer otherwise warns about seek failures), assemble, then re-transcribe the assembly and diff it against the intended text.Docs
cut-mistakes/SKILL.md: retimed-transcript precision section (fix 1), new "Audio-guided edges" section (fix 2), pointer to cut-phrases.cut-silences/SKILL.md: precision section with the A/B numbers.short-form-edit/SKILL.md: cut-phrases, keyframes, and lint + validate +--strictbefore every render.docs/WORKFLOW.md: the new flag and script.npm test: 14/14. Skills synced to.agents/.🤖 Generated with Claude Code
https://claude.ai/code/session_01PbLt6aLpY5gruwSHS2AXaC